CN114638762B - Modularized hyperspectral image scene self-adaptive panchromatic sharpening method - Google Patents
Modularized hyperspectral image scene self-adaptive panchromatic sharpening method Download PDFInfo
- Publication number
- CN114638762B CN114638762B CN202210293380.7A CN202210293380A CN114638762B CN 114638762 B CN114638762 B CN 114638762B CN 202210293380 A CN202210293380 A CN 202210293380A CN 114638762 B CN114638762 B CN 114638762B
- Authority
- CN
- China
- Prior art keywords
- adaptive
- scene
- convolution
- image
- scene self
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 64
- 238000012549 training Methods 0.000 claims abstract description 40
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 33
- 238000000605 extraction Methods 0.000 claims abstract description 25
- 238000003062 neural network model Methods 0.000 claims abstract description 14
- 238000012360 testing method Methods 0.000 claims abstract description 14
- 238000007781 pre-processing Methods 0.000 claims abstract description 7
- 230000006870 function Effects 0.000 claims description 24
- 238000001228 spectrum Methods 0.000 claims description 20
- 230000003044 adaptive effect Effects 0.000 claims description 18
- 238000005457 optimization Methods 0.000 claims description 9
- 238000010586 diagram Methods 0.000 claims description 8
- 238000006243 chemical reaction Methods 0.000 claims description 7
- 238000005070 sampling Methods 0.000 claims description 7
- 230000006835 compression Effects 0.000 claims description 5
- 238000007906 compression Methods 0.000 claims description 5
- 230000002776 aggregation Effects 0.000 claims description 4
- 238000004220 aggregation Methods 0.000 claims description 4
- 230000006872 improvement Effects 0.000 claims description 4
- 230000015556 catabolic process Effects 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 3
- 238000006731 degradation reaction Methods 0.000 claims description 3
- 238000009499 grossing Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 230000004044 response Effects 0.000 claims description 3
- 238000003860 storage Methods 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 8
- 239000011159 matrix material Substances 0.000 description 38
- 230000008569 process Effects 0.000 description 21
- 230000004913 activation Effects 0.000 description 15
- 238000012545 processing Methods 0.000 description 11
- 230000003595 spectral effect Effects 0.000 description 9
- 239000011800 void material Substances 0.000 description 6
- 238000011426 transformation method Methods 0.000 description 4
- 238000006467 substitution reaction Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000002708 enhancing effect Effects 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000002347 injection Methods 0.000 description 1
- 239000007924 injection Substances 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000000513 principal component analysis Methods 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 238000007711 solidification Methods 0.000 description 1
- 230000008023 solidification Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a modularized hyperspectral image scene self-adaptive full-color sharpening method, which comprises the following steps: reading a hyperspectral image and a full-color image matched with the hyperspectral image, and preprocessing; constructing a training data set and a test data set; constructing a multi-scale feature extraction module, further obtaining a scene self-adaptive sharpening module, and constructing a modularized scene self-adaptive convolutional neural network model; initializing the weight and bias of each convolution layer of the modularized scene self-adaptive convolution neural network model, inputting a training data set into the modularized scene self-adaptive convolution neural network model, and obtaining a predicted image through forward propagation of the network to finish training; and inputting the test data set into the trained modularized scene self-adaptive convolutional neural network model to obtain a hyperspectral image with high spatial resolution. The invention effectively reduces local distortion in the sharpening result and enhances the full-color sharpening effect.
Description
Technical Field
The invention relates to the field of remote sensing image processing, in particular to a modularized hyperspectral image scene self-adaptive panchromatic sharpening method.
Background
In order to ensure that the imaging result has an acceptable signal-to-noise ratio, in general, a spectrum imaging system needs to comprehensively consider the mutual constraint factors between the spatial resolution and the spectral resolution of a remote sensing image, and it is difficult to directly acquire a hyperspectral image with high spatial resolution through a single sensor, but acquire a hyperspectral image with relatively low spatial resolution and a single-band full-color image with high spatial resolution, and then the spectral information and the spatial information of the two types of images are fused through an image processing technology, so that a hyperspectral image with high spatial resolution is generated. This process is known as full color sharpening of hyperspectral images.
Traditional hyperspectral image full-color sharpening methods can be divided into three major categories, including component substitution methods, multi-resolution analysis methods, and methods based on variation optimization. The component substitution method replaces the spatial component of the low spatial resolution hyperspectral image with the full-color image through a domain transformation technology so as to realize the sharpening effect, and the method mainly comprises a principal component analysis transformation method, a Schmidt orthogonal transformation method and the like. The multi-resolution analysis method relates to space detail information extraction and re-injection under multiple scales, and mainly comprises a wavelet transformation method, a Laplacian pyramid transformation method and the like. The method based on variation optimization reconstructs a high-spatial resolution hyperspectral image by solving an image degradation inverse problem through a variation theory and an iterative optimization algorithm, and mainly comprises a non-negative matrix factorization algorithm, a Bayesian algorithm and the like.
In recent years, many hyperspectral image full-color sharpening methods based on neural networks are sequentially proposed thanks to the mining of large-scale data and the improvement of hardware computing power, and the performance effect far exceeding that of the traditional full-color sharpening method is achieved. These methods typically utilize convolutional neural networks to obtain high spatial resolution hyperspectral images from low spatial resolution hyperspectral images as well as panchromatic images in an end-to-end fashion. However, due to the weight sharing characteristic of the convolution kernel and the solidification of the kernel parameters during the test, the existing convolutional neural network-based method tends to focus on the global sharpening effect, neglecting the fact that the local scene characteristics are different, and lacking a means for adaptively adjusting according to the scene characteristics of different spatial positions in the input image, so that the sharpening result is easy to generate local distortion.
Disclosure of Invention
In order to overcome the defects and shortcomings in the prior art, the invention provides a modularized hyperspectral image scene self-adaptive full-color sharpening method.
The method follows the modularized thought and designs a multi-scale feature extraction module and a scene self-adaptive sharpening module. The multi-scale feature extraction module can provide receptive fields of multiple scales by combining with cavity convolution of different cavity rates, so that targets with different sizes in the input feature map are fully captured, and space information of the input feature map is effectively extracted. The scene self-adaptive sharpening module can adaptively generate a group of modulation convolution kernels according to scene characteristics of different spatial positions in the input feature map, so that scenes of the different spatial positions of the input feature map are subjected to fine adjustment through the group of convolution kernels, and local distortion in a sharpening result is reduced. On the basis, the invention constructs a modularized scene self-adaptive convolutional neural network by using the designed multi-scale feature extraction module and the scene self-adaptive sharpening module, and the network can carry out gradual self-adaptive adjustment on scenes at different spatial positions according to the difference of scene characteristics at different spatial positions in an input image through the serial connection of a plurality of scene self-adaptive sharpening modules, thereby effectively reducing local distortion in sharpening results and enhancing full-color sharpening effects.
The invention adopts the following technical scheme:
a modularizable hyperspectral image scene adaptive panchromatic sharpening method, comprising:
Reading a hyperspectral image and a full-color image matched with the hyperspectral image, and preprocessing;
Constructing a training data set and a testing data set based on the preprocessed hyperspectral image and full-color image;
Constructing a multi-scale feature extraction module, further obtaining a scene self-adaptive sharpening module, and constructing a modularized scene self-adaptive convolutional neural network model according to the scene self-adaptive sharpening module;
the method comprises the following steps: the multi-scale feature extraction module comprises three layers of convolution layers, wherein one aggregation convolution layer is input into the three layers of convolution layers, and each layer of convolution layer comprises a first cavity convolution layer, a second cavity convolution layer and a third cavity convolution layer;
The scene self-adaptive sharpening module comprises a multi-scale feature extraction module, an expansion convolution layer, a spatial spectrum conversion layer, two scene fine adjustment convolution layers and a scene self-adaptive convolution layer;
Initializing the weight and bias of each convolution layer of the modularized scene self-adaptive convolution neural network model, inputting a training data set into the modularized scene self-adaptive convolution neural network model, and obtaining a predicted image through forward propagation of the network to finish training;
and inputting the test data set into the trained modularized scene self-adaptive convolutional neural network model to obtain a hyperspectral image with high spatial resolution.
Further, the spatial dimensions of the hyperspectral image and the panchromatic image satisfy the following relationship:
r＝h2/h1＝w2/w1
where r represents the ratio of the spatial resolution of the full-color image to the hyperspectral image, h 1 and w 1 represent the height and width of the hyperspectral image, respectively, and h 2 and w 2 represent the height and width of the full-color image, respectively.
Further, the preprocessing includes:
Performing low-pass filtering on the hyperspectral image and the full-color image by using a smoothing filter with adaptive frequency response to obtain a first hyperspectral image and a first full-color image;
downsampling the first hyperspectral image and the first panchromatic image to obtain a second hyperspectral image and a second panchromatic image with r times of the degradation of the spatial resolution;
And up-sampling the second hyperspectral image by using a polynomial interpolation method to obtain a third hyperspectral image with r times of spatial resolution improvement.
Further, in the multi-scale feature extraction module, the void ratios of the three first void convolution layers are different, the void ratios of the three second void convolution layers are different, and the void ratios of the three third void convolution layers are different.
Further, the scene adaptive sharpening module includes:
multi-scale feature extraction module and input feature map Output 64 feature maps/>Extracting multi-scale space characteristic information is realized;
An expanded convolution layer comprising 576 standard convolution kernels with a receptive field of 3×3, and an input feature map 576 Feature maps E (i) are output;
The space spectrum conversion layer inputs the characteristic diagram E (i) and outputs a scene self-adaptive convolution kernel K (i) with the receptive field of 64×h 2×w2 groups of 3×3;
The first scene fine-tunes the convolutional layer, comprising 64 standard convolutional kernels with a receptive field of 3x 3. Inputting a feature map Output 64 feature maps/>
The second scene fine-tunes the convolutional layer, comprising 64 standard convolutional kernels with a receptive field of 3x 3. Inputting a feature mapOutput 64 feature maps/>
Scene self-adaptive convolution layer and input feature mapAnd a scene self-adaptive convolution kernel K (i) for outputting 64 feature graphs
Further, constructing a modularized scene self-adaptive convolutional neural network model according to the scene self-adaptive sharpening module, specifically:
A first spectrum compression convolution layer comprising 64 standard convolution kernels with receptive fields of 1×1, and inputting hyperspectral image training samples Output 64 feature maps/>
A second spectrum compression convolution layer comprising 64 standard convolution kernels with 1×1 receptive fields and an input feature mapOutput 64 feature maps/>
Splicing layer for inputting full-color image training sampleAnd feature map/>Outputting 65 feature maps C (i);
The first scene self-adaptive sharpening module inputs a feature map C (i) to realize first-stage scene self-adaptive modulation;
The second scene self-adaptive sharpening module inputs the output of the first-stage scene self-adaptive modulation and realizes the second-stage scene self-adaptive modulation;
The third scene self-adaptive sharpening module inputs the output of the second stage scene self-adaptive modulation and outputs 64 feature maps Realizing third-level scene self-adaptive modulation;
A first spectrum reconstruction convolution layer comprising 64 standard convolution kernels with a receptive field of 1×1, and an input feature map Output 64 feature maps/>
The second spectral reconstruction convolution layer contains 64 standard convolution kernels with a receptive field of 1 x 1. Inputting a feature mapOutput b feature maps/>
The spectrum compensation layer inputs a hyperspectral image training sampleAnd feature map/>The predicted image O (i) is output.
Further, the weight and bias of each convolution layer of the modularized scene self-adaptive convolution neural network model are initialized, a training data set is input into the modularized scene self-adaptive convolution neural network model, a predicted image is obtained through forward propagation of the network, and training is completed, specifically:
presetting fixed value parameters, wherein the fixed value parameters comprise learning rate, iteration times and input sample number, and initializing weights and offsets of all convolution layers of the modularized scene self-adaptive convolution neural network model;
acquiring a training sample with low spatial resolution from a training data set, inputting the training sample into a modularized scene self-adaptive convolutional neural network model, and obtaining a predicted image through forward propagation of a network;
Selecting an average absolute error as a loss function, calculating an error value between a predicted image and a high spatial resolution reference image, minimizing the error value by using a gradient-based optimization algorithm, and iteratively updating the weight and bias of the modularized scene self-adaptive convolutional neural network;
and when the error value is converged to the minimum value, obtaining and storing the optimal weight and bias of the modularized scene self-adaptive convolutional neural network, and obtaining the trained modularized scene self-adaptive convolutional neural network model.
Further, the expression of the average absolute error as a loss function is:
Wherein phi represents the input-output mapping relation of the modularized scene self-adaptive convolutional neural network, theta represents the weight and bias of the network, N p represents the number of training samples input during each round of iterative optimization, and I F represents the Frobenius norm.
A storage medium having stored thereon a computer program which when executed by a processor implements the hyperspectral image scene adaptive panchromatic sharpening method.
An apparatus comprising a memory, a processor, and a hyperspectral image scene adaptive panchromatic sharpening method stored on the memory and executable on the processor.
The invention has the beneficial effects that:
(1) The modularized hyperspectral image scene self-adaptive panchromatic sharpening method provided by the invention designs a multi-scale feature extraction module, and the module can provide a plurality of scales of receptive fields by combining with cavity convolution with different cavity rates, so that targets with different sizes in an input feature map are fully captured, and the spatial information of the input feature map is effectively extracted.
(2) The modularized hyperspectral image scene self-adaptive panchromatic sharpening method designs a scene self-adaptive sharpening module, and the module can adaptively generate a group of modulation convolution kernels according to scene characteristics of different spatial positions in an input feature map, so that the scenes of the different spatial positions of the input feature map are subjected to fine adjustment through the group of convolution kernels, and local distortion in a sharpening result is reduced.
(3) The modularized hyperspectral image scene self-adaptive panchromatic sharpening method provided by the invention uses the designed multi-scale feature extraction module and scene self-adaptive sharpening module to construct a modularized scene self-adaptive convolutional neural network, and the network can carry out gradual self-adaptive adjustment on scenes at all spatial positions according to the differences of scene characteristics at different spatial positions in an input image through the serial connection of a plurality of scene self-adaptive sharpening modules, thereby effectively reducing local distortion in sharpening results and enhancing panchromatic sharpening effects.
Drawings
FIG. 1 is a workflow diagram of the present invention;
FIG. 2 is a block diagram of a multi-scale feature extraction module of the present invention;
FIG. 3 is a block diagram of a scene adaptive sharpening module of the present invention;
FIG. 4 is a block diagram of a scene adaptive convolutional neural network model of the present invention;
Fig. 5 (a) is a Houston hyperspectral reference image, fig. 5 (b) is a third hyperspectral image after up-sampling processing using bicubic interpolation, fig. 5 (c) is an image after processing using a non-negative matrix factorization algorithm, fig. 5 (d) is an image after processing using a bayesian algorithm, and fig. 5 (e) is an image after processing using the method described in this embodiment.
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but embodiments of the present invention are not limited thereto.
Example 1
As shown in fig. 1-4, a modularized hyperspectral image scene self-adaptive full-color sharpening method comprises the following steps:
S1, reading an original hyperspectral image and a matched full-color image, and preprocessing;
Read hyperspectral image Paired panchromatic image/>The space size of (2) satisfies the following quantitative relationship:
r＝h2/h1＝w2/w1
Where r represents the ratio of the spatial resolution of the full-color image to the hyperspectral image, h 1 and w 1 represent the height and width of the hyperspectral image, respectively, and h 2 and w 2 represent the height and width of the full-color image, respectively. b represents the number of spectral channels of the hyperspectral image.
Further, the specific process of pretreatment is as follows: using a filter with a specific frequency response to an original hyperspectral imageAnd panchromatic image/>Smoothing to obtain a first hyperspectral image and a first full-color image, and then performing downsampling treatment of reducing the spatial resolution by r times on the first hyperspectral image and the first full-color image to obtain a second hyperspectral imageAnd a second panchromatic image/>Then, the polynomial interpolation method is used for carrying out up-sampling treatment on the second hyperspectral image with the improved spatial resolution by r times, and a third hyperspectral image/>, is obtained
S2, constructing a training data set and a testing data set based on the preprocessed hyperspectral image and full-color image;
further, according to the principle of non-overlapping each other, from the third hyperspectral image Is a specific region of the second full-color image/>To intercept partial sub-images/>, at fixed sampling intervalsAnd/>As training samples,/>And/>Randomly sequencing to form a training data set; from the third hyperspectral image/>Is the remaining area of the second full color image/>The corresponding position cut-out sub-images of (2) are used as test samples and constitute a test data set.
It is generally considered that the training set needs to include the characteristics of all features as much as possible, and is representative, so that a partial region must be an image region that is selected to be richer in feature types. The specific region in this embodiment is the above-described relatively rich image region.
S3, constructing a multi-scale feature extraction module.
Further, the multi-scale feature extraction module comprises three layers of convolution layers, wherein one aggregation convolution layer is input into the three layers of convolution layers, and each layer of convolution layer comprises a first cavity convolution layer, a second cavity convolution layer and a third cavity convolution layer.
Specifically comprising:
The first hole convolution layer D-conv1_1 contains 64 hole convolution kernels with a receptive field of 3×3 and a hole rate d=1. Inputting a feature map Output 64 feature maps/>The computational process can be expressed as/> Wherein/>And/>The weight matrix and the bias matrix representing the layer of hole convolution kernels respectively,Representation Leaky Relu activates a function;
the first hole convolution layer D-conv1_2 contains 64 hole convolution kernels with a receptive field of 3×3 and a hole rate d=2. Inputting a feature map Output 64 feature maps/>The computational process can be expressed as/> Wherein/>And/>Weight matrix and bias matrix respectively representing the layer cavity convolution kernel,/>Also denoted Leaky Relu is an activation function;
The first hole convolution layer D-conv1_3 contains 64 hole convolution kernels with a receptive field of 3×3 and a hole rate d=3. Inputting a feature map Output 64 feature maps/>The computational process can be expressed as/> Wherein/>And/>Weight matrix and bias matrix respectively representing the layer cavity convolution kernel,/>Also denoted Leaky Relu is an activation function;
the second hole convolution layer D-conv2_1 contains 64 hole convolution kernels with a receptive field of 3×3 and a hole rate d=1. Inputting a feature map Output 64 feature maps/>The computational process can be expressed as/> Wherein/>And/>Weight matrix and bias matrix respectively representing the layer cavity convolution kernel,/>Also denoted Leaky Relu is an activation function;
The second hole convolution layer D-conv2_2 contains 64 hole convolution kernels with a receptive field of 3×3 and a hole rate d=2. Inputting a feature map And/>Output 64 feature maps/>The operational process can be expressed asWherein/>And/>Weight matrix and bias matrix respectively representing the layer cavity convolution kernel,/>Representation will/>And/>Splicing in channel dimension,/>Also denoted Leaky Relu is an activation function;
The second hole convolution layer D-conv2_3 contains 64 hole convolution kernels with a receptive field of 3×3 and a hole rate d=3. Inputting a feature map And/>Output 64 feature maps/>The operational process can be expressed asWherein/>And/>Weight matrix and bias matrix respectively representing the layer cavity convolution kernel,/>Representation will/>And/>Splicing in channel dimension,/>Also denoted Leaky Relu is an activation function;
The third hole convolution layer D-conv3_1 contains 64 hole convolution kernels with a receptive field of 3×3 and a hole rate d=1. Inputting a feature map Output 64 feature maps/>The computational process can be expressed as/> Wherein/>And/>Weight matrix and bias matrix respectively representing the layer cavity convolution kernel,/>Also denoted Leaky Relu is an activation function;
The third hole convolution layer D-conv3_2 contains 64 hole convolution kernels with a receptive field of 3×3 and a hole rate d=2. Inputting a feature map And/>Output 64 feature maps/>The operational process can be expressed asWherein/>And/>Weight matrix and bias matrix respectively representing the layer cavity convolution kernel,/>Representation will/>And/>Splicing in channel dimension,/>Also denoted Leaky Relu is an activation function;
the third hole convolution layer D-conv3_3 contains 64 hole convolution kernels with a receptive field of 3×3 and a hole rate d=3. Inputting a feature map And/>Output 64 feature maps/>The operational process can be expressed asWherein/>And/>Weight matrix and bias matrix respectively representing the layer cavity convolution kernel,/>Representation will/>And/>Splicing in channel dimension,/>Also denoted Leaky Relu is an activation function;
The aggregate convolution layer a-Conv contains 64 standard convolution kernels with a receptive field of 3 x 3. Inputting a feature map And/>Output 64 feature maps/>The computational process can be expressed as/> Wherein W A and B A represent the weight matrix and bias matrix, respectively, of the standard convolution kernel of the layer,/>Representation will/>And/>The stitching is performed in the dimension of the channel,Also denoted Leaky Relu is the activation function.
S4, constructing a scene self-adaptive sharpening module.
The scene self-adaptive sharpening module comprises a multi-scale feature extraction module, an expansion convolution layer, a spatial spectrum conversion layer, two scene fine adjustment convolution layers and a scene self-adaptive convolution layer.
Multi-scale feature extraction module and input feature mapOutput 64 feature maps/>Extracting multi-scale space characteristic information is realized;
the dilated convolution layer E-Conv contains 576 standard convolution kernels with a receptive field of 3X 3. Inputting a feature map 576 Feature maps E (i) are output. The computational process can be expressed as/>Where W E and B E represent the weight matrix and bias matrix, respectively, of the standard convolution kernel of the layer, which does not use an activation function.
The space spectrum conversion layer C-to-S inputs the characteristic diagram E (i) and outputs a scene self-adaptive convolution kernel K (i) with a 64×h 2×w2 group receptive field of 3×3. The layer rearranges the channel dimension of the input feature map E (i), converts the channel dimension into a space dimension, and generates a 3×3 scene self-adaptive convolution kernel K (i);
the first scene fine-tuning convolutional layer S-Conv1 contains 64 standard convolutional kernels with a receptive field of 3 x 3. Inputting a feature map Output 64 feature maps/>The computational process can be expressed as/>Wherein/>And/>Weight matrix and bias matrix respectively representing the standard convolution kernel of the layer,/> The representation Relu activates the function.
The second scene fine-tuning convolutional layer S-Conv2 contains 64 standard convolutional kernels with a receptive field of 3 x 3. Inputting a feature mapOutput 64 feature maps/>The computational process can be expressed as/>Wherein/>And/>Weight matrix and bias matrix respectively representing the standard convolution kernel of the layer,/>Also denoted Relu is the activation function.
Scene self-adaptive convolution layer SA-Conv, and input feature mapAnd a scene self-adaptive convolution kernel K (i), outputting 64 feature graphs/>The computational process can be expressed as/>Wherein/>A pixel-by-pixel convolution operation is represented,Also denoted Relu is the activation function.
S5, constructing a modularized scene self-adaptive convolutional neural network model according to the scene self-adaptive sharpening module;
further, the method comprises the steps of:
The first spectral compression convolution layer Conv1 contains 64 standard convolution kernels with a receptive field of 1 x 1. Training sample for inputting hyperspectral image Output 64 feature maps/>The computational process can be expressed as/> Wherein W 1 and B 1 represent the weight matrix and bias matrix, respectively, of the standard convolution kernel of the layer,/>Representation Relu activates a function;
The second spectrally compressed convolution layer Conv2 comprises 64 standard convolution kernels with a receptive field of 1 x 1. Inputting a feature map Output 64 feature maps/>The computational process can be expressed as/>Wherein W 2 and B 2 represent the weight matrix and bias matrix, respectively, of the standard convolution kernel of the layer,/>Also denoted Relu is an activation function;
splice layer Concat. Input panchromatic image training samples And feature map/>65 Feature maps C (i) are output. This layer trains the sample/>, full color imageAnd feature map/>Splicing is carried out in the channel dimension to obtain a spliced characteristic diagram C (i);
The first scene adaptive sharpening module inputs a feature map C (i). Realizing first-stage scene self-adaptive modulation;
And the second scene self-adaptive sharpening module is used for inputting the output of the first-stage scene self-adaptive modulation. Realizing second-stage scene self-adaptive modulation;
The third scene self-adaptive sharpening module inputs the output of the second stage scene self-adaptive modulation and outputs 64 feature maps Realizing third-level scene self-adaptive modulation;
the first spectral reconstruction convolution layer Conv3 contains 64 standard convolution kernels with a receptive field of 1 x 1. Inputting a feature map Output 64 feature maps/>The computational process can be expressed as/>Wherein W 3 and B 3 represent the weight matrix and bias matrix, respectively, of the standard convolution kernel of the layer,/>Also denoted Relu is an activation function;
the second spectral reconstruction convolution layer Conv4 contains 64 standard convolution kernels with a receptive field of 1 x 1. Inputting a feature map Output b feature maps/>The computational process can be expressed as/>Wherein W 4 and B 4 represent the weight matrix and bias matrix, respectively, of the standard convolution kernel of the layer, which layer does not use an activation function;
And a spectral compensation layer Compense. Training sample for inputting hyperspectral image And feature map/>The predicted image O (i) is output. The computational process can be expressed as/>Wherein/>Representing a pixel-by-pixel addition operation.
S6, setting super parameters and initializing the weight and bias of each convolution layer of the modularized scene self-adaptive convolution neural network.
The super parameter is a preset fixed value parameter, and the fixed value parameter comprises a learning rate, iteration times, the number of input samples and the like.
S7, based on the training data set, acquiring a training sample with low spatial resolution, inputting the training sample into a modularized scene self-adaptive convolutional neural network, and performing forward propagation through the network to obtain a predicted image;
s8, selecting an average absolute error as a loss function, calculating an error value between the predicted image and the high spatial resolution reference image, minimizing the error value by using a gradient-based optimization algorithm, and iteratively updating the weight and bias of the modularized scene self-adaptive convolutional neural network. Repeating S7 to S8 when the error value does not converge to the minimum value;
The average absolute error loss function expression selected in the step 8 is:
Wherein phi represents the input-output mapping relation of the modularized scene self-adaptive convolutional neural network, theta represents the weight and bias of the network, N p represents the number of training samples input during each round of iterative optimization, and I F represents the Frobenius norm.
S9, when the error value converges to the minimum value, obtaining and storing the optimal weight and bias of the modularized scene self-adaptive convolutional neural network;
s10 multiplexing the modularized scene self-adaptive convolutional neural network structure, loading optimal network weight and biasing into the network structure;
S11, based on the test data set, a test sample with low spatial resolution is obtained and input into the modularized scene self-adaptive convolutional neural network loaded with the optimal weight and offset, and a hyperspectral image with high spatial resolution is output.
This example selects Houston hyperspectral images and panchromatic images from ITRES CASI-1500 sensors for verification. The hyperspectral image comprises 144 spectral channels, the spatial resolution is 30 multiplied by 30, the spatial resolution of the full-color image is 150 multiplied by 150, and the ratio of the spatial resolution to the spatial resolution of the full-color image is 1:5.
Fig. 5 (a) is a Houston hyperspectral reference image, fig. 5 (b) is a third hyperspectral image after up-sampling processing using bicubic interpolation, fig. 5 (c) is an image after processing using a non-negative matrix factorization algorithm, fig. 5 (d) is an image after processing using a bayesian algorithm, and fig. 5 (e) is an image after processing using the method described in this embodiment. As can be seen from the figure, compared with the reference image, the image after up-sampling processing by using bicubic interpolation method loses a great deal of space detail information, and has serious space distortion problem; the image processed by the non-negative matrix factorization algorithm has a good space detail reconstruction effect, but a certain degree of spectrum distortion occurs in a partial region; the image processed by the Bayesian algorithm has higher spectrum fidelity, but a more obvious space blurring phenomenon still exists in a partial area; the image processed by the method of the embodiment not only realizes the best global sharpening effect, but also effectively reduces the local distortion phenomenon, and achieves remarkable improvement in two aspects of space detail reconstruction and spectrum fidelity.
Example 2
A storage medium having stored thereon a computer program which when executed by a processor implements the hyperspectral image scene adaptive panchromatic sharpening method.
The method comprises the steps of reading a hyperspectral image and a full-color image matched with the hyperspectral image, and preprocessing;
Constructing a training data set and a testing data set based on the preprocessed hyperspectral image and full-color image;
Constructing a multi-scale feature extraction module, further obtaining a scene self-adaptive sharpening module, and constructing a modularized scene self-adaptive convolutional neural network model according to the scene self-adaptive sharpening module;
the method comprises the following steps: the multi-scale feature extraction module comprises three layers of convolution layers, wherein one aggregation convolution layer is input into the three layers of convolution layers, and each layer of convolution layer comprises a first cavity convolution layer, a second cavity convolution layer and a third cavity convolution layer;
The scene self-adaptive sharpening module comprises a multi-scale feature extraction module, an expansion convolution layer, a spatial spectrum conversion layer, two scene fine adjustment convolution layers and a scene self-adaptive convolution layer;
Initializing the weight and bias of each convolution layer of the modularized scene self-adaptive convolution neural network model, inputting a training data set into the modularized scene self-adaptive convolution neural network model, and obtaining a predicted image through forward propagation of the network to finish training;
and inputting the test data set into the trained modularized scene self-adaptive convolutional neural network model to obtain a hyperspectral image with high spatial resolution.
Example 3
An apparatus comprising a memory, a processor, and a hyperspectral image scene adaptive panchromatic sharpening method stored on the memory and executable on the processor.
The embodiments described above are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the embodiments described above, and any other changes, modifications, substitutions, combinations, and simplifications that do not depart from the spirit and principles of the present invention should be made in the equivalent manner, and are included in the scope of the present invention.
Claims (6)
1. A modularizable hyperspectral image scene adaptive panchromatic sharpening method, comprising:
Reading a hyperspectral image and a full-color image matched with the hyperspectral image, and preprocessing;
Constructing a training data set and a testing data set based on the preprocessed hyperspectral image and full-color image;
Constructing a multi-scale feature extraction module, further obtaining a scene self-adaptive sharpening module, and constructing a modularized scene self-adaptive convolutional neural network model according to the scene self-adaptive sharpening module;
the method comprises the following steps: the multi-scale feature extraction module comprises three layers of convolution layers, wherein one aggregation convolution layer is input into the three layers of convolution layers, and each layer of convolution layer comprises a first cavity convolution layer, a second cavity convolution layer and a third cavity convolution layer;
The scene self-adaptive sharpening module comprises a multi-scale feature extraction module, an expansion convolution layer, a spatial spectrum conversion layer, two scene fine adjustment convolution layers and a scene self-adaptive convolution layer;
Initializing the weight and bias of each convolution layer of the modularized scene self-adaptive convolution neural network model, inputting a training data set into the modularized scene self-adaptive convolution neural network model, and obtaining a predicted image through forward propagation of the network to finish training;
Inputting the test data set into a trained modularized scene self-adaptive convolutional neural network model to obtain a hyperspectral image with high spatial resolution;
The scene adaptive sharpening module comprises:
multi-scale feature extraction module and input feature map Output 64 feature maps/>Extracting multi-scale space characteristic information is realized;
An expanded convolution layer comprising 576 standard convolution kernels with a receptive field of 3×3, and an input feature map 576 Feature maps E (i) are output;
The space spectrum conversion layer inputs the characteristic diagram E (i) and outputs a scene self-adaptive convolution kernel K (i) with the receptive field of 64×h 2×w2 groups of 3×3;
The first scene fine tuning convolution layer comprises 64 standard convolution kernels with a receptive field of 3×3, and an input feature map Output 64 feature maps/>
A second scene fine tuning convolution layer comprising 64 standard convolution kernels with a receptive field of 3×3, and an input feature mapOutput 64 feature maps/>
Scene self-adaptive convolution layer and input feature mapAnd a scene self-adaptive convolution kernel K (i), outputting 64 feature graphs/>
Constructing a modularized scene self-adaptive convolutional neural network model according to the scene self-adaptive sharpening module, wherein the modeling scene self-adaptive convolutional neural network model comprises the following concrete steps:
A first spectrum compression convolution layer comprising 64 standard convolution kernels with receptive fields of 1×1, and inputting hyperspectral image training samples Output 64 feature maps/>
A second spectrum compression convolution layer comprising 64 standard convolution kernels with 1×1 receptive fields and an input feature mapOutput 64 feature maps/>
Splicing layer for inputting full-color image training sampleAnd feature map/>Outputting 65 feature maps C (i);
The first scene self-adaptive sharpening module inputs a feature map C (i) to realize first-stage scene self-adaptive modulation;
The second scene self-adaptive sharpening module inputs the output of the first-stage scene self-adaptive modulation and realizes the second-stage scene self-adaptive modulation;
The third scene self-adaptive sharpening module inputs the output of the second stage scene self-adaptive modulation and outputs 64 feature maps Realizing third-level scene self-adaptive modulation;
A first spectrum reconstruction convolution layer comprising 64 standard convolution kernels with a receptive field of 1×1, and an input feature map Output 64 feature maps/>
A second spectrum reconstruction convolution layer comprising 64 standard convolution kernels with a receptive field of 1×1, and an input feature mapOutput b feature maps/>
The spectrum compensation layer inputs a hyperspectral image training sampleAnd feature map/>Outputting a predicted image O (i);
the weight and bias of each convolution layer of the initialization modularized scene self-adaptive convolution neural network model are input into the modularized scene self-adaptive convolution neural network model, a predicted image is obtained through forward propagation of the network, and training is completed, specifically:
presetting fixed value parameters, wherein the fixed value parameters comprise learning rate, iteration times and input sample number, and initializing weights and offsets of all convolution layers of the modularized scene self-adaptive convolution neural network model;
acquiring a training sample with low spatial resolution from a training data set, inputting the training sample into a modularized scene self-adaptive convolutional neural network model, and obtaining a predicted image through forward propagation of a network;
Selecting an average absolute error as a loss function, calculating an error value between a predicted image and a high spatial resolution reference image, minimizing the error value by using a gradient-based optimization algorithm, and iteratively updating the weight and bias of the modularized scene self-adaptive convolutional neural network;
When the error value converges to the minimum value, obtaining and storing the optimal weight and bias of the modularized scene self-adaptive convolutional neural network, and obtaining a trained modularized scene self-adaptive convolutional neural network model;
the expression of the average absolute error as a loss function is:
wherein phi represents the input-output mapping relation of the modularized scene self-adaptive convolutional neural network, theta represents the weight and bias of the network, N p represents the number of training samples input during each round of iterative optimization, I F represents the Frobenius norm, Samples were trained for hyperspectral images.
2. The method of claim 1, wherein the spatial dimensions of the hyperspectral image and the panchromatic image satisfy the following relationship:
r＝h2/h1＝w2/w1
where r represents the ratio of the spatial resolution of the full-color image to the hyperspectral image, h 1 and w 1 represent the height and width of the hyperspectral image, respectively, and h 2 and w 2 represent the height and width of the full-color image, respectively.
3. The hyperspectral image scene adaptive panchromatic sharpening method of claim 1, wherein the preprocessing includes:
Performing low-pass filtering on the hyperspectral image and the full-color image by using a smoothing filter with adaptive frequency response to obtain a first hyperspectral image and a first full-color image;
downsampling the first hyperspectral image and the first panchromatic image to obtain a second hyperspectral image and a second panchromatic image with r times of the degradation of the spatial resolution;
And up-sampling the second hyperspectral image by using a polynomial interpolation method to obtain a third hyperspectral image with r times of spatial resolution improvement.
4. The hyperspectral image scene adaptive panchromatic sharpening method of claim 1, wherein in the multiscale feature extraction module, the three first hole convolution layers have different hole ratios, the three second hole convolution layers have different hole ratios, and the three third hole convolution layers have different hole ratios.
5. A storage medium having stored thereon a computer program which, when executed by a processor, implements the hyperspectral image scene adaptive panchromatic sharpening method of any one of claims 1 to 4.
6. A hyperspectral image scene adaptive panchromatic sharpening device comprising a memory, a processor, and a hyperspectral image scene adaptive panchromatic sharpening method stored on the memory and executable on the processor as recited in any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210293380.7A CN114638762B (en) | 2022-03-24 | 2022-03-24 | Modularized hyperspectral image scene self-adaptive panchromatic sharpening method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210293380.7A CN114638762B (en) | 2022-03-24 | 2022-03-24 | Modularized hyperspectral image scene self-adaptive panchromatic sharpening method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114638762A CN114638762A (en) | 2022-06-17 |
| CN114638762B true CN114638762B (en) | 2024-05-24 |
Family
ID=81948897
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210293380.7A Active CN114638762B (en) | 2022-03-24 | 2022-03-24 | Modularized hyperspectral image scene self-adaptive panchromatic sharpening method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114638762B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109727207A (en) * | 2018-12-06 | 2019-05-07 | 华南理工大学 | High spectrum image sharpening method based on Forecast of Spectra residual error convolutional neural networks |
| CN109859110A (en) * | 2018-11-19 | 2019-06-07 | 华南理工大学 | The panchromatic sharpening method of high spectrum image of control convolutional neural networks is tieed up based on spectrum |
| CN110533077A (en) * | 2019-08-01 | 2019-12-03 | 南京理工大学 | Form adaptive convolution deep neural network method for classification hyperspectral imagery |
| CN111008936A (en) * | 2019-11-18 | 2020-04-14 | 华南理工大学 | Multispectral image panchromatic sharpening method |
| CN113362223A (en) * | 2021-05-25 | 2021-09-07 | 重庆邮电大学 | Image super-resolution reconstruction method based on attention mechanism and two-channel network |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8203114B2 (en) * | 2009-05-14 | 2012-06-19 | Raytheon Company | Adaptive spatial-spectral processing (ASSP) |
-
2022
- 2022-03-24 CN CN202210293380.7A patent/CN114638762B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109859110A (en) * | 2018-11-19 | 2019-06-07 | 华南理工大学 | The panchromatic sharpening method of high spectrum image of control convolutional neural networks is tieed up based on spectrum |
| CN109727207A (en) * | 2018-12-06 | 2019-05-07 | 华南理工大学 | High spectrum image sharpening method based on Forecast of Spectra residual error convolutional neural networks |
| CN110533077A (en) * | 2019-08-01 | 2019-12-03 | 南京理工大学 | Form adaptive convolution deep neural network method for classification hyperspectral imagery |
| CN111008936A (en) * | 2019-11-18 | 2020-04-14 | 华南理工大学 | Multispectral image panchromatic sharpening method |
| CN113362223A (en) * | 2021-05-25 | 2021-09-07 | 重庆邮电大学 | Image super-resolution reconstruction method based on attention mechanism and two-channel network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114638762A (en) | 2022-06-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110415199B (en) | Multispectral remote sensing image fusion method and device based on residual learning | |
| CN111192200A (en) | Image super-resolution reconstruction method based on fusion attention mechanism residual error network | |
| Chen et al. | Content-aware scalable deep compressed sensing | |
| CN112215755B (en) | Image super-resolution reconstruction method based on back projection attention network | |
| US20160180502A1 (en) | Method for upscaling an image and apparatus for upscaling an image | |
| CN112488978A (en) | Multi-spectral image fusion imaging method and system based on fuzzy kernel estimation | |
| CN111553867B (en) | Image deblurring method and device, computer equipment and storage medium | |
| CN111008936B (en) | Multispectral image panchromatic sharpening method | |
| CN111861886B (en) | Image super-resolution reconstruction method based on multi-scale feedback network | |
| CN112801904B (en) | Hybrid degraded image enhancement method based on convolutional neural network | |
| CN111353939B (en) | Image super-resolution method based on multi-scale feature representation and weight sharing convolution layer | |
| CN112132741A (en) | Conversion method and system of face photo image and sketch image | |
| CN113744136A (en) | Image super-resolution reconstruction method and system based on channel constraint multi-feature fusion | |
| WO2022100490A1 (en) | Methods and systems for deblurring blurry images | |
| CN111402138A (en) | Image super-resolution reconstruction method of supervised convolutional neural network based on multi-scale feature extraction fusion | |
| CN112669248A (en) | Hyperspectral and panchromatic image fusion method based on CNN and Laplacian pyramid | |
| CN117252761A (en) | Cross-sensor remote sensing image super-resolution enhancement method | |
| Zhang et al. | Deformable and residual convolutional network for image super-resolution | |
| CN115760814A (en) | Remote sensing image fusion method and system based on double-coupling deep neural network | |
| Zhao et al. | Deep pyramid generative adversarial network with local and nonlocal similarity features for natural motion image deblurring | |
| CN114638762B (en) | Modularized hyperspectral image scene self-adaptive panchromatic sharpening method | |
| Mandal et al. | Employing structural and statistical information to learn dictionary (s) for single image super-resolution in sparse domain | |
| CN109064435B (en) | Gram-Schmdit fusion rapid processing method based on multispectral image | |
| CN114638761A (en) | Hyperspectral image panchromatic sharpening method, device and medium | |
| CN113362338B (en) | Rail segmentation method, device, computer equipment and rail segmentation processing system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |