CN113284067A - Hyperspectral panchromatic sharpening method based on depth detail injection network - Google Patents
Hyperspectral panchromatic sharpening method based on depth detail injection network Download PDFInfo
- Publication number
- CN113284067A CN113284067A CN202110602214.6A CN202110602214A CN113284067A CN 113284067 A CN113284067 A CN 113284067A CN 202110602214 A CN202110602214 A CN 202110602214A CN 113284067 A CN113284067 A CN 113284067A
- Authority
- CN
- China
- Prior art keywords
- image
- hyperspectral
- formula
- residual error
- shallow
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000002347 injection Methods 0.000 title claims abstract description 20
- 239000007924 injection Substances 0.000 title claims abstract description 20
- 230000004927 fusion Effects 0.000 claims abstract description 30
- 230000006870 function Effects 0.000 claims description 19
- 238000005070 sampling Methods 0.000 claims description 18
- 238000012360 testing method Methods 0.000 claims description 18
- 230000008569 process Effects 0.000 claims description 11
- 238000007781 pre-processing Methods 0.000 claims description 6
- 238000012549 training Methods 0.000 claims description 6
- 238000012795 verification Methods 0.000 claims description 6
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical compound C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 claims description 3
- 238000012935 Averaging Methods 0.000 claims description 3
- 230000004913 activation Effects 0.000 claims description 3
- 238000000605 extraction Methods 0.000 abstract description 5
- 230000000694 effects Effects 0.000 abstract description 2
- 238000007499 fusion processing Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 15
- 230000003595 spectral effect Effects 0.000 description 15
- 238000011156 evaluation Methods 0.000 description 8
- 239000011159 matrix material Substances 0.000 description 5
- 238000000354 decomposition reaction Methods 0.000 description 4
- 238000013527 convolutional neural network Methods 0.000 description 3
- 238000012847 principal component analysis method Methods 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 239000002131 composite material Substances 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 238000011426 transformation method Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F18/2135—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on approximation criteria, e.g. principal component analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10052—Images from lightfield camera
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A40/00—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production
- Y02A40/10—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production in agriculture
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Multimedia (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a hyperspectral panchromatic sharpening method based on a depth detail injection network, which comprises the steps of firstly selecting two hyperspectral image data sets respectively covering an indoor scene and an outdoor scene, namely a Cave data set and a PaviaCenter data set; the hyperspectral image with low resolution is up-sampled and combined with the full-color image, and the combined image is input into the convolutional layer to extract shallow layer characteristics; the extracted shallow features are sent to the convolutional layer again for further extraction; then inputting the shallow layer characteristics extracted for the second time into a residual error dense block network; finally, performing global feature fusion on all residual error dense blocks to obtain the hierarchical features of the combined image; performing residual error operation on the shallow layer characteristics and the hierarchical characteristics; and finally, carrying out convolution to obtain a fusion result of the hyperspectral panchromatic sharpening method based on the depth detail injection network. The invention solves the problem of limited fusion effect caused by insufficient detail extraction in the hyperspectral fusion process in the prior art.
Description
Technical Field
The invention belongs to the technical field of remote sensing images, and particularly relates to a hyperspectral panchromatic sharpening method based on a depth detail injection network.
Background
The hyperspectral image is three-dimensional cubic data containing spatial information and spectral information. In the spatial domain, each band can be represented as a scene description of the corresponding wavelength. Spectrally, any pixel forms a high-resolution spectral curve under the response of hundreds of different wavelengths, and can be used for distinguishing different material properties, namely spectral identification. The unique spectrum identification of the hyperspectral remote sensing image enables the hyperspectral remote sensing image to be widely applied to the fields of military rescue, environmental monitoring and the like. However, due to the limitation of hardware, it is difficult for the spectral imager to obtain a remote sensing image with both high spatial resolution and high spectral resolution. In the practical application process, more data corresponding to different loads are obtained on the remote sensing platform. The hyperspectral image obtained by the imaging spectrometer contains abundant spectral information but has lower spatial resolution, and the remote sensing image obtained by the panchromatic camera has higher spatial resolution but less wave bands. The hyperspectral panchromatic sharpening is to obtain a hyperspectral image with both high spatial resolution and high spectral resolution by fusing a hyperspectral image and a panchromatic image.
The existing hyperspectral full-color fusion method can be divided into component replacement, multiresolution analysis, matrix decomposition-based and deep learning-based methods. The component replacement method is to use a full-color image to replace one component after the inverse transformation of the hyperspectral image. The multiresolution analysis method injects spatial detail information of the panchromatic image into the up-sampled hyperspectral image. The matrix decomposition-based method is to model the process of generating a panchromatic image and an input hyperspectral image by decomposing a target hyperspectral image into spectral bases and corresponding subspace coefficients. In order to solve the modeling function, a priori knowledge of a given image is required, and the given priori knowledge of the image often cannot completely describe the characteristics of the image, so that the detail of the fused image is distorted.
The hyperspectral panchromatic sharpening method based on deep learning is characterized in that a hyperspectral image and a panchromatic image are simultaneously input into a network, and input and output are learned, namely the mapping relation between ideal hyperspectral images. The learned relationship is generalized to the rest of the images to obtain a high spatial resolution hyperspectral image.
Disclosure of Invention
The invention aims to provide a hyperspectral panchromatic sharpening method based on a depth detail injection network, and solves the problem that in the prior art, the fusion effect is limited due to insufficient detail extraction in a hyperspectral fusion process.
The technical scheme adopted by the invention is that the hyperspectral panchromatic sharpening method based on the depth detail injection network is implemented according to the following steps:
step 2, performing up-sampling on the low-resolution hyperspectral image in the data set in the step 1, combining the hyperspectral image with the full-color image, and inputting the combined image into the convolutional layer to extract the shallow feature of the combined image;
3, sending the shallow layer features extracted in the step 2 to the convolutional layer again, and further extracting the shallow layer features; then inputting the shallow layer characteristics extracted for the second time into a residual error dense block network; finally, performing primary global feature fusion on all residual error dense blocks to obtain the hierarchical features of the combined image;
step 4, performing residual error operation on the shallow layer characteristics obtained in the step 2 and the hierarchical characteristics obtained in the step 3; and finally, carrying out convolution operation once to obtain a fusion result of the hyperspectral panchromatic sharpening method based on the depth detail injection network.
The present invention is also characterized in that,
the step 1 is as follows:
step 1.1, adopting a Cave data set to represent an indoor scene, and adopting a Pavia Center data set to represent an outdoor scene; the original hyperspectral image in the data set is used as a reference image, the simulated low-resolution hyperspectral image is obtained by down-sampling the reference image, and the simulated high-resolution panchromatic image is obtained by averaging the third dimension of the reference image;
step 1.2, respectively dividing a training set, a verification set and a test set for the two data sets, namely a Cave data set and a Pavia Center data set, wherein the number of images in each training set is 80% of the whole image data set, the number of images in each test set is 10% of the whole image data set, and the number of images in each verification set is 10% of the whole image data set;
and 1.3, dividing the data set, performing data preprocessing, and uniformly adjusting the size of the image to 64 multiplied by 64.
The step 2 is as follows:
step 2.1, performing up-sampling on the low-resolution hyperspectral image obtained in the step 1.1, as shown in formula (1):
in the formula: HSbRepresenting a low-resolution hyperspectral image, wherein b is 1,2,3, and n is the number of wave bands of the hyperspectral image; f. ofupRefers to a bicubic interpolation function of relative multiple of a hyperspectral image with low spatial resolution,representing an upsampled hyperspectral image;
step 2.2, performing combined operation on the high-resolution full-color image obtained in the step 1.1 and the hyperspectral image obtained in the step 2.1 to obtain a combined image with n +1 wave band numbers, wherein the combined process is shown as a formula (2):
in the formula: b ═ 1,2,3,. n, n + 1; as indicated by a full color image andthe PAN represents a full-color image,representing a joint image;
and 2.3, extracting shallow features of the combined image by using a 3 × 3 convolutional layer in the preprocessing step, wherein the formula (3) is as follows:
F-1=fCONV(HSin), (3)
in the formula: f. ofCONVRepresents a convolution operation; f-1And the shallow feature of the combined image is represented and simultaneously used as an input for extracting the hierarchical feature, and the global residual error learning is also used.
Step 3, the shallow features of the combined image extracted in the step 2 are sent to the convolutional layer again for re-extraction; then inputting the shallow layer characteristics extracted for the second time into a residual error dense block network; and finally, performing one-time global feature fusion on all residual error dense blocks, which specifically comprises the following steps:
step 3.1, performing convolution operation of 3 × 3 on the shallow layer feature extracted in step 2, as shown in formula (4):
F0=fCONV(F-1), (4)
in the formula: f0Representing shallow features extracted by performing a second convolution operation on the joint image;
step 3.2, assuming that the residual error dense blocks RDBs include D residual error dense blocks RDB and the D-th residual error dense block RDB, as shown in formula (5), in one residual error dense block RDB, first, the state of the previous residual error dense block RDB needs to be transferred to each convolution layer and rectification linear unit in the current residual error dense block RDB, as shown in formula (6):
in the formula:a complex function representing the d-th residual dense block RDB;representing the weight of the ith layer in the residual dense block RDB;an ith convolution layer representing a d-th residual dense block RDB; i ═ 1,2,3,. ·, I, ReLu stands for activation function;
step 3.3, secondly, the state of the previous residual error dense block RDB and the state of the whole convolution layer in the current residual error dense block RDB are connected and adaptively fused together through convolution of 1 × 1, as shown in formula (1):
in the formula:represents the 1 × 1 convolutional layer function in the d RDB, Fd,LFLocal features representing the d-th residual dense block RDB;
step 3.4, finally, a formula (8) is obtained by summing the local features of the previous residual error dense block RDB and the d-th residual error dense block RDB obtained in step 3.3, as follows:
Fd=Fd-1+Fd,LF, (8)
in the formula: fd-1Represents the d-1 th residual dense block RDB, FdRepresents the d-th residual dense block RDB;
step 3.5, performing global feature fusion on all residual error dense blocks RDBs obtained in the step 3.4, namely adaptively fusing features extracted by all RDBs together to obtain global features, wherein the process is defined as formula (9):
FGFF=HGFF([F1......,FD]), (9)
in the formula: [ f ] of1......,FD]Is the result of the 1 st RDB through the D < th > RDB, HGFFIs a function of the global feature fusion, FGFFRepresenting the hierarchical features of the joint image.
Step 4, carrying out residual error operation on the shallow layer characteristics obtained in the step 2.3 and the hierarchical characteristics obtained in the step 3; and finally, carrying out convolution operation once to obtain a high-resolution hyperspectral image, which specifically comprises the following steps:
the summation operation process of the shallow feature obtained in step 4.1 and step 2.3 and the hierarchical feature obtained in step 3 is shown in formula (10):
FRes=F-1+FGFF, (10)
in the formula: f-1And FGFFRespectively representing shallow and hierarchical features of the joint image, FResDense features representing the joint image;
step 4.2, performing convolution operation of 3 × 3 on the dense features of the combined image obtained in step 4.1 to obtain a high-resolution hyperspectral image with a wave band of n, as shown in formula (11):
HSfus=fconv(FRes), (11)
in the formula: fResDense features representing the joint image; HSfusRepresenting a high-resolution hyperspectral image.
The hyperspectral panchromatic sharpening method based on the depth detail injection network has the advantages that the residual error dense network is adopted to fully utilize all the hierarchical features of the input image, then the residual error dense block is used for extracting the local features, and finally the global residual error learning is used for combining the shallow features and the deep features together to obtain the hyperspectral image with high spatial resolution and high spectral resolution.
Drawings
FIG. 1 is a flow chart of a hyperspectral panchromatic sharpening method based on a depth detail injection network according to the invention;
FIG. 2 is a visual result of the Cave test set using different fusion algorithms with a sampling factor of 8 to obtain the 30 th waveband of the fusion image;
fig. 3 is a visualization of the 102 th band of the fused image obtained by the Pavia Center test set using different fusion algorithms with a sampling factor of 2.
Detailed Description
The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
The invention relates to a hyperspectral panchromatic sharpening method based on a depth detail injection network, which is implemented by the following steps in detail by combining a flow chart shown in figure 1 and figures 2-3:
the step 1 is as follows:
step 1.1, constructing a data set of a hyperspectral image, and adopting a Cave data set to represent an indoor scene and a Pavia Center data set to represent an outdoor scene; according to the Wald protocol, an original hyperspectral image in a data set is used as a reference image, a simulated low-resolution hyperspectral image is obtained by down-sampling the reference image, and a simulated high-resolution panchromatic image is obtained by averaging the third dimension of the reference image;
step 1.2, respectively dividing a training set, a verification set and a test set for the two data sets, namely a Cave data set and a Pavia Center data set, wherein the number of images in each training set is 80% of the whole image data set, the number of images in each test set is 10% of the whole image data set, and the number of images in each verification set is 10% of the whole image data set;
step 1.3, after the data set is divided, data preprocessing is carried out, and in order to guarantee feasibility of code operation, the size of the image is uniformly adjusted to be 64 x 64.
Step 2, performing up-sampling on the low-resolution hyperspectral image in the data set in the step 1, combining the hyperspectral image with the full-color image, inputting the combined image into the convolutional layer, and extracting the shallow layer feature F of the combined image-1;
Step 2, as shown in fig. 1, the low-resolution hyperspectral image in the data in step 1 is firstly up-sampled, then combined with the full-color image, and input into the convolutional layer, specifically as follows:
step 2.1, performing up-sampling on the low-resolution hyperspectral image obtained in the step 1.1, as shown in formula (1):
in the formula: HSbRepresenting a low-resolution hyperspectral image, wherein b is 1,2,3, and n is the number of wave bands of the hyperspectral image; fup refers to a bicubic interpolation function of relative multiples of a hyperspectral image with low spatial resolution,representing an upsampled hyperspectral image;
step 2.2, performing combined operation on the high-resolution full-color image obtained in the step 1.1 and the hyperspectral image obtained in the step 2.1 to obtain a combined image with n +1 wave band numbers, wherein the combined process is shown as a formula (2):
in the formula: b ═ 1,2,3,. n, n + 1; as indicated by a full color image andthe PAN represents a full-color image,representing a joint image;
and 2.3, extracting shallow features of the combined image by using a 3 × 3 convolutional layer in the preprocessing step, wherein the formula (3) is as follows:
F-1=fCONV(HSin), (3)
in the formula: f. ofCONVRepresents a convolution operation; f-1And the shallow feature of the combined image is represented and simultaneously used as an input for extracting the hierarchical feature, and the global residual error learning is also used.
Step 3, sending the shallow feature of the combined image extracted in the step 2 to the convolution layer again for extracting the shallow feature again; then inputting the shallow layer characteristics extracted for the second time into a residual error dense block network; finally, performing global feature fusion on all residual dense blocks once, as shown in fig. 1, specifically as follows:
step 3.1, performing convolution operation of 3 × 3 on the shallow layer feature extracted in step 2, as shown in formula (4):
F0=fCONV(F-1), (4)
in the formula: f0Representing shallow features extracted by performing a second convolution operation on the joint image;
step 3.2, assuming that the residual error dense blocks RDBs include D residual error dense blocks RDB and the D-th residual error dense block RDB, as shown in formula (5), in one residual error dense block RDB, first, the state of the previous residual error dense block RDB needs to be transferred to each convolution layer and rectification linear unit in the current residual error dense block RDB, as shown in formula (6):
in the formula:a complex function representing the d-th residual dense block RDB;representing the weight of the ith layer in the residual dense block RDB;an ith convolution layer representing a d-th residual dense block RDB; i ═ 1,2,3,. ·, I, ReLu stands for activation function;
step 3.3, secondly, the state of the previous residual error dense block RDB and the state of the whole convolution layer in the current residual error dense block RDB are connected and adaptively fused together through convolution of 1 × 1, as shown in formula (1):
in the formula:represents the 1 × 1 convolutional layer function in the d RDB, Fd,LFLocal features representing the d-th residual dense block RDB;
step 3.4, finally, a formula (8) is obtained by summing the local features of the previous residual error dense block RDB and the d-th residual error dense block RDB obtained in step 3.3, as follows:
Fd=Fd-1+Fd,LF, (8)
in the formula: fd-1Represents the d-1 th residual dense block RDB, FdRepresents the d-th residual dense block RDB;
step 3.5, performing global feature fusion on all residual error dense blocks RDBs obtained in the step 3.4, namely adaptively fusing features extracted by all RDBs together to obtain global features, wherein the global feature fusion comprises convolution operations of 1 × 1 and 3 × 3, and the convolution of 1 × 1 is to fuse a series of features; the 3x3 convolution is prepared for further feature extraction for the next global residual learning. The above process is defined as formula (9):
FGFF=HGFF([f1…...,FD]), (9)
in the formula: [ F ]1......,FD]Is the result of the 1 st RDB through the D < th > RDB, HGFFIs a function of the global feature fusion, FGFFRepresenting the hierarchical features of the joint image.
Step 4, performing residual error operation on the shallow layer characteristics obtained in the step 2 and the hierarchical characteristics obtained in the step 3; and finally, carrying out convolution operation once to obtain a fusion result of the hyperspectral panchromatic sharpening method based on the depth detail injection network.
Step 4, as shown in fig. 1, performing residual error operation on the shallow feature obtained in step 2.3 and the hierarchical feature obtained in step 3; and finally, carrying out convolution operation once to obtain a high-resolution hyperspectral image, which specifically comprises the following steps:
the summation operation process of the shallow feature obtained in step 4.1 and step 2.3 and the hierarchical feature obtained in step 3 is shown in formula (10):
FRes=F-1+FGFF, (10)
in the formula: f-1And FGFFRespectively representing shallow and hierarchical features of the joint image, FResDense features representing the joint image;
step 4.2, performing convolution operation of 3 × 3 on the dense features of the combined image obtained in step 4.1 to obtain a high-resolution hyperspectral image with a wave band of n, as shown in formula (11):
HSfus=fconv(FRes), (11)
in the formula: fResDense features representing the joint image; HSfusRepresenting a high-resolution hyperspectral image.
In addition, in order to perform comprehensive evaluation on the fused image, subjective evaluation and objective evaluation are adopted. Objective evaluation indices include Cross-Correlation (CC), Spectral Angle Mapping (SAM), Root-Mean-Squared Error (RMSE), Relative global dimensional composite Error (Erreal Relative Global evaluation nelle De Synth se, ERGAS), Peak Signal Noise Ratio (PSNR), and Structural Similarity (SSIM).
The invention herein is compared to the following six algorithms: bicubic (bicubic), a self-adaptive Schmidt orthogonal variation method (GSA), a Laplacian Pyramid method based on a Generalized modulation transfer function (MTF-Generalized Laplacian Pyramid, MTF-GLP), a Guided filter principal component analysis method (GFPCA), Coupled nonnegative matrix decomposition (mf cnc), and a panchromatic sharpening method based on a convolutional neural network (PNN).
For subjective evaluation, Flowers images in the Cave test set are selected, sampling factors are 8, and the images are presented by using the thirtieth wave band of the fused image. Fig. 2(a) is a reference diagram of Flowers, fig. 2(b) is a result diagram of a bicubic (bicubic) method of performing bicubic interpolation on Flowers, fig. 2(c) is a result diagram of a GSA (GSA) method of performing adaptive schmitt orthogonal transformation on Flowers, fig. 2(d) is a result diagram of a laplacian pyramid method (MTF-GLP) based on a generalized modulation transfer function on Flowers, fig. 2(e) is a result diagram of a guided filter principal component analysis method (GFPCA) on Flowers, fig. 2(f) is a result diagram of a coupled nonnegative matrix decomposition method (CNMF) on Flowers, fig. 2(g) is a result diagram of a PNN (PNN) method of performing convolutional neural network-based panchromatic sharpening on Flowers, and fig. 2(h) is a result diagram of a hyperspectral panchromatic method of performing depth-detail injection network-based on Flowers. As can be seen from fig. 2, the hyperspectral image of Bicubic appears with a large amount of blur; the GSA fused hyperspectral image has certain spectral distortion; the MTF-GLP fused hyperspectral image loses some spatial details; the GFPCA fused hyperspectral image is fuzzy in some detail areas; the CMNF and PNN fused hyperspectral image can be well maintained on spectral information and spatial information; the fused image of the algorithm provided by the invention effectively maintains the spectral information and enhances the spatial information. The average values for the Cave dataset using various fusion algorithms with sampling factors of 2, 4 and 8 are listed in table 1, with the optimal values being shown in bold. From the experimental results, it can be seen that the values of CC, PSNR and SSIM are the largest and the values of SAM, RMSE and ERGAS are the smallest for the present algorithm, regardless of the sampling factor.
TABLE 1 mean values of various fusion algorithms for Cave datasets using different sampling factors
Fig. 3 is a visualization result of 102 th band of a fused image obtained by a Pavia Center test set using different fusion algorithms with a sampling factor of 2, fig. 3(a) is a reference diagram of the test set, fig. 3(b) is a result diagram of a bicubic method of bicubic interpolation performed on the test set, fig. 3(c) is a result diagram of an adaptive schmidt orthogonal transformation method (GSA) performed on the test set, fig. 3(d) is a result diagram of a laplacian pyramid method based on a generalized modulation transfer function (MTF-GLP) performed on the test set, fig. 3(e) is a result diagram of a guided filtering principal component analysis method (GFPCA) performed on the test set, fig. 3(f) is a result diagram of a coupled non-negative matrix factorization method (CNMF) performed on the test set, fig. 3(g) is a result diagram of a full color sharpening method based on a convolutional neural network (PNN) performed on the test set, and fig. 3(h) is a hyperspectral method of a full color sharpening method based on depth injection network performed on the test set And (5) fruit pictures. The results of Bicubic appear to be a blurring of the patch in detail; the results of GSA, CMNF and PNN fusion are somewhat detailed lost; slight spectral distortion appears as a result of MTF-GLP fusion, and partial edge information is lost; the result of GFPCA fusion appears unclear in detail information and the image is dark overall. Compared to the above algorithms, the algorithms presented herein are enhanced in both spectral and spatial aspects. Table 2 is an objective evaluation index for various fusion algorithms using various sampling factors for the Pavia dataset. As can be seen from the table, each evaluation index of the algorithm herein is better than the optimum value and the value of PNN in the conventional method.
Table 2 mean values of various fusion algorithms using different sampling factors for the PaviaCenter dataset
Claims (5)
1. The hyperspectral panchromatic sharpening method based on the depth detail injection network is characterized by being implemented according to the following steps:
step 1, selecting data sets of two hyperspectral images, wherein the data sets of the two hyperspectral images respectively cover an indoor scene and an outdoor scene, the indoor scene is represented by a Cave data set, and the outdoor scene is represented by a PaviaCenter data set;
step 2, performing up-sampling on the low-resolution hyperspectral image in the data set in the step 1, combining the hyperspectral image with the full-color image, and inputting the combined image into the convolutional layer to extract the shallow feature of the combined image;
3, sending the shallow layer features extracted in the step 2 to the convolutional layer again, and further extracting the shallow layer features; then inputting the shallow layer characteristics extracted for the second time into a residual error dense block network; finally, performing primary global feature fusion on all residual error dense blocks to obtain the hierarchical features of the combined image;
step 4, performing residual error operation on the shallow layer characteristics obtained in the step 2 and the hierarchical characteristics obtained in the step 3; and finally, carrying out convolution operation once to obtain a fusion result of the hyperspectral panchromatic sharpening method based on the depth detail injection network.
2. The hyperspectral panchromatic sharpening method based on the depth detail injection network according to claim 1, wherein the step 1 is as follows:
step 1.1, adopting a Cave data set to represent an indoor scene, and adopting a Pavia Center data set to represent an outdoor scene; the original hyperspectral image in the data set is used as a reference image, the simulated low-resolution hyperspectral image is obtained by down-sampling the reference image, and the simulated high-resolution panchromatic image is obtained by averaging the third dimension of the reference image;
step 1.2, respectively dividing a training set, a verification set and a test set for the two data sets, namely a Cave data set and a Pavia Center data set, wherein the number of images in each training set is 80% of the whole image data set, the number of images in each test set is 10% of the whole image data set, and the number of images in each verification set is 10% of the whole image data set;
and 1.3, dividing the data set, performing data preprocessing, and uniformly adjusting the size of the image to 64 multiplied by 64.
3. The hyperspectral panchromatic sharpening method based on the depth detail injection network according to claim 2, wherein the step 2 is as follows:
step 2.1, performing up-sampling on the low-resolution hyperspectral image obtained in the step 1.1, as shown in formula (1):
in the formula, HSbRepresenting a low-resolution hyperspectral image, b is 1,2,3, …, n, wherein n is the number of wave bands of the hyperspectral image; f. ofupRefers to a bicubic interpolation function of relative multiple of a hyperspectral image with low spatial resolution,representing an upsampled hyperspectral image;
step 2.2, performing combined operation on the high-resolution full-color image obtained in the step 1.1 and the hyperspectral image obtained in the step 2.1 to obtain a combined image with n +1 wave band numbers, wherein the combined process is shown as a formula (2):
wherein B is 1,2,3, …, n, n + 1; as indicated by a full color image andcascaded operation of (5), PAN representing full colorThe image is a picture of a person to be imaged,representing a joint image;
and 2.3, extracting shallow features of the combined image by using a 3 × 3 convolutional layer in the preprocessing step, wherein the formula (3) is as follows:
F-1=fCONV(HSin), (3)
in the formula fCONVRepresents a convolution operation; f-1And the shallow feature of the combined image is represented and simultaneously used as an input for extracting the hierarchical feature, and the global residual error learning is also used.
4. The hyperspectral panchromatic sharpening method based on the depth detail injection network is characterized in that the step 3 is used for sending the shallow features of the combined image extracted in the step 2 to the convolution layer again for extracting the shallow features again; then inputting the shallow layer characteristics extracted for the second time into a residual error dense block network; and finally, performing one-time global feature fusion on all residual error dense blocks, which specifically comprises the following steps:
step 3.1, performing convolution operation of 3 × 3 on the shallow layer feature extracted in step 2, as shown in formula (4):
F0=fCONV(F-1), (4)
in the formula: f0Representing shallow features extracted by performing a second convolution operation on the joint image;
step 3.2, assuming that the residual error dense blocks RDBs include D residual error dense blocks RDB and the D-th residual error dense block RDB, as shown in formula (5), in one residual error dense block RDB, first, the state of the previous residual error dense block RDB needs to be transferred to each convolution layer and rectification linear unit in the current residual error dense block RDB, as shown in formula (6):
in the formula:a complex function representing the d-th residual dense block RDB;representing the weight of the ith layer in the residual dense block RDB;an ith convolution layer representing a d-th residual dense block RDB; i ═ 1,2,3, …, I, ReLu, for activation functions;
step 3.3, secondly, the state of the previous residual error dense block RDB and the state of the whole convolution layer in the current residual error dense block RDB are connected and adaptively fused together through convolution of 1 × 1, as shown in formula (1):
in the formula:represents the 1 × 1 convolutional layer function in the d RDB, Fd,LFLocal features representing the d-th residual dense block RDB;
step 3.4, finally, a formula (8) is obtained by summing the local features of the previous residual error dense block RDB and the d-th residual error dense block RDB obtained in step 3.3, as follows:
Fd=Fd-1+Fd,LF, (8)
in the formula, Fd-1Represents the d-1 th residual dense block RDB, FdRepresents the d-th residual dense block RDB;
step 3.5, performing global feature fusion on all residual error dense blocks RDBs obtained in the step 3.4, namely adaptively fusing features extracted by all RDBs together to obtain global features, wherein the process is defined as formula (9):
FGFF=HGFF([F1……,FD]), (9)
in the formula: [ F ]1 … …,FD]Is the result of the 1 st RDB through the D < th > RDB, HGFFIs a function of the global feature fusion, FGFFRepresenting the hierarchical features of the joint image.
5. The hyperspectral panchromatic sharpening method based on the depth detail injection network according to claim 4, wherein the step 4 is to perform residual operation on the shallow feature obtained in the step 2.3 and the hierarchical feature obtained in the step 3; and finally, carrying out convolution operation once to obtain a high-resolution hyperspectral image, which specifically comprises the following steps:
the summation operation process of the shallow feature obtained in step 4.1 and step 2.3 and the hierarchical feature obtained in step 3 is shown in formula (10):
FRes=F-1+FGFF, (10)
in the formula, F-1And FGFFRespectively representing shallow and hierarchical features of the joint image, FResDense features representing the joint image;
step 4.2, performing convolution operation of 3 × 3 on the dense features of the combined image obtained in step 4.1 to obtain a high-resolution hyperspectral image with a wave band of n, as shown in formula (11):
HSfus=fconv(FRes), (11)
in the formula: fResDense features representing the joint image; HSfusRepresenting a high-resolution hyperspectral image.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110602214.6A CN113284067B (en) | 2021-05-31 | 2021-05-31 | Hyperspectral panchromatic sharpening method based on depth detail injection network |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110602214.6A CN113284067B (en) | 2021-05-31 | 2021-05-31 | Hyperspectral panchromatic sharpening method based on depth detail injection network |
Publications (2)
Publication Number | Publication Date |
---|---|
CN113284067A true CN113284067A (en) | 2021-08-20 |
CN113284067B CN113284067B (en) | 2024-02-09 |
Family
ID=77282855
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110602214.6A Active CN113284067B (en) | 2021-05-31 | 2021-05-31 | Hyperspectral panchromatic sharpening method based on depth detail injection network |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN113284067B (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114897714A (en) * | 2022-04-15 | 2022-08-12 | 华南理工大学 | Hyperspectral image sharpening method based on dual-scale fusion network |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140301659A1 (en) * | 2013-04-07 | 2014-10-09 | Bo Li | Panchromatic Sharpening Method of Spectral Image Based on Fusion of Overall Structural Information and Spatial Detail Information |
CN109727207A (en) * | 2018-12-06 | 2019-05-07 | 华南理工大学 | High spectrum image sharpening method based on Forecast of Spectra residual error convolutional neural networks |
CN109903255A (en) * | 2019-03-04 | 2019-06-18 | 北京工业大学 | A kind of high spectrum image Super-Resolution method based on 3D convolutional neural networks |
CN111127374A (en) * | 2019-11-22 | 2020-05-08 | 西北大学 | Pan-sharing method based on multi-scale dense network |
AU2020100200A4 (en) * | 2020-02-08 | 2020-06-11 | Huang, Shuying DR | Content-guide Residual Network for Image Super-Resolution |
-
2021
- 2021-05-31 CN CN202110602214.6A patent/CN113284067B/en active Active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140301659A1 (en) * | 2013-04-07 | 2014-10-09 | Bo Li | Panchromatic Sharpening Method of Spectral Image Based on Fusion of Overall Structural Information and Spatial Detail Information |
CN109727207A (en) * | 2018-12-06 | 2019-05-07 | 华南理工大学 | High spectrum image sharpening method based on Forecast of Spectra residual error convolutional neural networks |
CN109903255A (en) * | 2019-03-04 | 2019-06-18 | 北京工业大学 | A kind of high spectrum image Super-Resolution method based on 3D convolutional neural networks |
CN111127374A (en) * | 2019-11-22 | 2020-05-08 | 西北大学 | Pan-sharing method based on multi-scale dense network |
AU2020100200A4 (en) * | 2020-02-08 | 2020-06-11 | Huang, Shuying DR | Content-guide Residual Network for Image Super-Resolution |
Non-Patent Citations (2)
Title |
---|
张少磊;付光远;汪洪桥;赵玉清;: "基于向量总变差约束局部光谱解混的高光谱图像超分辨", 光学精密工程, no. 12 * |
雷鹏程;刘丛;唐坚刚;彭敦陆;: "分层特征融合注意力网络图像超分辨率重建", 中国图象图形学报, no. 09 * |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114897714A (en) * | 2022-04-15 | 2022-08-12 | 华南理工大学 | Hyperspectral image sharpening method based on dual-scale fusion network |
Also Published As
Publication number | Publication date |
---|---|
CN113284067B (en) | 2024-02-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Shao et al. | Remote sensing image fusion with deep convolutional neural network | |
CN110533620B (en) | Hyperspectral and full-color image fusion method based on AAE extraction spatial features | |
Zhou et al. | Pyramid fully convolutional network for hyperspectral and multispectral image fusion | |
CN110119780B (en) | Hyper-spectral image super-resolution reconstruction method based on generation countermeasure network | |
Acharya et al. | Image processing: principles and applications | |
González-Audícana et al. | A low computational-cost method to fuse IKONOS images using the spectral response function of its sensors | |
CN109272010B (en) | Multi-scale remote sensing image fusion method based on convolutional neural network | |
CN106920214B (en) | Super-resolution reconstruction method for space target image | |
Qu et al. | A dual-branch detail extraction network for hyperspectral pansharpening | |
Huang et al. | Deep hyperspectral image fusion network with iterative spatio-spectral regularization | |
CN112507997A (en) | Face super-resolution system based on multi-scale convolution and receptive field feature fusion | |
CN116152120B (en) | Low-light image enhancement method and device integrating high-low frequency characteristic information | |
CN108090872B (en) | Single-frame multispectral image super-resolution reconstruction method and system based on gradient extraction | |
Sdraka et al. | Deep learning for downscaling remote sensing images: Fusion and super-resolution | |
CN113191325B (en) | Image fusion method, system and application thereof | |
CN113793289A (en) | Multi-spectral image and panchromatic image fuzzy fusion method based on CNN and NSCT | |
Pan et al. | Structure–color preserving network for hyperspectral image super-resolution | |
Nie et al. | Unsupervised hyperspectral pansharpening by ratio estimation and residual attention network | |
Licciardi et al. | Fusion of hyperspectral and panchromatic images: A hybrid use of indusion and nonlinear PCA | |
CN113284067B (en) | Hyperspectral panchromatic sharpening method based on depth detail injection network | |
Lu et al. | Pan-sharpening by multilevel interband structure modeling | |
CN114511470B (en) | Attention mechanism-based double-branch panchromatic sharpening method | |
CN114638761B (en) | Full-color sharpening method, equipment and medium for hyperspectral image | |
Jain et al. | Multimodal image fusion employing discrete cosine transform | |
CN115861749A (en) | Remote sensing image fusion method based on window cross attention |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |