CN111192193B - Hyperspectral single-image super-resolution method based on 1-dimensional-2-dimensional convolution neural network - Google Patents
Hyperspectral single-image super-resolution method based on 1-dimensional-2-dimensional convolution neural network Download PDFInfo
- Publication number
- CN111192193B CN111192193B CN201911176083.9A CN201911176083A CN111192193B CN 111192193 B CN111192193 B CN 111192193B CN 201911176083 A CN201911176083 A CN 201911176083A CN 111192193 B CN111192193 B CN 111192193B
- Authority
- CN
- China
- Prior art keywords
- neural network
- dimensional
- resolution
- hyperspectral
- super
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000013528 artificial neural network Methods 0.000 title claims description 68
- 230000003595 spectral effect Effects 0.000 claims abstract description 68
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 36
- 238000012549 training Methods 0.000 claims abstract description 35
- 238000012545 processing Methods 0.000 claims abstract description 28
- 230000007246 mechanism Effects 0.000 claims abstract description 23
- 238000012360 testing method Methods 0.000 claims abstract description 16
- 238000003062 neural network model Methods 0.000 claims abstract description 13
- 230000000750 progressive effect Effects 0.000 claims abstract description 9
- 238000013507 mapping Methods 0.000 claims abstract description 7
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 40
- 238000001228 spectrum Methods 0.000 claims description 25
- 230000006870 function Effects 0.000 claims description 20
- 230000004927 fusion Effects 0.000 claims description 14
- 239000011159 matrix material Substances 0.000 claims description 7
- 238000007781 pre-processing Methods 0.000 claims description 6
- 239000013598 vector Substances 0.000 claims description 6
- 238000013461 design Methods 0.000 claims description 5
- 239000000284 extract Substances 0.000 claims description 5
- 238000005070 sampling Methods 0.000 claims description 4
- 230000004913 activation Effects 0.000 claims description 3
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 claims description 3
- OAICVXFJPJFONN-UHFFFAOYSA-N Phosphorus Chemical compound [P] OAICVXFJPJFONN-UHFFFAOYSA-N 0.000 claims description 2
- 238000004364 calculation method Methods 0.000 abstract description 3
- 238000013135 deep learning Methods 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 4
- 238000002474 experimental method Methods 0.000 description 4
- 238000003384 imaging method Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000002708 enhancing effect Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 240000005160 Parkia speciosa Species 0.000 description 1
- 235000017333 Parkia speciosa Nutrition 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- ZZUFCTLCJUWOSV-UHFFFAOYSA-N furosemide Chemical compound C1=C(Cl)C(S(=O)(=O)N)=CC(C(O)=O)=C1NCC1=CC=CO1 ZZUFCTLCJUWOSV-UHFFFAOYSA-N 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 238000000513 principal component analysis Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000013526 transfer learning Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
The invention belongs to the technical field of hyperspectral image super-resolution processing, and discloses a hyperspectral single-image super-resolution method based on a 1-dimensional-2-dimensional convolutional neural network, wherein a 1-dimensional convolutional neural network model is built to obtain spectral information; constructing a 2-dimensional convolution neural network model based on an attention mechanism to obtain spatial information; fusing spatial and spectral information in a progressive manner; training a neural network model of a parallel structure by using a training data set, and adjusting internal parameters of the neural network model; and checking the neural network model by using the test data set, solving the average peak signal-to-noise ratio (MPSNR), the average structure similarity index (MSSIM), the average root mean square error (MRMSE) and the Spectral Angle Mapping (SAM) according to the output of the model, and evaluating the super-resolution processing performance of the neural network model. The neural network model parameter setting is simple, the implementation is easy, the calculation amount is small, and the super-resolution processing result can be obtained quickly; spectral fidelity can be more effectively achieved.
Description
Technical Field
The invention belongs to the technical field of hyperspectral image super-resolution processing, and particularly relates to a hyperspectral single-image super-resolution method based on a 1-dimensional-2-dimensional convolutional neural network.
Background
Currently, the closest prior art: the hyperspectral images are widely applied in the fields of computer vision, remote sensing and the like, but are limited by hardware equipment of the existing imaging system, and the application of the hyperspectral images is limited because the hyperspectral images with high resolution are difficult to obtain, so that the hyperspectral image super-resolution processing is widely concerned. The current hyperspectral image super-resolution processing method mainly comprises two modes: the super-resolution algorithm based on a single image and the super-resolution algorithm based on fusion of a plurality of images. Other higher spatial resolution images (e.g., panchromatic images and multispectral images) are required for image fusion based methods. The hyperspectral panchromatic sharpening method is a typical algorithm based on image fusion, and has attracted much attention on the aspect of hyperspectral image resolution enhancement. Hyperspectral sharpening methods can be divided into five categories: component replacement, multiresolution analysis, Bayesian, matrix decomposition based methods and hybrid methods. The component replacement method and the multi-resolution analysis method are relatively traditional hyperspectral panchromatic sharpening algorithms. The component replacement method generally projects a hyperspectral image into a new domain to separate spatial information and spectral information, and then replaces spatial information components in the hyperspectral image with a panchromatic image. Typical component replacement methods include principal component analysis algorithms, intensity-hue-saturation algorithms, orthogonal variation algorithms, and the like, but these algorithms tend to cause spectral distortion; typical multiresolution analysis methods are intensity modulation based on smoothing filtering, MTF-Generalized Laplace pyramid with high-pass modulation, and "a-trous" wavelet transform, among others. Although the multi-resolution analysis method can maintain the spectrum consistency, the algorithm needs a large amount of calculation and the parameter setting is very complicated. Typical methods based on matrix factorization are coupling non-negative matrix factorization and non-negative sparse coding, etc. The Bayes method is to convert the hyperspectral sharpening problem into a specific probability frame and carry out regularization processing on the hyperspectral sharpening problem by selecting proper prior distribution, wherein the prior distribution comprises Bayes sparseness, Bayes theorem, Bayes naive Gaussian prior and the like. Both bayesian and matrix factorization methods trade computational increase for spatial resolution enhancement. The hybrid method is a combination of different algorithms, such as the pilot filter PCA and several variants of PCA.
The hyper-spectral single-image super-resolution enhancement algorithm does not need prior distribution and auxiliary images. Traditional single-image super-resolution is achieved based on filtering methods, but these methods usually result in edge blurring or spectral distortion due to the fact that the inherent characteristics of the image are not considered. The regularization-based method is to use image statistical distribution as a prior, and then regularize the solution space by using a prior hypothesis, such as discrete wavelet transform algorithm based on sparsity regularization proposed by Paterl, and algorithm based on total variation regularization and low rank tensor proposed by Wang et al. Algorithms based on sub-pixel mapping and self-similarity can also be used for single-map hyper-segmentation, but they ignore the self-properties of hyperspectral images, such as correlation between spectral bands. Li et al combine sparsity and non-local self-similarity in the spatial and spectral domains, and propose an efficient algorithm that can obtain spatial and spectral information simultaneously, but these heuristic model algorithms cannot learn deep spatial spectral features due to the complexity of high-resolution image details.
The deep learning based algorithm is also applied to the problem of enhancing the resolution of the hyperspectral image. Originally, convolutional neural networks were often used in the classification problem of hyperspectral images to improve the accuracy of classification. Chen et al demonstrate that a three-dimensional convolutional neural network can extract deep spatial spectral information for classification problems. Recently, Yuan et al proposed that an algorithm based on transfer learning could reduce the above classification errors, but such an algorithm cannot be used for end-to-end network learning of spectral features and is therefore of limited application. Mei et al propose three-dimensional convolutional neural networks that can jointly encode spatial and spectral information, but such three-dimensional neural networks are not optimal algorithms. Wang et al propose a convolutional network based on depth residuals, but the learning process of this network is after upsampling and therefore takes a lot of computation. Lei et al propose a partial-global combination network, but the network parameters are too numerous to adjust through extensive training. Li et al propose a deep recursive residual network of packets to improve generalization capability. Jia et al propose connecting spatial and spectral networks in series to make full use of spatial and spectral information. zheng et al propose a starting network of separable spectra to enhance the image spatial resolution from coarse to fine in order to resolve the spectral clutter caused by the two-dimensional convolution.
The existing hyperspectral image super-resolution algorithm is still concentrated on three-dimensional convolution, but a three-dimensional convolution neural network is difficult to effectively train. Considering that the hyperspectral image can be divided into a spatial domain and a spectral domain, the hyperspectral image super-resolution problem is decomposed into enhancement of spatial resolution and fidelity of spectrum. Decomposing the three-dimensional convolution into 1-2-dimensional convolution is an effective idea, and the hyperspectral image super-resolution problem can be solved by using two parts of networks based on one-dimensional convolution and two-dimensional convolution through the idea, for example, the one-dimensional convolution can extract effective spectral features, and the two-dimensional convolution can acquire spatial information under the condition of not using complete spectral bands.
In summary, the problems of the prior art are as follows:
(1) the prior art is limited by hardware equipment of the existing imaging system, and is difficult to have a hyperspectral image with high resolution; the super-resolution algorithm based on the fusion of multiple images is difficult to obtain multiple images which are completely matched, and spectral distortion is easily caused.
(2) The traditional super-resolution algorithm based on a single image often causes edge blurring and spectrum distortion, and meanwhile, the correlation between non-local pixel points is not considered.
(3) The existing single image super-resolution algorithm based on deep learning is mainly focused on an algorithm based on a three-dimensional convolution neural network, but the three-dimensional convolution parameter setting is complex, the calculation amount is large, and effective training is difficult to obtain according to the existing hyperspectral data set.
The difficulty of solving the technical problems is as follows: the high-resolution hyperspectral image is obtained by improving hardware equipment of an imaging system, the size of a sensor needs to be increased, the size and the weight of the equipment are increased, difficulty is brought to the emission of a remote sensing satellite, and the cost is high; the difficulty of the algorithm based on the fusion of the multiple images lies in the acquisition of the multiple images; the traditional super-resolution algorithm based on a single image has the difficulty that the spatial resolution is improved, and meanwhile, the spectral fidelity is required; the difficulty of the single image super-resolution algorithm based on deep learning lies in reducing parameter setting of three-dimensional convolution and acquiring more data sets for training.
The significance of solving the technical problems is as follows: by solving the problems, the limitation of hardware equipment on the spatial resolution of the hyperspectral image can be overcome, a more ideal hyperspectral high-resolution image is obtained, and the quality of the remote sensing image is improved.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a hyperspectral single-image super-resolution method based on a 1-dimensional-2-dimensional convolutional neural network.
The invention is realized in such a way that a hyperspectral single-image super-resolution method based on a 1-dimensional-2-dimensional convolutional neural network comprises the following steps:
firstly, building a parallel 1-2 dimensional convolution neural network for acquiring spectral information and spatial information; embedding the residual block into a space network and a spectrum network; embedding an attention mechanism between residual blocks of the spatial network; building a progressive fusion network for fusing the spatial and spectral information acquired by the neural network;
secondly, preprocessing the existing hyperspectral image data set to obtain a training data set and a test data set of a neural network;
then, training a neural network, taking the low-resolution hyperspectral image and the high-resolution hyperspectral image in the training data set as input data and output data of the neural network, training parameters in the neural network until the processing error of the neural network on the training data set meets the requirement of a minimum threshold value, and stopping training;
and finally, testing the neural network, taking the low-resolution hyperspectral image in the test data set as input data of the neural network, obtaining output data of the neural network, and evaluating the super-resolution processing performance of the neural network according to the output data.
Further, the hyperspectral single-map super-resolution method based on the 1-dimensional-2-dimensional convolutional neural network specifically comprises the following steps:
the method comprises the following steps that firstly, a two-dimensional convolution neural network used for obtaining space information and a one-dimensional convolution neural network used for obtaining spectrum information are built;
secondly, embedding a residual block in the space network and the spectrum network; output of the (i + 1) th spatial residual block
Ui+1＝Fi+1(Fi)+Fi；
Wherein, Fi+1(Fi)＝vi+1βi+1(wi+1(Fi))，θi+1(·)，βi+1(. h) is the activation function of the i +1 th residual block, Wi+1For the first filter in the i +1 th residual block, vi+1A second filter in the (i + 1) th residual block; the design of the spectral residual block is similar to that of the spatial residual block;
and
is the input of the (i + 2) th spatial residual block;
thirdly, embedding an attention mechanism between residual blocks of the space network; output characteristic value of ith residual block
Conversion into three feature spaces f1、f2、f3A characteristic value of the attention mechanism is calculated,
is a weight matrix obtained by learning, the relationship of the positions p and q is:
wherein, N is m multiplied by N,
attention was paid to the output of the mechanical layer:
L＝(L1，L2，…，Lt，…LN)；
the characteristic values of the attention mechanism are:
psi is a scale factor, initially 0;
fourthly, designing an objective function loss:
loss＝ll1_loss+αls_loss；
wherein, l, m, n and r are the spectral band, height, width and up-sampling factor, | · | | purple sweet1Is 11Norm, zi，jThe spectral vectors of the ith row and the jth column in the high-spectrum high-resolution image,
representing the reconstructed spectrum vector of the same spatial position after the super-resolution processing;
fifthly, building a progressive fusion network for fusing the space and spectrum information acquired by the neural network; compressing the two spectral bands into a frame channel, and fusing in a point-by-point addition manner;
sixthly, preprocessing the existing hyperspectral image data set, obtaining a low-resolution hyperspectral image through a Gaussian filter, and dividing the hyperspectral image into a training data set and a test data set according to a certain rule;
seventhly, continuously training the network by taking minimization of the target function loss as a target, and adjusting parameters in the network; when the target function is smaller than the super-resolution processing precision requirement epsilon, stopping training to obtain an optimal super-resolution processing result;
eighthly, taking the low-resolution hyperspectral image in the test data set as input data of the neural network to obtain output data of the neural network, and evaluating the super-resolution processing performance of the neural network according to the average peak signal-to-noise ratio (MPSNR), the average structure similarity index (MSSIM), the average root mean square error (MRMSE) and the Spectrum Angle Mapping (SAM) of the output data; the index is defined as follows:
wherein S and
representing true and reconstructed images, MAX, respectivelykIs the maximum intensity of the k-th frequency band,
and muSAre respectively
And the mean value of the sum of the S,
and σSAre respectively
And the variance of the sum of the S,
is that
And S covariance, C1And C2Is two constants for improving stability, n ═ w × l is the number of pixels,<zi，z′i>representing two spectra ziAnd z'iThe dot product, | · | | non-conducting phosphor2Is represented by2And (5) carrying out norm operation.
Further, the first step is used for acquiring the spatial information and the spectral information respectively by a two-dimensional convolutional neural network and a one-dimensional convolutional neural network which are parallel, and extracting the spatial information and the spectral information respectively.
Further, the second step embeds the residual block into the spatial network and the spectral network, and the input of the i +2 th spatial residual block is
And
further, the attention mechanism of the third step is embedded between the residual blocks of the spatial network.
Further, the objective function of the fourth step is l1Norm loss function and weighted loss of spectral loss function.
Further, the fifth step is a progressive fusion network for fusing the spatial information and the spectral information, and two spectral bands are compressed into one frame channel in the fusion process and fused in a point-by-point addition manner.
Further, in the sixth step, the existing hyperspectral image is processed by a gaussian filter to obtain a low-resolution hyperspectral image, and the number of the finally obtained training data sets is greater than that of the test data sets.
Further, the neural network training process in the seventh step is to adjust parameters in the constructed neural network model so that an objective function value of the neural network is smaller than a fitting precision requirement epsilon to obtain an optimal neural network model;
and the eighth step of calculating average peak signal-to-noise ratio MPSNR, average structure similarity index MSSIM, average root mean square error MRMSE and spectral angle mapping SAM, and evaluating the performance of the neural network model.
The invention also aims to provide a hyperspectral single super-resolution processing system applying the hyperspectral single super-resolution method based on the 1-dimensional-2-dimensional convolutional neural network.
In summary, the advantages and positive effects of the invention are: the method respectively uses a one-dimensional convolution neural network and a two-dimensional convolution neural network to respectively extract the spectral information and the spatial information of the hyperspectral image; embedding a residual block in the neural network to compensate for information loss caused by input data passing through the neural network; an attention mechanism is embedded between residual blocks in a two-dimensional convolutional neural network for acquiring spatial information to consider the correlation between non-local pixel points.
The invention aims to enhance the spatial resolution of a hyperspectral single image under the condition of spectrum undistorted, provides a hyperspectral single image super-resolution algorithm based on a 1-dimensional-2-dimensional attention mechanism convolutional neural network, and performs spatial resolution enhancement on the hyperspectral single image by using the 1-dimensional-2-dimensional attention mechanism convolutional neural network. According to the method, the resolution is enhanced according to the hyperspectral image acquired by the existing imaging system, and the hyperspectral image with higher spatial resolution can be acquired without changing the existing hardware equipment.
Drawings
Fig. 1 is a flow chart of a hyperspectral single-graph super-resolution method based on a 1-dimensional-2-dimensional convolutional neural network according to an embodiment of the invention.
FIG. 2 is a flow chart of an implementation of a hyperspectral single-graph super-resolution method based on a 1-dimensional to 2-dimensional convolutional neural network according to an embodiment of the present invention.
Fig. 3 is a schematic diagram of a residual block design according to an embodiment of the present invention.
Fig. 4 is a schematic illustration of an attention mechanism design provided by an embodiment of the present invention.
FIG. 5 is a schematic diagram of a 1-D-2-D convolutional neural network based on an attention mechanism according to an embodiment of the present invention.
Fig. 6 is a comparison graph of a high spectral image experiment based on the university of parkia provided by an embodiment of the invention.
FIG. 7 is a comparison graph of a high spectral image experiment based on the Parveasian center provided by the embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Aiming at the problems in the prior art, the invention provides a hyperspectral single-image super-resolution method based on a 1-dimensional-2-dimensional convolutional neural network, and the invention is described in detail below by combining the attached drawings.
As shown in fig. 1, the hyperspectral single-image super-resolution method based on the 1-dimensional to 2-dimensional convolutional neural network provided by the embodiment of the invention includes the following steps:
s101: building a neural network for acquiring spatial information and spectral information, wherein the neural network is a two-dimensional convolution neural network and a one-dimensional convolution neural network;
s102: embedding the residual block into a space network and a spectrum network;
s103: embedding an attention mechanism between residual blocks of the spatial network;
s104: building a progressive fusion network for fusing spatial information and spectral information acquired by a neural network;
s105: preprocessing an existing hyperspectral image data set to obtain a training data set and a test data set of a neural network;
s106: training a neural network, taking the low-resolution hyperspectral image and the high-resolution hyperspectral image in the training data set as input data and output data of the neural network, training parameters in the neural network until the processing error of the neural network on the training data set meets the requirement of a minimum threshold value, and stopping training;
s107: and testing the neural network, taking the low-resolution hyperspectral image in the test data set as input data of the neural network, obtaining output data of the neural network, and evaluating the super-resolution processing performance of the neural network according to the input data and the output data.
The application of the principles of the present invention will now be described in further detail with reference to the accompanying drawings.
The method of the invention is used for enhancing the spatial resolution of the hyperspectral single image, and as shown in figure 2, the method comprises the following specific steps:
the method comprises the following steps: and constructing a two-dimensional convolution neural network for acquiring spatial information and a one-dimensional convolution neural network for acquiring spectral information.
Step two: the residual block is embedded in the spatial network and the spectral network. Output of the (i + 1) th spatial residual block
The method is calculated by the following formula:
Ui+1＝Fi+1(Fi)+Fi；
wherein, Fi(HLR) Is the input of the i +1 th residual module, HLRFor high spectral low resolution images
Fi+1(Fi)＝vi+1βi+1(wi+1(Fi))；
θi+1(·)，βi+1() is i +1 activation function of residual module; wi+1For the first filter in the i +1 th residual block, vi+1A second filter in the (i + 1) th residual block; the design of the spectral residual block is similar to that of the spatial residual block;
and
is the input of the i +2 th spatial residual block, and so on. The detailed structure of the residual block is shown in fig. 3.
Step three: a mechanism of attention is embedded between the residual blocks of the spatial network. Output characteristic value of ith residual block
Conversion into three feature spaces f1、f2、f3A characteristic value of the attention mechanism is calculated,
is a weight matrix obtained by learning. The relationship between positions p and q is:
wherein, N is m multiplied by N,
attention was paid to the output of the mechanical layer:
L＝(L1，L2，…，Lt，…LN)；
the characteristic values of the attention mechanism are:
psi is a scale factor, initially 0. Note that the detailed structure of the mechanism is shown in fig. 4.
Step four: the objective function loss is designed.
loss＝ll1_loss+∝ls_loss；
Wherein, l, m, n and r are the spectral band, height, width and up-sampling factor (down-sampling factor) respectively | · | | | purple light1Is 11Norm, zi，jThe spectral vectors of the ith row and the jth column in the high-spectrum high-resolution image,
representing the reconstructed spectrum vector of the same spatial position after the super-resolution processing; oc is 0.01.
Step five: building a progressive fusion network for fusing the spatial and spectral information acquired by the neural network; compressing the two spectral bands into a frame channel, fusing them in a point-by-point addition manner; a 1-dimensional-2-dimensional convolutional neural network based on the attention mechanism is shown in fig. 5.
Step six: for the existing high-spectrum high-resolution image HHRAnd (4) preprocessing is carried out, and samples without useful information in the image are removed. Let HHRObtaining a low-resolution hyperspectral image H by a Gaussian filterLRThe 144 x 144 sub-pixel area was selected for evaluating the performance of the algorithm proposed by the present invention, and the rest was used for training.
Step seven: will be used for trainingH of the region (2)LRAnd HHRThe input quantity and the output quantity of the neural network are respectively used, the minimization of the target function is taken as a target, relevant parameters in the neural network are adjusted, and the neural network is trained. And when the target function meets the requirement of enhancing the super-resolution of the hyperspectral single image, stopping training to obtain a trained neural network model.
Step eight: h of sub-pixel region to be selectedLRAs input quantities, output quantities are obtained, and an average peak signal-to-noise ratio (MPSNR), an average structure similarity index (MSSIM), an average root mean square error (MRMSE), and a Spectral Angle Mapping (SAM) are respectively obtained for evaluating the performance of the neural network.
The following will describe the effects of the present invention in detail.
The method (deployed) is applied to the hyper-resolution processing of the hyper-spectral simple map, and comparison experiments of different algorithms are respectively carried out on two hyper-spectral images obtained by the university of Pavea and the center of Pavea through a ROSIS sensor. Because the performance of the hyper-spectral simple graph super-resolution algorithm based on deep learning is far better than that of the traditional algorithm, the three algorithms selected by the invention for comparison are the optimal deep learning algorithms (LapSRN, 3DFCNN and GDRRN) with open programs.
TABLE 1 Hyperspectral image super-resolution comparison based on university of Pavea
TABLE 2 super-resolution comparison of hyperspectral images based on the Pavea center
As can be seen from tables 1 and 2, the hyper-spectral single-image super-resolution algorithm based on the 1-d-2 d attention-based convolutional neural network has the highest average peak signal-to-noise ratio (MPSNR) and average structural similarity index (MSSIM) and the lowest average root mean square error (MRMSE) and Spectral Angle Mapper (SAM) in different experiments using high spectral image contrast. The spectral feature extraction method acquires more spatial features through an attention mechanism, and the spectral feature extraction method extracts spectral features and spatial features through the one-dimensional convolutional neural network and the two-dimensional convolutional neural network respectively, so that spectral fidelity is achieved to the maximum extent. The results of the above experimental super-resolution processing are shown in fig. 6 and 7.
In conclusion, the method has obvious technical effects, provides better technical contribution to the development of a hyperspectral single image super-resolution processing system and the research of an image processing technology, has wide application prospect in the field of hyperspectral single image super-resolution processing, and has considerable economic benefit.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (9)
1. A hyperspectral single-image super-resolution method based on a 1-dimensional to 2-dimensional convolutional neural network is characterized by comprising the following steps of:
firstly, building a parallel 1-2 dimensional convolution neural network for acquiring spectral information and spatial information; embedding the residual block into a space network and a spectrum network; embedding an attention mechanism between residual blocks of the spatial network; building a progressive fusion network for fusing the spatial and spectral information acquired by the neural network;
secondly, preprocessing the existing hyperspectral image data set to obtain a training data set and a test data set of a neural network;
then, training a neural network, taking the low-resolution hyperspectral image and the high-resolution hyperspectral image in the training data set as input data and output data of the neural network, training parameters in the neural network until the processing error of the neural network on the training data set meets the requirement of a minimum threshold value, and stopping training;
finally, testing the neural network, taking the low-resolution hyperspectral image in the test data set as input data of the neural network, obtaining output data of the neural network, and evaluating the super-resolution processing performance of the neural network according to the output data;
the hyperspectral single-image super-resolution method based on the 1-dimensional-2-dimensional convolutional neural network specifically comprises the following steps of:
the method comprises the following steps that firstly, a two-dimensional convolution neural network used for obtaining space information and a one-dimensional convolution neural network used for obtaining spectrum information are built;
secondly, embedding a residual block in the space network and the spectrum network; output of the (i + 1) th spatial residual block
Ui+1＝Fi+1(Fi)+Fi：
Wherein, Fi+1(Fi)＝vi+1βi+1(wi+1(Fi))，θi+1(·)，βi+1(. h) is the activation function of the i +1 th residual block, wi+1For the first filter in the i +1 th residual block, vi+1A second filter in the (i + 1) th residual block; the design of the spectral residual block is similar to that of the spatial residual block;
and
is the input of the (i + 2) th spatial residual block;
thirdly, embedding the attention mechanism into the residual error of the space networkBetween blocks; output characteristic value of ith residual block
Conversion into three feature spaces f1、f2、f3A characteristic value of the attention mechanism is calculated,
is a weight matrix obtained by learning, the relationship of the positions p and q is:
wherein, N is m multiplied by N,
attention was paid to the output of the mechanical layer:
L＝(L1，L2，…，Lt，…LN)；
the characteristic values of the attention mechanism are:
psi is a scale factor, initially 0;
fourthly, designing an objective function loss:
loss＝ll1_loss+αls_loss；
wherein, l, m, n and r are the spectral band, height, width and up-sampling factor, | · | | purple sweet1Is 11Norm, zi，jThe spectral vectors of the ith row and the jth column in the high-spectrum high-resolution image,
representing the reconstructed spectrum vector of the same spatial position after the super-resolution processing;
fifthly, building a progressive fusion network for fusing the space and spectrum information acquired by the neural network; compressing the two spectral bands into a frame channel, and fusing in a point-by-point addition manner;
sixthly, preprocessing the existing hyperspectral image data set, obtaining a low-resolution hyperspectral image through a Gaussian filter, and dividing the hyperspectral image into a training data set and a test data set according to a certain rule;
seventhly, continuously training the network by taking minimization of the target function loss as a target, and adjusting parameters in the network; when the target function is smaller than the super-resolution processing precision requirement epsilon, stopping training to obtain an optimal super-resolution processing result;
eighthly, taking the low-resolution hyperspectral image in the test data set as input data of the neural network to obtain output data of the neural network, and evaluating the super-resolution processing performance of the neural network according to the average peak signal-to-noise ratio (MPSNR), the average structure similarity index (MSSIM), the average root mean square error (MRMSE) and the Spectrum Angle Mapping (SAM) of the output data; the index is defined as follows:
wherein S and
representing true and reconstructed images, MAX, respectivelykIs the maximum intensity of the k-th frequency band,
and muSAre respectively
And the mean value of the sum of the S,
and σSAre respectively
And the variance of the sum of the S,
is that
And S covariance, C1And C2Is two constants for improving stability, n ═ w × l is the number of pixels,<zi，z′i>representing two spectra ziAnd z'iThe dot product, | · | | non-conducting phosphor2Is represented by2And (5) carrying out norm operation.
2. The hyperspectral single-map super-resolution method based on the 1-dimensional to 2-dimensional convolutional neural network as claimed in claim 1, wherein the first step is to acquire the spatial information and the spectral information respectively by a two-dimensional convolutional neural network and a one-dimensional convolutional neural network, and the two are parallel, and respectively extract the spatial information and the spectral information.
3. The hyperspectral single-map super-resolution method based on 1-dimensional-2-dimensional convolutional neural network of claim 1, wherein the second step embeds the residual block into the spatial network and the spectral network, and the input of the (i + 2) th spatial residual block is
And
4. the hyperspectral single-map super-resolution method based on the 1-dimensional-2-dimensional convolutional neural network of claim 1, wherein the attention mechanism of the third step is embedded between the residual blocks of the spatial network.
5. The hyperspectral single-map super-resolution method based on 1-dimensional-2-dimensional convolutional neural network of claim 1, wherein the objective function of the fourth step is i1Norm loss function and weighted loss of spectral loss function.
6. The hyperspectral single-map super-resolution method based on the 1-dimensional to 2-dimensional convolutional neural network as claimed in claim 1, wherein the fifth step is a progressive fusion network for fusing spatial information and spectral information, and two spectral segments are compressed into one frame channel during fusion and fused in a point-by-point addition manner.
7. The hyperspectral single-image super-resolution method based on the 1-dimensional to 2-dimensional convolutional neural network of claim 1, wherein in the sixth step, the existing hyperspectral image is processed by a Gaussian filter to obtain a low-resolution hyperspectral image, and the number of the finally obtained training data sets is greater than that of the test data sets.
8. The hyperspectral single-map super-resolution method based on the 1-dimensional-2-dimensional convolutional neural network as claimed in claim 1, wherein the neural network training process in the seventh step is to adjust parameters in the built neural network model so that the objective function value of the neural network is smaller than the fitting precision requirement epsilon to obtain an optimal neural network model;
and the eighth step of calculating average peak signal-to-noise ratio MPSNR, average structure similarity index MSSIM, average root mean square error MRMSE and spectral angle mapping SAM, and evaluating the performance of the neural network model.
9. A hyperspectral single image super-resolution processing system applying the hyperspectral single image super-resolution method based on the 1-dimensional-2-dimensional convolutional neural network according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911176083.9A CN111192193B (en) | 2019-11-26 | 2019-11-26 | Hyperspectral single-image super-resolution method based on 1-dimensional-2-dimensional convolution neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911176083.9A CN111192193B (en) | 2019-11-26 | 2019-11-26 | Hyperspectral single-image super-resolution method based on 1-dimensional-2-dimensional convolution neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111192193A CN111192193A (en) | 2020-05-22 |
| CN111192193B true CN111192193B (en) | 2022-02-01 |
Family
ID=70709455
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911176083.9A Active CN111192193B (en) | 2019-11-26 | 2019-11-26 | Hyperspectral single-image super-resolution method based on 1-dimensional-2-dimensional convolution neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111192193B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111965116A (en) * | 2020-07-21 | 2020-11-20 | 天津大学 | Hyperspectrum-based airport gas detection system and method |
| CN112767243B (en) * | 2020-12-24 | 2023-05-26 | 深圳大学 | Method and system for realizing super-resolution of hyperspectral image |
| CN112801881B (en) * | 2021-04-13 | 2021-06-22 | 湖南大学 | High-resolution hyperspectral calculation imaging method, system and medium |
| CN113435265B (en) * | 2021-06-09 | 2024-07-09 | 深圳大学 | Hyperspectral image classification method, hyperspectral image classification device, electronic equipment and storage medium |
| CN114155145B (en) * | 2021-11-12 | 2024-05-17 | 哈尔滨工业大学 | Space spectrum simultaneous super-resolution reconstruction method based on depth expansion network |
| CN115880157B (en) * | 2023-01-06 | 2023-05-26 | 中国海洋大学 | K-space pyramid feature fusion stereo image super-resolution reconstruction method |
| CN115719309A (en) * | 2023-01-10 | 2023-02-28 | 湖南大学 | Spectrum super-resolution reconstruction method and system based on low-rank tensor network |
| CN117132473B (en) * | 2023-10-20 | 2024-01-23 | 中国海洋大学 | Underwater rare earth spectrum detection method and spectrum super-resolution reconstruction model building method thereof |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108830796A (en) * | 2018-06-20 | 2018-11-16 | 重庆大学 | Based on the empty high spectrum image super-resolution reconstructing method combined and gradient field is lost of spectrum |
| CN110490799A (en) * | 2019-07-25 | 2019-11-22 | 西安理工大学 | Based on the target in hyperspectral remotely sensed image super-resolution method from fusion convolutional neural networks |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9672594B2 (en) * | 2014-10-21 | 2017-06-06 | The Boeing Company | Multiple pixel pitch super resolution |
| US10496883B2 (en) * | 2017-01-27 | 2019-12-03 | Signal Processing, Inc. | Method and system for enhancing predictive accuracy of planet surface characteristics from orbit |
| CN109903255A (en) * | 2019-03-04 | 2019-06-18 | 北京工业大学 | A kind of high spectrum image Super-Resolution method based on 3D convolutional neural networks |
-
2019
- 2019-11-26 CN CN201911176083.9A patent/CN111192193B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108830796A (en) * | 2018-06-20 | 2018-11-16 | 重庆大学 | Based on the empty high spectrum image super-resolution reconstructing method combined and gradient field is lost of spectrum |
| CN110490799A (en) * | 2019-07-25 | 2019-11-22 | 西安理工大学 | Based on the target in hyperspectral remotely sensed image super-resolution method from fusion convolutional neural networks |
Non-Patent Citations (1)
| Title |
|---|
| DUAL 1D-2D SPATIAL-SPECTRAL CNN FOR HYPERSPECTRAL IMAGE SUPER-RESOLUTION;Jiaojiao Li等;《2019 IEEE International Geoscience and Remote Sensing Symposium》;20190802;第3113-3116页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111192193A (en) | 2020-05-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111192193B (en) | Hyperspectral single-image super-resolution method based on 1-dimensional-2-dimensional convolution neural network | |
| CN111161141B (en) | Hyperspectral simple graph super-resolution method for counterstudy based on inter-band attention mechanism | |
| Lin et al. | Hyperspectral image denoising via matrix factorization and deep prior regularization | |
| CN104112263B (en) | The method of full-colour image and Multispectral Image Fusion based on deep neural network | |
| CN114119444B (en) | Multi-source remote sensing image fusion method based on deep neural network | |
| CN110544212B (en) | Convolutional neural network hyperspectral image sharpening method based on hierarchical feature fusion | |
| CN111080567A (en) | Remote sensing image fusion method and system based on multi-scale dynamic convolution neural network | |
| CN109636769A (en) | EO-1 hyperion and Multispectral Image Fusion Methods based on the intensive residual error network of two-way | |
| CN109509160A (en) | Hierarchical remote sensing image fusion method utilizing layer-by-layer iteration super-resolution | |
| CN108520495B (en) | Hyperspectral image super-resolution reconstruction method based on clustering manifold prior | |
| Patel et al. | Super-resolution of hyperspectral images: Use of optimum wavelet filter coefficients and sparsity regularization | |
| CN113240040A (en) | Polarized SAR image classification method based on channel attention depth network | |
| Liu et al. | Research on super-resolution reconstruction of remote sensing images: A comprehensive review | |
| Sui et al. | Fusion of hyperspectral and multispectral images based on a Bayesian nonparametric approach | |
| CN115100075B (en) | Hyperspectral panchromatic sharpening method based on spectrum constraint and residual attention network | |
| CN114511470B (en) | Attention mechanism-based double-branch panchromatic sharpening method | |
| CN114998167A (en) | Hyperspectral and multispectral image fusion method based on space-spectrum combined low rank | |
| CN115131258A (en) | Hyperspectral, multispectral and panchromatic image fusion method based on sparse tensor prior | |
| Li et al. | An effective multimodel fusion method for SAR and optical remote sensing images | |
| Liu et al. | MWLN: Multilevel Wavelet Learning Network for Continuous-Scale Remote Sensing Image Super-Resolution | |
| CN108520210A (en) | Based on wavelet transformation and the face identification method being locally linear embedding into | |
| Lu et al. | Multi-Supervised Recursive-CNN for Hyperspectral and Multispectral Image Fusion | |
| CN110097501B (en) | NDVI image fusion method based on non-local mean gradient sparse regularization | |
| CN107871307A (en) | full-colour image sharpening method based on spatial probability PCA and NSCT | |
| Ahmed et al. | Convolutional sparse coding using wavelets for single image super-resolution |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |