CN110533077A - Form adaptive convolution deep neural network method for classification hyperspectral imagery - Google Patents
Form adaptive convolution deep neural network method for classification hyperspectral imagery Download PDFInfo
- Publication number
- CN110533077A CN110533077A CN201910709042.5A CN201910709042A CN110533077A CN 110533077 A CN110533077 A CN 110533077A CN 201910709042 A CN201910709042 A CN 201910709042A CN 110533077 A CN110533077 A CN 110533077A
- Authority
- CN
- China
- Prior art keywords
- convolution
- space
- spectrum
- dimensional
- feature extraction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 230000003044 adaptive effect Effects 0.000 title claims abstract description 21
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 14
- 238000001228 spectrum Methods 0.000 claims abstract description 50
- 238000000605 extraction Methods 0.000 claims abstract description 36
- 238000013527 convolutional neural network Methods 0.000 claims description 19
- 230000006870 function Effects 0.000 claims description 19
- 238000012549 training Methods 0.000 claims description 12
- 230000003595 spectral effect Effects 0.000 claims description 10
- 238000010586 diagram Methods 0.000 claims description 9
- 230000004913 activation Effects 0.000 claims description 8
- 241000287196 Asthenes Species 0.000 claims description 3
- 230000035945 sensitivity Effects 0.000 claims description 3
- 230000009466 transformation Effects 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 claims description 2
- 238000010606 normalization Methods 0.000 claims description 2
- 238000005070 sampling Methods 0.000 claims description 2
- 230000007547 defect Effects 0.000 abstract description 3
- 238000004422 calculation algorithm Methods 0.000 abstract description 2
- 235000008331 Pinus X rigitaeda Nutrition 0.000 description 10
- 235000011613 Pinus brutia Nutrition 0.000 description 10
- 241000018646 Pinus brutia Species 0.000 description 10
- 238000013135 deep learning Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 238000002474 experimental method Methods 0.000 description 4
- 238000011160 research Methods 0.000 description 3
- 238000004088 simulation Methods 0.000 description 3
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 230000014759 maintenance of location Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000012805 post-processing Methods 0.000 description 2
- 238000000513 principal component analysis Methods 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 238000010521 absorption reaction Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000003331 infrared imaging Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000012417 linear regression Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Chemical compound O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a kind of form adaptive convolution deep neural network methods for classification hyperspectral imagery, this method comprises: spatial structural form is taken to learn branch；It is taken based on the form adaptive convolution kernel of guiding figure and can train；One-dimensional convolutional layer is tieed up by spectrum and space dimension two-dimensional convolution layer constitutes sky-spectrum signature extraction unit, each unit is gathered around there are two inputting, respectively characteristic pattern and guiding figure；Depth network is stacked by multiple skies-spectrum signature extraction unit, and skip floor connection is established between every two feature extraction unit；Network losses function is weighting cross entropy.The present invention passes through the spatial coherence between adjacent picture elements in study sky-modal data, the acceptance region shape of convolution algorithm can be adaptively adjusted according to the space structure relationship between explicit definition pixel, the defect of anisotropic character cannot be captured by overcoming fixed rectangular convolution, and excellent classification and Generalization Capability are all had to the high spectrum image of different resolution different scenes complexity.
Description
Technical Field
The invention relates to a hyperspectral image classification technology, in particular to a shape adaptive convolution depth neural network method for hyperspectral image classification.
Background
The hyperspectral camera can acquire cubic 'atlas-in-one' data rich in material information, can have nanometer (nm) level spectral resolution in the range of visible light-near infrared, short wave infrared and even middle infrared and thermal infrared bands, has hundreds of continuous and narrow band spectral band images, and is widely applied to the fields of military reconnaissance, environmental monitoring, geological exploration, target detection and the like. The supervised classification of hyperspectral images (HSI) is one of the most important research contents in the field.
Over the past decade, researchers have proposed many supervised classification methods for HSI. From simple models based on statistics to complex methods based on feature representation, HSI classification has become a targeted research content in the field of remote sensing. General classification methods, such as linear or non-linear regression (LR or NLR), Support Vector Machine (SVM), Extreme Learning Machine (ELM), and multi-kernel learning (MKL), can only roughly divide spectral data in some high-dimensional space without specifying the discriminative features of the spectrum. In order to explore the structure of hyperspectral data, methods based on feature representation, such as Sparse Representation (SR), Dictionary Learning (DL), manifold learning, wavelet transformation, Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), and the like, reveal spectral discriminant features to some extent. However, noise caused by aspects of low imaging quality (e.g. low resolution, illumination, shooting angle, etc.) and coarse labeling, etc., leads to the phenomenon that some pixels belonging to different classes have the same or similar spectra. In order to reduce the influence of these noises, researchers have proposed a classification method that improves the smoothness of the classification map by using the aggregation property of the pixels in the homogeneous region, i.e., a spatio-spectral joint classification method. The method based on space-spectrum feature extraction and the method based on post-processing are two common space-spectrum combined classification methods. In the space-spectrum feature extraction method, artificial features such as Gabor features, morphological features and texture features are generally used for representing the spatial structure of the HSI. In addition, post-processing based methods, such as Markov Random Fields (MRF), local classification voting, and relearning, utilize a priori local aggregation of pixels to correct partially misclassified pixels, ultimately improving the accuracy of the classification.
However, when processing different types of hyperspectral data, classification methods based on artificial features have certain limitations. For example, a method with a suitable parameter configuration for certain data sets may not perform well for other types of data acquired by different types of cameras. In other words, most conventional classification methods do not have sufficient generalization capability. Fortunately, deep learning methods can learn hierarchical feature representations directly from raw data, which provides another effective solution to the above-mentioned problem. Researchers have conducted extensive research, and some typical deep learning methods, such as Convolutional Neural Networks (CNNs), Deep Belief Networks (DBNs), and Stacked Automatic Encoders (SAE), have been applied to HSI classification. Although deep learning has powerful feature learning and representation capabilities, the traditional HSI deep learning classification architecture still has limitations. In particular, conventional CNNs that perform well on 2D data (e.g., images) have difficulty in handling 3D data (e.g., HSIs) well. This is because the spatial structure information of HSI exists only in local spatial regions and not in the global spatial domain, and the spectrum is the main source of information for distinguishing substances, while the spatial information plays only an auxiliary role. For the reasons described above, many deep learning methods take HSI neighborhood blocks as the input to the algorithm, thereby utilizing both spatial and spectral information. However, the standard CNN for HSI classification has significant drawbacks. In particular, due to the fixed geometry of the CNN module, the convolution unit samples the input feature map at a fixed location, which introduces interference information into the computation of pixel-level feature extraction and leads to misclassification of pixels near the boundaries between different materials. The obvious result of this CNN deficiency is that the classification map becomes too smooth and loses the detail information of many scenes, and the HSI classification effect is poor for rich scene details.
Disclosure of Invention
The invention aims to provide a shape adaptive convolution depth neural network method for hyperspectral image classification.
The technical solution for realizing the purpose of the invention is as follows: a shape adaptive convolution depth neural network method for hyperspectral image classification comprises the following steps:
the method comprises the steps that firstly, a convolutional neural network branch is used for learning space structure information of a hyperspectral image and storing the space structure information in a guide graph;
secondly, constructing shape self-adaptive convolution and extracting anisotropic space-spectrum characteristics by matching with a guide graph;
thirdly, a space-spectrum feature extraction unit is formed by the spectrum dimension one-dimensional convolution layer and the space dimension two-dimensional shape self-adaptive convolution layer, and spectrum dimension one-dimensional convolution and space dimension two-dimensional shape self-adaptive convolution are executed in sequence; each feature extraction unit has two inputs, namely a feature graph and a guide graph;
fourthly, the deep network is formed by stacking a plurality of space-spectrum feature extraction units, and layer jump connection is established between every two feature extraction units, namely the plurality of feature extraction units are stacked layer by layer to form the deep network, and the input of each unit is formed by splicing the outputs of all the units in front;
and fifthly, constructing a weighted cross entropy loss function.
Compared with the prior art, the invention has the remarkable advantages that: (1) extracting spatial structure information of the space-spectrum data in a network learning mode; (2) the shape adaptive convolution can dynamically adjust the shape of a convolution receiving domain according to the distribution of real ground objects, and the phenomenon of misclassification of image elements near the edge caused by the fixed geometric structure of the traditional convolution is avoided; (3) by using a feature extraction unit formed by the spectrum dimension one-dimensional convolution layer and the space dimension two-dimensional shape self-adaptive convolution layer, anisotropic space-spectrum features can be effectively extracted; (4) the network model is an end-to-end classification model, all learning modules are trained and inferred in a unified mode, extra supervision and training processes are not needed, and the method has excellent generalization and classification performance.
Drawings
FIG. 1 is a flow chart of a shape adaptive convolution depth neural network method for hyperspectral image classification according to the invention.
Fig. 2 is a schematic diagram of shape adaptive convolution.
Fig. 3 is a structural diagram of a spatio-spectral feature extraction unit.
FIG. 4 is a graph of results of different methods for classifying a synthetic dataset.
FIG. 5 is a diagram of results of different methods for classifying Indian Pines datasets.
Detailed Description
With reference to fig. 1, a shape adaptive convolution depth neural network method for hyperspectral image classification includes the following steps:
firstly, a spatial structure information learning branch is adopted, namely a convolutional neural network branch is used for learning spatial structure information of a hyperspectral image and is stored in a feature map called a directed graph. Note the bookAndthe three-dimensional space-spectrum data and the guide graph input by the network are respectively represented, wherein H, W, B, N is the height, the width, the channel number of the three-dimensional space-spectrum data and the channel number of the guide graph respectively. For each spatial coordinate p on the input spatio-spectral data0The guide map is calculated as (x, y):
Gj(p0)＝f(Wj·X(p0)+bj)
wherein, X (p)0) Representing a spatial coordinate p in the input spatio-spectral data0Pixel of (2), WjAnd bjRespectively representing the jth one-dimensional convolution kernel and the deviation, GjRepresents the jth band of the output directed graph, and f (-) represents the softsign activation function.
And secondly, adopting a shape adaptive convolution kernel based on a guide graph and performing trainable training, namely constructing a shape adaptive convolution different from a traditional fixed position sampling mode, and extracting anisotropic space-spectrum characteristics by matching the guide graph, as shown in fig. 2. Note the bookThe acceptance domain in convolution operation is represented, for example, by 3 × 3, as:
for each spatial coordinate p on the input feature map0(x, y) shape is self-consistent regardless of bias and activation functionThe adaptive convolution operation is represented as:
wherein S isiDenotes the ith deformable convolution kernel, yiAnd G is a guide graph. The deformable convolution kernel can be separated into the product of two independent kernels, expressed as:
wherein,represents an isotropic kernel, which is the same as a standard convolution kernel; k is a radical ofanisRepresents the anisotropic kernel, calculated as:
wherein G is a guide graph, and σ is an adjustment sensitivity parameter, | · |2Representing the norm of L2, exp (·) is an exponential function based on a natural constant e.
Thirdly, a space-spectrum feature extraction unit is formed by the spectrum dimension one-dimensional convolution layer and the space dimension two-dimensional shape adaptive convolution layer, and spectrum dimension one-dimensional convolution and space dimension two-dimensional shape adaptive convolution are executed in sequence, as shown in fig. 3. Each feature extraction unit has two inputs, respectively a feature map and a guide map. Let the input of the first hidden layer unit beOutput is asThen, firstly, batch normalization is performed, specifically:
where E (-) and Var (-) represent the mean and variance functions, respectively. And then performing spectrum dimension one-dimensional convolution, specifically as follows:
wherein k isl|jAnd bl|jRespectively representing the jth one-dimensional convolution kernel and deviation, T, in the ith feature extraction unitl|jThe jth channel of the output characteristic diagram is shown, and f (-) represents the softsign activation function. Finally, the space dimension two-dimensional shape self-adaptive convolution is executed, and a space-spectrum characteristic diagram o is outputlIs concretely provided with
Wherein s isl|jAnd pl|jRespectively representing the jth two-dimensional deformable convolution kernel and the corresponding deviation, p, in the ith feature extraction unitnEnumerating accepting domainsCoordinate of (5), G is a guide map, ol|jThe jth channel representing the output spatio-spectral feature map, f (-) represents the softsign activation function.
And fourthly, the deep network is formed by stacking a plurality of space-spectrum feature extraction units, layer jump connection is established between every two space-spectrum feature extraction units, namely the plurality of feature extraction units are stacked layer by layer to form the deep network, and the input of each unit is formed by splicing the outputs of all the units in the front. Let the input of the first hidden layer unit beOutput is asThen IlThe calculation formula of (A) is as follows:
Il＝[O1,O2,…,Ol-1]
wherein [ … ] represents stitching multiple signatures along the spectral dimension.
And fifthly, the network loss function is weighted cross entropy, namely, a weighted cross entropy loss function for relieving the class imbalance problem is constructed. Remember the network input asThe pixels of the spatio-spectral data may be divided into c different classes, and the output of the network isH, W, B, C are height, width, number of channels and number of classes of the three-dimensional space-spectrum data, respectively. The network is formed by stacking L (1 is equal to or less than L) hidden layer units, and the output of the first (1 is equal to or less than L) hidden layer unit is OlThen, the representation of the feature map input to the classification layer by the network hidden layer is as follows:
I＝[O1,O2,…,oL]
the transformation from the feature map of the spatio-spectral data to the pixel generic probability data is represented as:
wherein, [ … ]]Representing the stitching of multiple signatures along a spectral dimension, p0Space coordinates of the image elements in the spatio-spectral data are expressed as (x, y), kjAnd bjRespectively representing the jth one-dimensional convolution kernel and the deviation, Yj(p0) Representing a hyperspectral image at p0Probability that the picture element of the position belongs to the j-th class. Order toExpressed as a set of spatial coordinates of all training samples in the hyperspectral image, L (p)t) Represents a sample X (p)t) The vectorization tag of (a) is,Nc(1 ≦ C ≦ C) represents the number of jth class training samples, and the weighted cross-entropy loss function is expressed as:
wherein p istEnumerationAll coordinates of (1), Lc(pt) Representation vectorization tag L (p)t) Middle c value, YcRepresenting the c-th channel of the probability map Y.
The method has the capability of adaptively adjusting the convolution receiving domain shape and reserving the details of the classified scene, can be suitable for supervision and classification of the hyperspectral images with different resolutions and different scene complexities, and has excellent generalization and classification performances.
The network can adaptively adjust the shape of the acceptance domain of convolution operation according to the spatial structure relationship between explicit definition pixels by learning the spatial correlation between adjacent pixels in the space-spectrum data, overcomes the defect that fixed square convolution cannot capture anisotropic characteristics, and has excellent classification and generalization performance on hyperspectral images with different resolutions and scene complexities.
The effects of the present invention can be further illustrated by the following simulation experiments.
Examples
The hyperspectral image is typical three-dimensional space-spectrum data, and a set of synthetic hyperspectral data (synthesis dataset) and a set of real hyperspectral data (Indian Pines) are adopted in a simulation experiment. The synthetic dataset contained 162 spectral bands, a wavelength range of 0.4-2.5 μm, an image size of 200 × 200, 5 different classes of ground objects, for a total of 40000 annotated samples. The Indian Pines dataset is a hyperspectral remote sensing image acquired by an airborne visible infrared imaging spectrometer (AVIRIS) in an Indian Pines experimental area, indiana, usa. The image contains 220 bands in total, the spatial resolution is 20m, and the image size is 145 × 145. After removing 20 water vapor absorption and low signal-to-noise ratio bands (band numbers 104-. The area contains 10366 samples of 16 known land features. For the synthetic data set, 1% of each type of sample in the experiment is randomly selected as a training set, 1% is randomly selected as a verification set, and the rest 98% is used as a test set. For the Indian Pines data set, 10% of samples of each type are randomly selected as a training set in the experiment, 1% of samples are randomly selected as a verification set, and the rest of samples are used as a test set. The two experiments were repeated 10 times and averaged to obtain the final result, and OA (overhead Accuracy), AA (average Accuracy) and Kappa coefficients were used as evaluation indexes. Both sets of data were without any pre-processing. Further, the comparison method includes: a 2D convolutional neural network (2D-CNN) method, a two-channel convolutional neural network (DC-CNN) method, a 3D convolutional neural network (3D-CNN) method, a multi-channel convolutional neural network (MC-CNN) method, a deep space-spectrum residual error network (SSRN) method, and a fast dense space-spectrum convolutional depth network (FDSSC) method.
The modules of the network under experiment include a 1 × 1 convolutional layer (called a pilot layer) for generating a pilot graph and 5 spatio-spectral feature extraction units, where: the number of output channels of the guide layer is set to be 3; the number of channels of the output feature map of the 1 st feature extraction unit is set to be 128, and the number of channels of the output feature map of the 2 nd-5 th feature extraction units is set to be 32; in all the feature extraction units, the size of the deformable convolution kernel is set to 5 × 5, and the initial value of the sensitivity parameter σ is set to 1. In addition, the network optimizer adopts an Adam optimizer, wherein the learning rate of sigma is 0.01, the learning rate of the residual parameters is 0.001, and the first moment estimation exponential decay rate beta1Set to 0.9, second moment estimates the exponential decay rate beta2Set to 0.999, ∈ set to 1e-8, and the number of iterations set to 500. The experimental environment was as follows: a CPU: i7-8700K, GPU: GTX-1080Ti, memory: 32GB, Tensorflow-1.12.
Table 1 and Table 2 show the classification accuracy of simulation experiments performed on the synthetic data set and the Indian Pines data set by the method of the present invention, respectively.
TABLE 1 results of classification of synthetic datasets by different methods
TABLE 2 results of classification of Indian Pines datasets by different methods
From experimental results, the method is very effective for synthesizing data sets, and the performance of the method is obviously higher than that of the advanced methods including SSRN and FDSSC. Due to the inherent defects of the 2D convolution, the 2D-CNN, the DC-CNN, the SSRN and the FDSSC all show an over-smooth phenomenon on the data set, and the method effectively retains the original scene detail information, obtains a better classification effect and proves the effectiveness of the method. For the composite dataset, the classification maps obtained by the different methods are shown in FIG. 4. While for Indian Pines datasets, the method still achieves the best classification results in all comparison methods. As the noise contained in the Indian Pines data set is more, and the training set also contains noise, the method can automatically adjust the retention degree of scene details according to the training set so as to achieve the optimal classification precision. For the Indian Pines dataset, classification plots obtained by different methods are shown in fig. 5. The result shows that the method can effectively learn the structural information of the space-spectrum data, adjust the retention degree of scene details according to the training sample and achieve a better classification effect.
Claims (6)
1. A shape adaptive convolution depth neural network method for hyperspectral image classification is characterized by comprising the following steps:
the method comprises the steps that firstly, a convolutional neural network branch is used for learning space structure information of a hyperspectral image and storing the space structure information in a guide graph;
secondly, constructing shape self-adaptive convolution and extracting anisotropic space-spectrum characteristics by matching with a guide graph;
thirdly, a space-spectrum feature extraction unit is formed by the spectrum dimension one-dimensional convolution layer and the space dimension two-dimensional shape self-adaptive convolution layer, and spectrum dimension one-dimensional convolution and space dimension two-dimensional shape self-adaptive convolution are executed in sequence; each feature extraction unit has two inputs, namely a feature graph and a guide graph;
fourthly, the deep network is formed by stacking a plurality of space-spectrum feature extraction units, and layer jump connection is established between every two feature extraction units, namely the plurality of feature extraction units are stacked layer by layer to form the deep network, and the input of each unit is formed by splicing the outputs of all the units in front;
and fifthly, constructing a weighted cross entropy loss function.
2. The shape-adaptive convolution depth neural network method for hyperspectral image classification according to claim 1, characterized in that the first step is specifically:
adopting a spatial structure information learning branch, namely learning spatial structure information of a hyperspectral image by using a convolutional neural network branch, and storing the spatial structure information in a feature map called a directed graph;
note the bookAndrespectively representing three-dimensional space-spectrum data and a guide graph input by a network, wherein H, W, B, N is the height, width and channel number of the three-dimensional space-spectrum data and the channel number of the guide graph respectively; for each spatial coordinate p on the input spatio-spectral data0The guide map is calculated as (x, y):
Gj(p0)＝f(Wj·X(p0)+bj)
wherein, X (p)0) Representing a spatial coordinate p in the input spatio-spectral data0Pixel of (2), WjAnd bjRespectively representing the jth one-dimensional convolution kernel and the deviation, GjRepresents the jth band of the output directed graph, and f (-) represents the softsign activation function.
3. The shape-adaptive convolution depth neural network method for hyperspectral image classification according to claim 1, characterized in that the second step is specifically:
adopting a shape self-adaptive convolution kernel based on a guide graph and performing trainable training, namely constructing a shape self-adaptive convolution different from a traditional fixed position sampling mode, and extracting anisotropic space-spectrum characteristics by matching the guide graph;
note the bookRepresenting the acceptance domain in a convolution operation, for each spatial coordinate p on the input profile0Without considering the bias and activation function, (x, y), the shape adaptive convolution operation is expressed as:
wherein S isiDenotes the ith deformable convolution kernel, yiRepresenting the ith channel of the output characteristic diagram, wherein G is a guide diagram;
the deformable convolution kernel can be separated into the product of two independent kernels, expressed as:
wherein,represents an isotropic kernel, which is the same as a standard convolution kernel; k is a radical ofanisRepresents the anisotropic kernel, calculated as:
wherein G is a guide graph, and σ is an adjustment sensitivity parameter, | · |2Denotes the L2 norm, exp (-) is an exponential function with the natural constant e as the baseAnd (4) counting.
4. The shape adaptive convolution depth neural network method for hyperspectral image classification according to claim 1, characterized in that in the third step, a space-spectrum feature extraction unit is composed of a spectrum dimension one-dimensional convolution layer and a space dimension two-dimensional shape adaptive convolution layer, and the spectrum dimension one-dimensional convolution and the space dimension two-dimensional shape adaptive convolution are performed in sequence; each feature extraction unit has two inputs, namely a feature graph and a guide graph, and specifically comprises the following steps:
let the input of the first hidden layer unit beOutput is asFirstly, batch normalization is executed, specifically:
wherein E (-) and Var (-) represent the mean and variance functions, respectively;
and then performing spectrum dimension one-dimensional convolution, specifically as follows:
wherein k isl|jAnd bl|jRespectively representing the jth one-dimensional convolution kernel and deviation, T, in the ith feature extraction unitl|jThe jth channel representing the output characteristic diagram, and f (-) represents the softsign activation function;
finally, the space dimension two-dimensional shape self-adaptive convolution is executed, and a space-spectrum characteristic diagram O is outputLIs concretely provided with
Wherein S isl|jAnd pl|jRespectively representing the jth two-dimensional deformable convolution kernel and the corresponding deviation, p, in the ith feature extraction unitnEnumerating accepting domainsCoordinate of (5), G is a guide map, Ol|jThe jth channel representing the output spatio-spectral feature map, f (-) represents the softsign activation function.
5. The shape-adaptive convolution depth neural network method for hyperspectral image classification according to claim 1, wherein in the fourth step, the depth network is formed by stacking a plurality of space-spectrum feature extraction units, and a layer-skipping connection is established between every two space-spectrum feature extraction units, that is, the depth network is formed by stacking a plurality of feature extraction units layer by layer, and the input of each unit is formed by splicing the outputs of all the units in front, specifically:
let the input of the first hidden layer unit beOutput is asThen IlThe calculation formula of (A) is as follows:
Il＝[O1,O2,…,Ol-1]
wherein [ … ] represents stitching multiple signatures along the spectral dimension.
6. The shape-adaptive convolution depth neural network method for hyperspectral image classification according to claim 1, characterized in that in the fifth step, the network loss function is weighted cross entropy, that is, a weighted cross entropy loss function for alleviating the class imbalance problem is constructed, specifically:
remember the network input asThe space-spectrumThe pixels of the data can be divided into c different classes, and the output of the network isH, W, B, C are height, width, number of channels and number of classes of the three-dimensional space-spectrum data, respectively. The network is formed by stacking L hidden layer units, and the output of the first hidden layer unit is OlL is more than or equal to 1 and less than or equal to L, and L is more than or equal to 1, the representation of the characteristic graph input into the classification layer by the network hidden layer is as follows:
I＝[O1,O2,…,OL]
the transformation from the feature map of the spatio-spectral data to the pixel generic probability data is represented as:
wherein, [ … ]]Representing the stitching of multiple signatures along a spectral dimension, p0Space coordinates of the image elements in the spatio-spectral data are expressed as (x, y), kjAnd bjRespectively representing the jth one-dimensional convolution kernel and the deviation, Yj(p0) Representing a hyperspectral image at p0Probability that the pixel of the position belongs to the jth class; order toExpressed as a set of spatial coordinates of all training samples in the hyperspectral image, L (p)t) Represents a sample X (p)t) The vectorization tag of (a) is,Ncrepresenting the number of j-th class training samples, C is more than or equal to 1 and less than or equal to C, then the weighted cross entropy loss function is represented as:
wherein p istEnumerationAll coordinates of (1), Lc(pt) Representation vectorization tag L (p)t) Middle c value, YcRepresenting the c-th channel of the probability map Y.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910709042.5A CN110533077B (en) | 2019-08-01 | 2019-08-01 | Shape adaptive convolution depth neural network method for hyperspectral image classification |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910709042.5A CN110533077B (en) | 2019-08-01 | 2019-08-01 | Shape adaptive convolution depth neural network method for hyperspectral image classification |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110533077A true CN110533077A (en) | 2019-12-03 |
| CN110533077B CN110533077B (en) | 2022-09-27 |
Family
ID=68662064
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910709042.5A Active CN110533077B (en) | 2019-08-01 | 2019-08-01 | Shape adaptive convolution depth neural network method for hyperspectral image classification |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110533077B (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111144423A (en) * | 2019-12-26 | 2020-05-12 | 哈尔滨工业大学 | Hyperspectral remote sensing data multi-scale spectral feature extraction method based on one-dimensional group convolution neural network |
| CN111612127A (en) * | 2020-04-29 | 2020-09-01 | 南京理工大学 | Multi-direction information propagation convolution neural network construction method for hyperspectral image classification |
| CN111667019A (en) * | 2020-06-23 | 2020-09-15 | 哈尔滨工业大学 | Hyperspectral image classification method based on deformable separation convolution |
| CN111797941A (en) * | 2020-07-20 | 2020-10-20 | 中国科学院长春光学精密机械与物理研究所 | Image classification method and system carrying spectral information and spatial information |
| CN112990315A (en) * | 2021-03-17 | 2021-06-18 | 北京大学 | 3D shape image classification method of equal-variation 3D convolution network based on partial differential operator |
| CN113743429A (en) * | 2020-05-28 | 2021-12-03 | 中国人民解放军战略支援部队信息工程大学 | Hyperspectral image classification method and device |
| CN114186641A (en) * | 2021-12-16 | 2022-03-15 | 长安大学 | Landslide susceptibility evaluation method based on deep learning |
| CN114638762A (en) * | 2022-03-24 | 2022-06-17 | 华南理工大学 | Modularized hyperspectral image scene self-adaptive panchromatic sharpening method |
| CN116612356A (en) * | 2023-06-02 | 2023-08-18 | 北京航空航天大学 | Hyperspectral anomaly detection method based on deep learning network |
| CN116704241A (en) * | 2023-05-22 | 2023-09-05 | 齐鲁工业大学(山东省科学院) | Full-channel 3D convolutional neural network hyperspectral remote sensing image classification method |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106845381A (en) * | 2017-01-16 | 2017-06-13 | 西北工业大学 | Sky based on binary channels convolutional neural networks composes united hyperspectral image classification method |
| CN109376753A (en) * | 2018-08-31 | 2019-02-22 | 南京理工大学 | A kind of the three-dimensional space spectrum separation convolution depth network and construction method of dense connection |
-
2019
- 2019-08-01 CN CN201910709042.5A patent/CN110533077B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106845381A (en) * | 2017-01-16 | 2017-06-13 | 西北工业大学 | Sky based on binary channels convolutional neural networks composes united hyperspectral image classification method |
| CN109376753A (en) * | 2018-08-31 | 2019-02-22 | 南京理工大学 | A kind of the three-dimensional space spectrum separation convolution depth network and construction method of dense connection |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111144423B (en) * | 2019-12-26 | 2023-05-05 | 哈尔滨工业大学 | Hyperspectral remote sensing data multi-scale spectral feature extraction method based on one-dimensional group convolutional neural network |
| CN111144423A (en) * | 2019-12-26 | 2020-05-12 | 哈尔滨工业大学 | Hyperspectral remote sensing data multi-scale spectral feature extraction method based on one-dimensional group convolution neural network |
| CN111612127B (en) * | 2020-04-29 | 2022-09-06 | 南京理工大学 | Multi-direction information propagation convolution neural network construction method for hyperspectral image classification |
| CN111612127A (en) * | 2020-04-29 | 2020-09-01 | 南京理工大学 | Multi-direction information propagation convolution neural network construction method for hyperspectral image classification |
| CN113743429A (en) * | 2020-05-28 | 2021-12-03 | 中国人民解放军战略支援部队信息工程大学 | Hyperspectral image classification method and device |
| CN111667019A (en) * | 2020-06-23 | 2020-09-15 | 哈尔滨工业大学 | Hyperspectral image classification method based on deformable separation convolution |
| CN111797941A (en) * | 2020-07-20 | 2020-10-20 | 中国科学院长春光学精密机械与物理研究所 | Image classification method and system carrying spectral information and spatial information |
| CN112990315A (en) * | 2021-03-17 | 2021-06-18 | 北京大学 | 3D shape image classification method of equal-variation 3D convolution network based on partial differential operator |
| CN112990315B (en) * | 2021-03-17 | 2023-10-20 | 北京大学 | 3D shape image classification method of constant-variation 3D convolution network based on partial differential operator |
| CN114186641B (en) * | 2021-12-16 | 2022-08-09 | 长安大学 | Landslide susceptibility evaluation method based on deep learning |
| CN114186641A (en) * | 2021-12-16 | 2022-03-15 | 长安大学 | Landslide susceptibility evaluation method based on deep learning |
| CN114638762A (en) * | 2022-03-24 | 2022-06-17 | 华南理工大学 | Modularized hyperspectral image scene self-adaptive panchromatic sharpening method |
| CN114638762B (en) * | 2022-03-24 | 2024-05-24 | 华南理工大学 | Modularized hyperspectral image scene self-adaptive panchromatic sharpening method |
| CN116704241A (en) * | 2023-05-22 | 2023-09-05 | 齐鲁工业大学(山东省科学院) | Full-channel 3D convolutional neural network hyperspectral remote sensing image classification method |
| CN116612356A (en) * | 2023-06-02 | 2023-08-18 | 北京航空航天大学 | Hyperspectral anomaly detection method based on deep learning network |
| CN116612356B (en) * | 2023-06-02 | 2023-11-03 | 北京航空航天大学 | Hyperspectral anomaly detection method based on deep learning network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110533077B (en) | 2022-09-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110533077B (en) | Shape adaptive convolution depth neural network method for hyperspectral image classification | |
| CN106469316B (en) | Hyperspectral image classification method and system based on superpixel-level information fusion | |
| Jin et al. | A survey of infrared and visual image fusion methods | |
| Fu et al. | Hyperspectral anomaly detection via deep plug-and-play denoising CNN regularization | |
| Li et al. | Hyperspectral image super-resolution using deep convolutional neural network | |
| CN107316013B (en) | Hyperspectral image classification method based on NSCT (non-subsampled Contourlet transform) and DCNN (data-to-neural network) | |
| Xie et al. | Hyperspectral image super-resolution using deep feature matrix factorization | |
| CN108491849B (en) | Hyperspectral image classification method based on three-dimensional dense connection convolution neural network | |
| Xing et al. | Pan-sharpening via deep metric learning | |
| Zhang et al. | Sparse transfer manifold embedding for hyperspectral target detection | |
| CN110348399B (en) | Hyperspectral intelligent classification method based on prototype learning mechanism and multidimensional residual error network | |
| CN105608433B (en) | A kind of hyperspectral image classification method based on nuclear coordination expression | |
| Venugopal | Automatic semantic segmentation with DeepLab dilated learning network for change detection in remote sensing images | |
| CN109615008B (en) | Hyperspectral image classification method and system based on stack width learning | |
| CN108182449A (en) | A kind of hyperspectral image classification method | |
| CN111680579B (en) | Remote sensing image classification method for self-adaptive weight multi-view measurement learning | |
| CN112580480B (en) | Hyperspectral remote sensing image classification method and device | |
| CN113139512B (en) | Depth network hyperspectral image classification method based on residual error and attention | |
| Fu et al. | A novel spectral-spatial singular spectrum analysis technique for near real-time in situ feature extraction in hyperspectral imaging | |
| Plichoski et al. | A face recognition framework based on a pool of techniques and differential evolution | |
| Hou et al. | Spatial–spectral weighted and regularized tensor sparse correlation filter for object tracking in hyperspectral videos | |
| CN109190511A (en) | Hyperspectral classification method based on part Yu structural constraint low-rank representation | |
| CN107133579A (en) | Based on CSGF (2D)2The face identification method of PCANet convolutional networks | |
| CN113935433A (en) | Hyperspectral image classification method based on depth spectrum space inverse residual error network | |
| Song et al. | HDTFF-Net: Hierarchical deep texture features fusion network for high-resolution remote sensing scene classification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |