CN110321963B - Hyperspectral image classification method based on fusion of multi-scale and multi-dimensional space spectrum features - Google Patents
Hyperspectral image classification method based on fusion of multi-scale and multi-dimensional space spectrum features Download PDFInfo
- Publication number
- CN110321963B CN110321963B CN201910616371.5A CN201910616371A CN110321963B CN 110321963 B CN110321963 B CN 110321963B CN 201910616371 A CN201910616371 A CN 201910616371A CN 110321963 B CN110321963 B CN 110321963B
- Authority
- CN
- China
- Prior art keywords
- dimensional
- layer
- feature
- hyperspectral image
- scale
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/13—Satellite images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/194—Terrestrial scenes using hyperspectral data, i.e. more or other wavelengths than RGB
Abstract
The invention discloses a hyperspectral image classification method based on fusion of multi-scale multidimensional space spectrum features, which comprises the following implementation steps of: (1) inputting a hyperspectral image; (2) preprocessing a hyperspectral image to be classified; (3) neighborhood block fetching; (4) generating a training set and a test set; (5) constructing a multi-scale space spectrum feature and multi-dimensional feature fusion network; (6) training a multi-scale space spectrum feature and multi-dimensional feature fusion network; (7) the test samples are classified. The method provided by the invention can effectively solve the problems that the convolutional neural network has too single characteristics and too single scale during training, can solve the problem that the average classification accuracy AA is low during hyperspectral classification, can keep the identification capability of small sample number categories while realizing higher classification accuracy, and has good classification performance.
Description
Technical Field
The invention belongs to the technical field of image processing, and further relates to a hyperspectral image classification method based on fusion of multi-scale multi-dimensional space spectrum features in the technical field of image classification. The method can be applied to various fields such as disaster monitoring, geological exploration, urban planning, target identification and the like by analyzing the types of the ground objects in the hyperspectral image.
Background
The hyperspectrum records the continuous spectrum characteristics of the ground object target by rich wave band information, and has the possibility of identifying more types of ground object targets and classifying the targets with higher precision. The key of the hyperspectral image classification technology is to obtain higher classification precision by using a small number of training samples. Recently, with the wide application of deep learning in various fields, a variety of deep learning classification methods have appeared in hyperspectral classification, such as an auto-coder ae (auto encoder), a convolutional Neural network cnn (convolutional Neural networks), a deep Belief network dbn (deep Belief network), and the like. Among these methods, CNN has the best classification performance, but never achieved human expectations. The hyperspectral image has serious phenomena of 'same-spectrum foreign matter, same-object different spectrum'.
A patent document applied by the university of electronic technology of Sigan in 'hyperspectral classification method based on fusion of space coordinates and space spectrum characteristics' (patent application No. 201710644479.6, application publication No. CN 107451614A) provides a hyperspectral image classification method based on fusion of space coordinates and space spectrum characteristics. The method comprises the steps of firstly carrying out space neighborhood division sampling on a hyperspectral image, then taking a space coordinate as a space feature, then respectively classifying the space feature and a spectrum feature by using a Support Vector Machine (SVM), taking the probability that a classified pixel belongs to each class as a probability feature, finally fusing the probability feature obtained by spatial feature classification and the probability feature obtained by the spectrum feature, and classifying by using the SVM (support Vector machine) again to obtain a final classification result. The method has the disadvantages that only the space coordinates are used as the space characteristics, the space information of the hyperspectral image is not fully utilized, and the space characteristics are not fully fused. Moreover, the one-dimensional feature vector is used as an input, scale features are not fully utilized, and the spatial coordinates and the one-dimensional feature vector have poor effect on classifying the ground object categories with non-concentrated sample distribution or small sample amount.
A method for classifying Hyperspectral images using an end-to-end spectrum-space Residual error Network (SSRN) is proposed in the published paper "Spectral-Spatial information Network for Hyperspectral Image Classification" A3-D Deep Learning frame "(IEEE Transactions on Geoscience and Remote Sensing,2017: 1-12). The method takes an original three-dimensional cube as input data and does not need characteristic engineering. In an end-to-end spectrum-space residual error network, a spectrum and space residual error block continuously learns identification characteristics from abundant spectrum characteristics and space backgrounds in a hyperspectral image, spectrum characteristics obtained by a three-dimensional convolutional neural network and space characteristics obtained by a two-dimensional convolutional neural network are fused in a cascading mode, and finally the fused characteristics are input into a classification layer to classify the hyperspectral image. The method has the disadvantages that the single scale is insufficient in utilization of scale features, and only neighborhood blocks of the single scale are extracted as input. While single scale features do not perform well in overall classification accuracy.
Disclosure of Invention
The invention aims to provide a hyperspectral image classification method based on fusion of multi-scale multi-dimensional space spectrum features aiming at the defects of the prior art, which is used for fusing related information among different scale features and realizing fusion of high-dimensional features and low-dimensional features as well as spatial features and spectral features, and is used for solving the problems of low classification precision, non-centralized sample distribution or poor classification effect on ground object categories with small sample amount in the existing hyperspectral image classification method.
The idea of realizing the purpose of the invention is to construct three feature extraction branches with the same structure and a combined classifier, then generate a multi-scale space-spectrum feature and multi-dimensional feature fusion network, input a multi-scale training sample into the multi-scale space-spectrum feature and multi-dimensional feature fusion network, extract multi-scale multi-dimensional space-spectrum joint features and classify, train the network by using a loss function, and finally input a test sample into the trained multi-scale space-spectrum feature and multi-dimensional feature fusion network to classify the hyperspectral images.
In order to achieve the above purpose, the specific steps of the invention comprise:
(1) inputting a hyperspectral image:
inputting a hyperspectral image which is a three-dimensional feature cube
Each wave band in the hyperspectral image corresponds to a two-dimensional matrix in the feature cube
Wherein e represents belonging to a symbol,
the method comprises the steps of representing a real number field symbol, wherein m represents the length of a hyperspectral image, n represents the width of the hyperspectral image, b represents the number of spectral wave bands of the hyperspectral image, i represents the serial number of the spectral wave bands in the hyperspectral image, and i is 1,2, … and b;
(2) preprocessing a hyperspectral image to be classified:
(2a) converting an m × n × b three-dimensional hyperspectral image matrix into an a × b two-dimensional feature matrix, wherein a is m × n, each column in the two-dimensional feature matrix represents a spectral dimension, and each row represents all spectral information of each sample;
(2b) performing normalization processing on the two-dimensional feature matrix by adopting a normalization formula;
(2c) converting the normalized two-dimensional feature matrix into a normalized three-dimensional feature matrix with the same size as the original hyperspectral image;
(3) neighborhood block fetching:
(3a) performing 0-pixel edge filling operation on the normalized three-dimensional feature matrix, wherein the sizes of 0 pixels of the edge filling are respectively 3, 5 and 7;
(3b) in the filled hyperspectral image, with each pixel point as the center, selecting 7 × 7, 11 × 11 and 15 × 15 neighborhood blocks respectively to obtain neighborhood blocks of three scales;
(4) generating a training set and a testing set:
(4a) respectively distributing the neighborhood blocks of the three scales to a set to which the class belongs according to the class of the central pixel point of the neighborhood blocks;
(4b) respectively taking the neighborhood blocks of known class labels in the neighborhood blocks in each class of sets as training sets, taking the central pixel point label of each neighborhood block as the label of the neighborhood block, and respectively taking the residual neighborhood blocks in each class of sets as test sets;
(5) constructing a multi-scale space spectrum feature and multi-dimensional feature fusion network:
(5a) three feature extraction branches with the same structure are respectively built, and the structure of each branch is as follows: the first three-dimensional convolution layer → the first normative layer → the first activation function layer → the second three-dimensional convolution layer → the second normative layer → the second activation function layer → the third three-dimensional convolution layer → the third normative layer → the third activation function layer → the first two-dimensional convolution layer → the fourth normative layer → the fourth activation function layer → the first fusion layer → the second two-dimensional convolution layer → the fifth normative layer → the fifth activation function layer → the second fusion layer → the third two-dimensional convolution layer → the sixth normative layer → the sixth activation function layer → the first maximum pooling layer → the fourth two-dimensional convolution layer → the seventh normative layer → the seventh activation function layer → the second maximum pooling layer; the first fusion layer fuses the first activation function layer and the fourth activation function layer through an addition operation; the second fusion layer fuses the second activation function layer and the fifth activation function layer through an addition operation;
(5b) three feature extraction branches with the same structure are laminated and fused through a concatenate and then are sequentially connected with a global average pooling layer and an output layer to form a multi-scale space spectrum feature and multi-dimensional feature fusion network;
(5c) the parameters of the multi-scale space spectrum feature and the multi-dimensional feature fusion network are set as follows: setting the number of the neurons of the first, second and third three-dimensional convolutional layers to be 24, setting the sizes of the convolutional cores to be (1, 1, 20), (1, 1, 3) and (1, 1, 10) in sequence, and setting the sizes of the convolutional steps to be 20, 1 and 1 in sequence; the number of the neurons of the first, second, third and fourth two-dimensional convolutional layers is set to be 240, 24 and 24 in sequence, the lengths of the convolutional kernels are all set to be 3, and the convolution step length is all set to be 1; setting the pooling length of each maximum pooling layer to be 3; setting the momentum factor of each specification layer to be 0.8; setting the pooling length of the global average pooling layer to 7; setting the activation function of each activation function layer as a ReLU activation function; setting the number of neurons in an output layer as the number of categories, and selecting a softmax function as an activation function;
(6) training a multi-scale space spectrum feature and multi-dimensional feature fusion network:
inputting the training set and the labels of the training set into a multi-scale space-spectrum feature and multi-dimensional feature fusion network for training to obtain a trained multi-scale space-spectrum feature and multi-dimensional feature fusion network;
(7) classifying the test samples:
inputting the test set into the trained multi-scale space spectrum feature and multi-dimensional feature fusion network, outputting a class label of the test sample, and using the output of the multi-scale space spectrum feature and multi-dimensional feature fusion network as a prediction label of the test sample to finish the hyperspectral image classification.
Compared with the prior art, the invention has the following advantages:
firstly, three feature extraction branches with the same structure are built for extracting scale features of different scales in a hyperspectral image, and the defects that in the prior art, the scale features are not fully utilized by a single scale and the classification precision is low are overcome, so that the method has the advantage of extracting multi-scale features, and the classification precision of ground objects in the hyperspectral image is improved.
Secondly, the first activation function layer and the fourth activation function layer are fused through addition operation, the second activation function layer and the fifth activation function layer are fused through addition operation, and the method is used for fusing the spectral characteristics extracted by the three-dimensional Convolutional Neural network 3D-CNN (3D Convolutional Neural Networks) and the spatial characteristics extracted by the two-dimensional Convolutional Neural network 2D-CNN (2D Convolutional Neural Networks), overcomes the defects that the spatial spectral characteristics are not fused sufficiently in the prior art, the sample distribution is not concentrated or the classification effect of the ground object class with few samples is poor, has the advantages of fully fusing the spectral characteristics and the spatial characteristics, and the high-dimensional characteristics and the low-dimensional characteristics, and improves the identification capability of the small-number sample class.
Drawings
FIG. 1 is a flow chart of an implementation of the present invention;
FIG. 2 is a block diagram of a multi-scale spatial spectral feature and multi-dimensional feature fusion network according to the present invention.
Detailed Description
The present invention is described in further detail below with reference to the attached drawing figures.
Referring to fig. 1, a specific implementation of the present invention is described in further detail.
Step 1, inputting a hyperspectral image.
Inputting a hyperspectral image which is a three-dimensional feature cube
Each wave band in the hyperspectral image corresponds to a two-dimensional matrix in the feature cube
Wherein e represents belonging to a symbol,
the real number field symbol is shown, m represents the length of the hyperspectral image, n represents the width of the hyperspectral image, b represents the number of the spectrum wave band of the hyperspectral image, i represents the serial number of the spectrum wave band in the hyperspectral image, and i is 1,2, … and b.
And 2, preprocessing the hyperspectral image to be classified.
And converting the m × n × b three-dimensional hyperspectral image matrix into an a × b two-dimensional feature matrix, wherein a is m × n, each column in the two-dimensional feature matrix represents a spectral dimension, and each row represents all spectral information of each sample.
And carrying out normalization processing on the two-dimensional characteristic matrix by adopting a normalization formula.
The normalization formula is as follows:
wherein the content of the first and second substances,
representing the jth surface feature target in the ith spectral band in the normalized two-dimensional feature matrix,
represents the average pixel value of all the ground object targets in the ith spectral band in the two-dimensional feature matrix,
and representing the variance value of all the ground object target pixel values in the ith spectral band in the two-dimensional characteristic matrix.
And converting the normalized two-dimensional characteristic matrix into a normalized three-dimensional characteristic matrix with the same size as the original hyperspectral image.
And 3, neighborhood block taking.
And performing 0-pixel edge filling operation on the normalized three-dimensional feature matrix, wherein the sizes of 0 pixels of the edge filling are respectively 3, 5 and 7.
In the filled hyperspectral image, 7 × 7, 11 × 11 and 15 × 15 neighborhood blocks are respectively selected by taking each pixel point as a center, so that the neighborhood blocks of three scales are obtained.
And 4, generating a training set and a testing set.
And respectively distributing the neighborhood blocks of the three scales to a set to which the class belongs according to the class of the central pixel point of the neighborhood blocks.
And respectively taking the neighborhood blocks with known class labels in the neighborhood blocks in each class of set as training sets, taking the central pixel point label of each neighborhood block as the label of the neighborhood block, and respectively taking the residual neighborhood blocks in each class of set as test sets.
And 5, constructing a multi-scale space spectrum feature and multi-dimensional feature fusion network.
The structure of the multi-scale space-spectral feature and multi-dimensional feature fusion network of the present invention is described in further detail with reference to fig. 2.
Three feature extraction branches with the same structure are respectively built, and the structure of each branch is as follows: the first three-dimensional convolution layer → the first normative layer → the first activation function layer → the second three-dimensional convolution layer → the second normative layer → the second activation function layer → the third three-dimensional convolution layer → the third normative layer → the third activation function layer → the first two-dimensional convolution layer → the fourth normative layer → the fourth activation function layer → the first fusion layer → the second two-dimensional convolution layer → the fifth normative layer → the fifth activation function layer → the second fusion layer → the third two-dimensional convolution layer → the sixth normative layer → the sixth activation function layer → the first maximum pooling layer → the fourth two-dimensional convolution layer → the seventh normative layer → the seventh activation function layer → the second maximum pooling layer; the first fusion layer fuses the first activation function layer and the fourth activation function layer through an addition operation; the second fusion layer fuses the second activation function layer and the fifth activation function layer through an addition operation.
And (3) connecting the three feature extraction branches with the same structure with the global average pooling layer and the output layer in sequence after merging and fusing the three feature extraction branches with the same structure through the concatenate layer to form a multi-scale space spectrum feature and multi-dimensional feature fusion network.
The parameters of the multi-scale space spectrum feature and the multi-dimensional feature fusion network are set as follows: setting the number of the neurons of the first, second and third three-dimensional convolutional layers to be 24, setting the sizes of the convolutional cores to be (1, 1, 20), (1, 1, 3) and (1, 1, 10) in sequence, and setting the sizes of the convolutional steps to be 20, 1 and 1 in sequence; the number of the neurons of the first, second, third and fourth two-dimensional convolutional layers is set to be 240, 24 and 24 in sequence, the lengths of the convolutional kernels are all set to be 3, and the convolution step length is all set to be 1; setting the pooling length of each maximum pooling layer to be 3; setting the momentum factor of each specification layer to be 0.8; setting the pooling length of the global average pooling layer to 7; setting the activation function of each activation function layer as a ReLU activation function; and setting the number of the neurons of the output layer as the number of categories, and selecting a softmax function as an activation function.
And 6, training a multi-scale space spectrum feature and multi-dimensional feature fusion network.
And inputting the training set and the labels of the training set into the multi-scale space-spectrum feature and multi-dimensional feature fusion network for training to obtain the trained multi-scale space-spectrum feature and multi-dimensional feature fusion network.
The specific operation steps of inputting the labels of the training set and the training set into the multi-scale space-spectrum feature and multi-dimensional feature fusion network for training are as follows:
and step 1, respectively inputting the training set into feature extraction branches of three different neighborhood blocks of the multi-scale space spectrum feature and multi-dimensional feature fusion network, and outputting a prediction label vector of a training sample.
And 2, optimizing network parameters by using a loss function of the multi-scale space spectrum feature and multi-dimensional feature fusion network by adopting a gradient descent method until the network parameters are converged, wherein the learning rate of the loss function is set to be 0.0001.
And 3, calculating the cross entropy between the predicted label vector and the real label vector by using a cross entropy formula.
Where L represents the cross entropy between the predicted tag vector and the true tag vector, Σ represents the summation operation, yiRepresenting the ith element in the predictor tag vector, ln represents a logarithmic operation based on a natural constant e,
representing the mth element in the prediction tag vector.
The effect of the present invention is further explained by combining the simulation experiment as follows:
1. simulation experiment conditions are as follows:
the hardware test platform adopted by the simulation experiment of the invention is as follows: the processor is an Inter Core i7-8750H, the main frequency is 2.20GHz, and the memory is 16 GB; the software platform is as follows: the Windows 10 enterprise version of the 64-bit operating system and python3.6 were subjected to emulation testing.
The hyperspectral image dataset used in the simulation experiment of the invention is an Indian pins dataset collected by an AVIRIS sensor on Indian pins test field in northwest of Indiana and a Pavia unidentivity dataset obtained by a ROSIS hyperspectral remote sensing satellite shooting at the university of Pavia in northern Italy, wherein the size of the Indian pins dataset image is 145, the image has 200 spectral bands and comprises 16 types of ground objects, and the category and the number of each type of ground object are shown in Table 1.
TABLE 1 Indian pins sample Categories and quantities
| Class label | Class of ground object | Number of |
| 1 | Alfalfa | 46 |
| 2 | Corn-notill | 1428 |
| 3 | Corn-mintill | 830 |
| 4 | Corn | 237 |
| 5 | Grass-pasture | 483 |
| 6 | Grass-trees | 730 |
| 7 | Grass-pasture-mowed | 28 |
| 8 | Hay-windrowed | 478 |
| 9 | Oats | 20 |
| 10 | Soybean-nottill | 972 |
| 11 | Soybean-mintill | 2455 |
| 12 | Soybean-clean | 593 |
| 13 | Wheat | 205 |
| 14 | Woods | 1265 |
| 15 | Buildings-grass-trees-drives | 386 |
| 16 | Stone-steel-towers | 93 |
The Pavia unity data set image size was 610 x 340 with 103 spectral bands containing 9 classes of terrain, the class and number of each class of terrain being shown in table 2.
TABLE 2 Pavia unity sample types and quantities
| Class label | Class of ground object | Number of |
| 1 | Asphalt | 6631 |
| 2 | Meadows | 18649 |
| 3 | Gravel | 2099 |
| 4 | Trees | 3064 |
| 5 | Sheets | 1345 |
| 6 | Bare soil | 5029 |
| 7 | Bitumen | 1330 |
| 8 | Bricks | 3682 |
| 9 | Shadows | 947 |
2. Simulation experiment content and result analysis:
the invention adopts multi-scale space spectrum characteristics and a multi-dimensional characteristic fusion network for classification, and the multi-scale idea is the key for solving the problem that the single-scale classification effect is not ideal, so that the simulation experiment 1 of the invention uses the multi-scale space spectrum characteristics and the multi-dimensional characteristic fusion network to classify two hyperspectral image data of Indian pines and Pavia unity, and compares the classification result with the classification results of three single-scale networks, thereby showing that the classification effect of the multi-scale characteristic fusion of the invention is superior to the classification effect based on single scale, and further proving the effectiveness of the multi-scale characteristic fusion.
In order to show that the hyperspectral image classification method based on the multi-scale space spectrum feature and the multi-dimensional feature fusion network has excellent classification capability, the multi-scale space spectrum feature and the multi-dimensional feature fusion network are used for classifying data of two hyperspectral images of Indian pines and Pavia unity in a simulation experiment 2, and classification results are compared with classification results of three existing hyperspectral image classification methods.
Simulation experiment 1:
the hyperspectral image data of Indian pines and Pavia unity are classified respectively by using the hyperspectral image classification method based on the single scale network. Wherein, the three single-scale networks are three branches of the multi-scale space spectrum feature and multi-dimensional feature fusion network in the invention. The simulation experiment 1 of the invention compares the multi-scale space spectrum characteristic and the multi-dimensional characteristic fusion network with three single-scale networks on the classification result respectively to prove the effectiveness of the multi-scale characteristic fusion.
In order to quantify the classification results, the following 3 evaluation indexes were used in experiment one.
(1) The overall accuracy OA (overall accuracycacy) is obtained by dividing the number of correctly classified pixels on the test set by the total number of pixels, and the value of the overall accuracy OA is between 0% and 100%, and the larger the value is, the better the classification effect is.
(2) The average precision AA (average accuracy) is obtained by dividing the number of correctly classified pixels of each type on the test set by the total number of all pixels of the type, the average value of the precisions of all the types is called as the average precision AA, the average precision AA is 0-100%, and the larger the value is, the better the classification effect is.
(3) Kappa (kappa coefficient) coefficient: the Kappa coefficient is an evaluation index defined on the confusion matrix X, elements on a diagonal line and elements deviating from the diagonal line of the confusion matrix are comprehensively considered, the classification performance of the algorithm is objectively reflected, the value of Kappa is in the range of-1, and the larger the value is, the better the classification effect is.
The classification performance of the two classification methods was evaluated using the above evaluation indexes, and the results are shown in table 3.
TABLE 3 comparison of the classification accuracy of three single-scale networks with the present invention
As can be seen from table 3, in the classification result of Indian pings data, the small neighborhood blocks 7 × 7 are significantly larger in AA than OA, which indicates that the small neighborhood blocks have better classification performance for samples with smaller number of ground object categories. And in the classification result of the large-scale neighborhood blocks of 15 × 15, OA is obviously greater than AA, which indicates that the classification performance of the large-scale neighborhood blocks on samples with larger number of ground object categories is better. The method and the device have the advantages that the features extracted by a plurality of scales are fused, the advantages of all scales are combined, and a more excellent classification result is obtained.
Meanwhile, on the classification result of the Pavia unity data, the multi-scale space spectrum feature and multi-dimensional feature fusion network provided by the invention is obviously superior to three single-scale networks on three classification precision indexes, and the invention has stronger generalization capability and robustness.
Simulation experiment 2:
in order to verify the effectiveness of the method provided by the invention, under the same condition, the classification results of the method on the hyperspectral datasets Indian pines are compared with the classification results of the three existing classification methods in the hyperspectral field, and the results are shown in table 4.
These three existing methods are:
1) a classical Support Vector Machine (SVM) is used for the method for classifying hyperspectral images, and spectral information is directly classified through the SVM.
2) A method for classifying Hyperspectral images using an end-to-end spectrum-space Residual error Network (SSRN) is proposed in the published paper "Spectral-Spatial information Network for Hyperspectral Image Classification" A3-D Deep Learning frame "(IEEE Transactions on Geoscience and Remote Sensing,2017: 1-12). The method takes an original three-dimensional cube as input data and does not need characteristic engineering. In an end-to-end spectrum-space residual error network, a spectrum and space residual error block continuously learns identification characteristics from abundant spectrum characteristics and space backgrounds in a hyperspectral image, spectrum characteristics obtained by a three-dimensional convolutional neural network and space characteristics obtained by a two-dimensional convolutional neural network are fused in a cascading mode, and finally the fused characteristics are input into a classification layer to classify the hyperspectral image.
3) Patent document "hyperspectral classification method based on fusion of spatial coordinates and spatial spectral features" (patent application No.: 201710644479.6, application publication number: CN 107451614A) proposes a hyperspectral image classification method SPE-SPA-SVM based on fusion of space coordinates and space spectrum features.
The comparison of the overall classification accuracy OA, the average classification accuracy AA and the Kappa coefficient of the two hyperspectral datasets of the three prior art is shown in table 4.
TABLE 4 comparison of prior art and present invention in terms of classification accuracy
As can be seen from Table 3, on the Indian pings dataset, the classification results of the present invention are significantly superior to the three prior art techniques in all 3 indicators regarding classification accuracy.
The method fuses the features extracted by the three-dimensional convolutional neural network and the features extracted by the two-dimensional convolutional neural network in a self-mapping mode, realizes the fusion of spectral features and spatial features, low-dimensional features and high-dimensional features, and greatly improves the classification capability of the network. Meanwhile, the multi-scale fusion idea provided by the invention greatly improves the problem of poor classification performance of a single-scale network, so that the classification performance of the method is obviously superior to that of other three existing classification methods.
By combining the result analysis of the simulation experiment 1 and the simulation experiment 2, the method provided by the invention can effectively solve the problems that the convolutional neural network has too single characteristics and too single scale during training, and can solve the problem that the average classification accuracy AA is low during hyperspectral classification.
Claims (3)
1. A hyperspectral image classification method based on fusion of multi-scale multi-dimensional space spectrum features is characterized in that a multi-scale space spectrum feature and multi-dimensional feature fusion network which is composed of three feature extraction branches with the same structure and a combination classifier is constructed, and hyperspectral image classification is carried out by combining the multi-scale space spectrum features and the multi-dimensional features of hyperspectral images; the method comprises the following specific steps:
(1) inputting a hyperspectral image:
inputting a hyperspectral image which is a three-dimensional feature cube
Each wave band in the hyperspectral image corresponds to a two-dimensional matrix in the feature cube
Wherein e represents belonging to a symbol,
the method comprises the steps of representing a real number field symbol, wherein m represents the length of a hyperspectral image, n represents the width of the hyperspectral image, b represents the number of spectral wave bands of the hyperspectral image, i represents the serial number of the spectral wave bands in the hyperspectral image, and i is 1,2, … and b;
(2) preprocessing a hyperspectral image to be classified:
(2a) converting an m × n × b three-dimensional hyperspectral image matrix into an a × b two-dimensional feature matrix, wherein a is m × n, each column in the two-dimensional feature matrix represents a spectral dimension, and each row represents all spectral information of each sample;
(2b) performing normalization processing on the two-dimensional feature matrix by adopting a normalization formula;
(2c) converting the normalized two-dimensional feature matrix into a normalized three-dimensional feature matrix with the same size as the original hyperspectral image;
(3) neighborhood block fetching:
(3a) performing 0-pixel edge filling operation on the normalized three-dimensional feature matrix, wherein the sizes of 0 pixels of the edge filling are respectively 3, 5 and 7;
(3b) in the filled hyperspectral image, with each pixel point as the center, selecting 7 × 7, 11 × 11 and 15 × 15 neighborhood blocks respectively to obtain neighborhood blocks of three scales;
(4) generating a training set and a testing set:
(4a) respectively distributing the neighborhood blocks of the three scales to a set to which the class belongs according to the class of the central pixel point of the neighborhood blocks;
(4b) respectively taking the neighborhood blocks of known class labels in the neighborhood blocks in each class of sets as training sets, taking the central pixel point label of each neighborhood block as the label of the neighborhood block, and respectively taking the residual neighborhood blocks in each class of sets as test sets;
(5) constructing a multi-scale space spectrum feature and multi-dimensional feature fusion network:
(5a) three feature extraction branches with the same structure are respectively built for feature extraction of three input neighborhood blocks with different scales, and the structure of each branch is as follows: the first three-dimensional convolution layer → the first normative layer → the first activation function layer → the second three-dimensional convolution layer → the second normative layer → the second activation function layer → the third three-dimensional convolution layer → the third normative layer → the third activation function layer → the first two-dimensional convolution layer → the fourth normative layer → the fourth activation function layer → the first fusion layer → the second two-dimensional convolution layer → the fifth normative layer → the fifth activation function layer → the second fusion layer → the third two-dimensional convolution layer → the sixth normative layer → the sixth activation function layer → the first maximum pooling layer → the fourth two-dimensional convolution layer → the seventh normative layer → the seventh activation function layer → the second maximum pooling layer; the first fusion layer fuses the first activation function layer and the fourth activation function layer through an addition operation; the second fusion layer fuses the second activation function layer and the fifth activation function layer through an addition operation;
(5b) three feature extraction branches with the same structure are laminated and fused through a concatenate and then are sequentially connected with a global average pooling layer and an output layer to form a multi-scale space spectrum feature and multi-dimensional feature fusion network;
(5c) the parameters of the multi-scale space spectrum feature and the multi-dimensional feature fusion network are set as follows: setting the number of the neurons of the first, second and third three-dimensional convolutional layers to be 24, setting the sizes of the convolutional cores to be (1, 1, 20), (1, 1, 3) and (1, 1, 10) in sequence, and setting the sizes of the convolutional steps to be 20, 1 and 1 in sequence; the number of the neurons of the first, second, third and fourth two-dimensional convolutional layers is set to be 240, 24 and 24 in sequence, the lengths of the convolutional kernels are all set to be 3, and the convolution step length is all set to be 1; setting the pooling length of each maximum pooling layer to be 3; setting the momentum factor of each specification layer to be 0.8; setting the pooling length of the global average pooling layer to 7; setting the activation function of each activation function layer as a ReLU activation function; setting the number of neurons in an output layer as the number of categories, and selecting a softmax function as an activation function;
(6) training a multi-scale space spectrum feature and multi-dimensional feature fusion network:
inputting the training set and the labels of the training set into a multi-scale space-spectrum feature and multi-dimensional feature fusion network for training to obtain a trained multi-scale space-spectrum feature and multi-dimensional feature fusion network;
(7) classifying the test samples:
inputting the test set into the trained multi-scale space spectrum feature and multi-dimensional feature fusion network, outputting a class label of the test sample, and using the output of the multi-scale space spectrum feature and multi-dimensional feature fusion network as a prediction label of the test sample to finish the hyperspectral image classification.
2. The hyperspectral image classification method based on fusion of multi-scale multi-dimensional space-spectral features according to claim 1 is characterized in that the normalization formula in the step (2b) is as follows:
wherein the content of the first and second substances,
representing the jth surface feature target in the ith spectral band in the normalized two-dimensional feature matrix,
represents the average pixel value of all the ground object targets in the ith spectral band in the two-dimensional feature matrix,
and representing the variance value of all the ground object target pixel values in the ith spectral band in the two-dimensional characteristic matrix.
3. The hyperspectral image classification method based on fusion of multi-scale and multi-dimensional space-spectrum features according to claim 1 is characterized in that the specific operation steps of inputting the labels of the training set and the training set into the fusion network of the multi-scale and multi-dimensional features for training in the step (6) are as follows:
firstly, respectively inputting a training set into feature extraction branches of three different neighborhood blocks of a multi-scale space spectrum feature and multi-dimensional feature fusion network, and outputting a prediction label vector of a training sample;
secondly, optimizing network parameters by using a loss function of a multi-scale space spectrum characteristic and multi-dimensional characteristic fusion network by adopting a gradient descent method until the network parameters are converged, wherein the learning rate of the loss function is set to be 0.0001;
thirdly, calculating the cross entropy between the predicted label vector and the real label vector by using the cross entropy formula;
where L represents the cross entropy between the predicted tag vector and the true tag vector, Σ represents the summation operation, yiRepresenting in a vector of predictive labelsThe ith element, ln, represents a logarithmic operation based on a natural constant e,
representing the mth element in the prediction tag vector.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910616371.5A CN110321963B (en) | 2019-07-09 | 2019-07-09 | Hyperspectral image classification method based on fusion of multi-scale and multi-dimensional space spectrum features |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910616371.5A CN110321963B (en) | 2019-07-09 | 2019-07-09 | Hyperspectral image classification method based on fusion of multi-scale and multi-dimensional space spectrum features |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110321963A CN110321963A (en) | 2019-10-11 |
| CN110321963B true CN110321963B (en) | 2022-03-04 |
Family
ID=68123179
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910616371.5A Active CN110321963B (en) | 2019-07-09 | 2019-07-09 | Hyperspectral image classification method based on fusion of multi-scale and multi-dimensional space spectrum features |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110321963B (en) |
Families Citing this family (34)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110991236A (en) * | 2019-10-29 | 2020-04-10 | 成都华为技术有限公司 | Image classification method and related device |
| CN111008664B (en) * | 2019-12-05 | 2023-04-07 | 上海海洋大学 | Hyperspectral sea ice detection method based on space-spectrum combined characteristics |
| CN111126256B (en) * | 2019-12-23 | 2022-02-15 | 武汉大学 | Hyperspectral image classification method based on self-adaptive space-spectrum multi-scale network |
| CN111160273B (en) * | 2019-12-31 | 2023-05-09 | 北京云智空间科技有限公司 | Hyperspectral image spatial spectrum joint classification method and device |
| CN111191736B (en) * | 2020-01-05 | 2022-03-04 | 西安电子科技大学 | Hyperspectral image classification method based on depth feature cross fusion |
| CN111368896B (en) * | 2020-02-28 | 2023-07-18 | 南京信息工程大学 | Hyperspectral remote sensing image classification method based on dense residual three-dimensional convolutional neural network |
| CN111539447B (en) * | 2020-03-17 | 2022-04-22 | 广东省科学院智能制造研究所 | Hyperspectrum and terahertz data depth fusion-based classification method |
| CN111368940A (en) * | 2020-03-25 | 2020-07-03 | 西北工业大学 | Hyperspectral image classification method based on guided filtering and kernel extreme learning machine |
| CN111563528B (en) * | 2020-03-31 | 2022-03-11 | 西北工业大学 | SAR image classification method based on multi-scale feature learning network and bilateral filtering |
| CN111652038B (en) * | 2020-04-13 | 2023-03-03 | 上海海洋大学 | Remote sensing sea ice image classification method based on convolutional neural network |
| CN111723838A (en) * | 2020-04-21 | 2020-09-29 | 中国人民解放军火箭军工程大学 | Hyperspectral terrain classification method integrating multi-core machine learning and multi-scale features |
| CN111639587B (en) * | 2020-05-27 | 2023-03-24 | 西安电子科技大学 | Hyperspectral image classification method based on multi-scale spectrum space convolution neural network |
| CN111695467B (en) * | 2020-06-01 | 2023-05-30 | 西安电子科技大学 | Spatial spectrum full convolution hyperspectral image classification method based on super-pixel sample expansion |
| CN111723731B (en) * | 2020-06-18 | 2023-09-29 | 西安电子科技大学 | Hyperspectral image classification method, storage medium and equipment based on spatial spectrum convolution kernel |
| CN112052755B (en) * | 2020-08-24 | 2023-06-02 | 西安电子科技大学 | Semantic convolution hyperspectral image classification method based on multipath attention mechanism |
| CN112052758B (en) * | 2020-08-25 | 2023-05-23 | 西安电子科技大学 | Hyperspectral image classification method based on attention mechanism and cyclic neural network |
| CN112101271A (en) * | 2020-09-23 | 2020-12-18 | 台州学院 | Hyperspectral remote sensing image classification method and device |
| CN112215267B (en) * | 2020-09-25 | 2022-11-01 | 天津大学 | Hyperspectral image-oriented depth space spectrum subspace clustering method |
| CN112577744B (en) * | 2020-11-02 | 2021-12-03 | 西南交通大学 | Rolling bearing fault identification method based on combination of SPA-map and ResNet |
| CN112308152B (en) * | 2020-11-03 | 2023-04-28 | 西安电子科技大学 | Hyperspectral image ground object classification method based on spectrum segmentation and homogeneous region detection |
| CN112541892A (en) * | 2020-12-07 | 2021-03-23 | 山东众阳健康科技集团有限公司 | Cerebral aneurysm image segmentation method based on 3D activation function |
| CN112733659B (en) * | 2020-12-30 | 2022-09-20 | 华东师范大学 | Hyperspectral image classification method based on self-learning double-flow multi-scale dense connection network |
| CN112733736A (en) * | 2021-01-13 | 2021-04-30 | 西安电子科技大学 | Class imbalance hyperspectral image classification method based on enhanced oversampling |
| CN112733769B (en) * | 2021-01-18 | 2023-04-07 | 西安电子科技大学 | Hyperspectral image classification method based on multiband entropy rate superpixel segmentation |
| CN112766199B (en) | 2021-01-26 | 2022-04-29 | 武汉大学 | Hyperspectral image classification method based on self-adaptive multi-scale feature extraction model |
| CN112949416B (en) * | 2021-02-04 | 2022-10-04 | 东华理工大学 | Supervised hyperspectral multiscale graph volume integral classification method |
| CN113095409B (en) * | 2021-04-13 | 2023-04-07 | 西安电子科技大学 | Hyperspectral image classification method based on attention mechanism and weight sharing |
| CN113392931B (en) * | 2021-07-02 | 2023-02-10 | 西安电子科技大学 | Hyperspectral open set classification method based on self-supervision learning and multitask learning |
| CN113361485B (en) * | 2021-07-08 | 2022-05-20 | 齐齐哈尔大学 | Hyperspectral image classification method based on spectrum space attention fusion and deformable convolution residual error network |
| CN113837314A (en) * | 2021-10-09 | 2021-12-24 | 陕西科技大学 | Hyperspectral image classification method based on hybrid convolutional neural network |
| CN113963207B (en) * | 2021-10-21 | 2024-03-29 | 江南大学 | Hyperspectral image classification method based on spatial spectrum information feature guided fusion network |
| CN114022960B (en) * | 2022-01-05 | 2022-06-14 | 阿里巴巴达摩院(杭州)科技有限公司 | Model training and behavior recognition method and device, electronic equipment and storage medium |
| CN115240074B (en) * | 2022-09-22 | 2023-08-11 | 山东锋士信息技术有限公司 | Hyperspectral image classification method and equipment based on covariance representation |
| CN116310572B (en) * | 2023-03-23 | 2024-01-23 | 齐齐哈尔大学 | Pyramid multi-scale convolution and self-attention combined hyperspectral image classification method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105740773A (en) * | 2016-01-25 | 2016-07-06 | 重庆理工大学 | Deep learning and multi-scale information based behavior identification method |
| CN107316013A (en) * | 2017-06-14 | 2017-11-03 | 西安电子科技大学 | Hyperspectral image classification method with DCNN is converted based on NSCT |
| WO2018052586A1 (en) * | 2016-09-14 | 2018-03-22 | Konica Minolta Laboratory U.S.A., Inc. | Method and system for multi-scale cell image segmentation using multiple parallel convolutional neural networks |
| CN109726757A (en) * | 2018-12-26 | 2019-05-07 | 山东女子学院 | A kind of DCNN hyperspectral image classification method merging multiscale space feature |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10417529B2 (en) * | 2015-09-15 | 2019-09-17 | Samsung Electronics Co., Ltd. | Learning combinations of homogenous feature arrangements |
-
2019
- 2019-07-09 CN CN201910616371.5A patent/CN110321963B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105740773A (en) * | 2016-01-25 | 2016-07-06 | 重庆理工大学 | Deep learning and multi-scale information based behavior identification method |
| WO2018052586A1 (en) * | 2016-09-14 | 2018-03-22 | Konica Minolta Laboratory U.S.A., Inc. | Method and system for multi-scale cell image segmentation using multiple parallel convolutional neural networks |
| CN107316013A (en) * | 2017-06-14 | 2017-11-03 | 西安电子科技大学 | Hyperspectral image classification method with DCNN is converted based on NSCT |
| CN109726757A (en) * | 2018-12-26 | 2019-05-07 | 山东女子学院 | A kind of DCNN hyperspectral image classification method merging multiscale space feature |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110321963A (en) | 2019-10-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110321963B (en) | Hyperspectral image classification method based on fusion of multi-scale and multi-dimensional space spectrum features | |
| CN111191736B (en) | Hyperspectral image classification method based on depth feature cross fusion | |
| Mei et al. | Hyperspectral image classification using group-aware hierarchical transformer | |
| CN108460342B (en) | Hyperspectral image classification method based on convolutional neural network and cyclic neural network | |
| CN112052755B (en) | Semantic convolution hyperspectral image classification method based on multipath attention mechanism | |
| CN107358260B (en) | Multispectral image classification method based on surface wave CNN | |
| CN107145836B (en) | Hyperspectral image classification method based on stacked boundary identification self-encoder | |
| CN110084159A (en) | Hyperspectral image classification method based on the multistage empty spectrum information CNN of joint | |
| CN103208011B (en) | Based on average drifting and the hyperspectral image space-spectral domain classification method organizing sparse coding | |
| CN103440505B (en) | The Classification of hyperspectral remote sensing image method of space neighborhood information weighting | |
| CN113095409B (en) | Hyperspectral image classification method based on attention mechanism and weight sharing | |
| CN108154094B (en) | Hyperspectral image unsupervised waveband selection method based on subinterval division | |
| CN112308152B (en) | Hyperspectral image ground object classification method based on spectrum segmentation and homogeneous region detection | |
| CN111639587B (en) | Hyperspectral image classification method based on multi-scale spectrum space convolution neural network | |
| CN110084311B (en) | Hyperspectral image wave band selection method based on ternary weight convolution neural network | |
| CN105184314B (en) | Wrapper formula EO-1 hyperion band selection methods based on pixel cluster | |
| CN111814685A (en) | Hyperspectral image classification method based on double-branch convolution self-encoder | |
| CN109359623A (en) | High spectrum image based on depth Joint Distribution adaptation network migrates classification method | |
| CN104809471B (en) | A kind of high spectrum image residual error integrated classification method based on spatial spectral information | |
| CN109034213B (en) | Hyperspectral image classification method and system based on correlation entropy principle | |
| CN115240072A (en) | Hyperspectral multi-class change detection method based on multidirectional multi-scale spectrum-space residual convolution neural network | |
| CN113705580A (en) | Hyperspectral image classification method based on deep migration learning | |
| CN113435254A (en) | Sentinel second image-based farmland deep learning extraction method | |
| Qi et al. | Global-local three-dimensional convolutional transformer network for hyperspectral image classification | |
| Ge et al. | Pyramidal multiscale convolutional network with polarized self-attention for pixel-wise hyperspectral image classification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |