CN114373080B

CN114373080B - Hyperspectral classification method of lightweight hybrid convolution model based on global reasoning

Info

Publication number: CN114373080B
Application number: CN202210279207.1A
Authority: CN
Inventors: 王武礼; 马晓虎; 冷林椿; 王歌; 李冲; 姜雨蒙
Original assignee: China University of Petroleum East China
Current assignee: China University of Petroleum East China
Priority date: 2022-03-22
Filing date: 2022-03-22
Publication date: 2022-07-29
Anticipated expiration: 2042-03-22
Also published as: CN114373080A

Abstract

The invention discloses a hyperspectral classification method of a lightweight mixed convolution model based on global reasoning, which belongs to the technical field of information processing and is mainly used for hyperspectral image classification of small sample data. The lightweight mixed convolution model based on the global reasoning comprises a layer of two-dimensional convolution, a layer of three-dimensional convolution and a global reasoning module; and a global reasoning module is added, and global characteristics and deep characteristic information of the hyperspectral image are effectively extracted through reasoning on context relations among different regions so as to replace characteristic extraction of deep three-dimensional convolution, so that the complexity and the calculation cost of the model are greatly reduced. The test results in the public data set show that the classification performance of the method is superior to that of the best classification method at present, only a small number of training samples are needed, the space-spectrum combined characteristic of the hyperspectral image can be effectively extracted, and the problems of channel relation information loss caused by only using two-dimensional convolution and the problems of greatly increased model complexity and calculation cost caused by adopting deep three-dimensional convolution are solved.

Description

Hyperspectral classification method of lightweight hybrid convolution model based on global reasoning

Technical Field

The invention discloses a hyperspectral classification method of a lightweight hybrid convolution model based on global reasoning, and belongs to the technical field of information processing.

Background

The hyperspectral image (HSI) brings great help to the extraction of the ground feature information due to the fact that the hyperspectral image has numerous spectral bands, redundancy of the ground feature information is caused by a large amount of spectral data, meanwhile, the relevance among the bands is strengthened due to the fact that the spectral resolution is improved, and therefore the hyperspectral image classification challenge is brought. Especially when there are fewer training samples, it will bring more difficulty to the classification of HSI.

Currently, there are two main categories of HSI classification: traditional methods based on spectral information and methods based on deep learning. The traditional method mainly utilizes spectral feature information of different ground features to classify, wherein a support vector machine, k neighbor, random forest and the like are most representative. In order to obtain better classification performance, methods such as Principal Component Analysis (PCA), Independent Component Analysis (ICA), and Linear Discriminant Analysis (LDA) are applied to efficiently perform data dimensionality reduction or feature extraction. Although the traditional HSI classification method achieves good classification performance, the traditional HSI classification method has limited feature expression capability because the traditional HSI classification method is processed by depending on manually made features or extracted shallow features to a large extent, and cannot be better adapted to complex HSI classification tasks.

In recent years, deep learning has been successfully applied to a plurality of computer vision tasks such as image classification, image segmentation, object detection and the like due to its strong feature expression capability. The great application potential enables the deep learning method to achieve a remarkable effect in the field of HSI classification, wherein the HSI classification method based on the Convolutional Neural Network (CNN) is the most popular. Hu et al used deep CNN to directly classify hyperspectral images in the spectral domain for the first time, and obtained a good classification effect. Konstatinos Makantasis et al use 2D-CNN to extract spatial features of the neighborhood of the hyperspectral pixel, and meanwhile, the PCA method is adopted to effectively reduce the dimension, and the calculation cost of the model is reduced. However, the model can only extract spatial feature information, and cannot obtain relationship information between spectral channels. Yushi Chen et al uses the 3D-CNN method for HSI task, and effectively improves the classification precision by extracting the spectrum-space combined features through the 3D-CNN. On the basis, M.He et al propose a multi-scale depth 3D-CNN model, which can learn multi-scale spatial features and spectral features from HSI data end to end, and further improve the classification performance of HSI.

However, the HSI classification using the 3D-CNN model requires deep convolutional layers to effectively extract the spatio-spectral joint features, which results in a great increase in model complexity and training sample number. Based on this, researchers have begun to combine 2D-CNN with 3D-CNN for HSI classification. In 2020, SwalpaKumarRoy et al provides an HSI classification method based on a hybrid SN model, the method combines 3D-CNN and 2D-CNN, and fully extracts spatial characteristic information and spectral characteristic information, compared with the HSI classification method which is carried out by singly using 3D-CNN, the HSI classification method has the advantages that the calculation cost and the classification performance are greatly improved. In 2021, SaeedGhaderizadeh et al proposed a 3D-2DCNN model. The model has higher robustness and efficiency by introducing a three-dimensional depth scrollable block and a fast scrollable block. However, it is worth noting that in order to make the model have stronger spatial feature extraction capability, the model often includes a plurality of even deep 3D convolution layers, and in literature, up to three layers of 3D convolution are adopted for feature extraction. The method for extracting deep features through stacking of multilayer 3D convolution increases the complexity and the calculation cost of a model on one hand, and also requires a large number of samples for training on the other hand, and for HSI, the labeling of the large number of samples is a small challenge.

Disclosure of Invention

The invention aims to provide a hyperspectral classification method of a lightweight hybrid convolution model based on global reasoning, and aims to solve the problems of complex model, high calculation cost and large sample requirement caused by the fact that the hyperspectral classification method comprises a plurality of three-dimensional convolution layers in the prior art.

A hyperspectral classification method of a lightweight hybrid convolution model based on global reasoning comprises the following steps:

s1, setting original hyperspectral data as I belongs to RM multiplied by N multiplied by D, wherein I is the original hyperspectral classified input, M and N are the width and height of a space dimension, and D is the number of spectral bands;

s2, reducing the dimension of the hyperspectral classified spectral information by adopting principal component analysis, representing hyperspectral data subjected to principal component analysis as X belongs to RM multiplied by N multiplied by B, keeping the spatial dimension unchanged, and changing the number of spectral bands from D to B;

s3, extracting neighborhood blocks of the hyperspectral classification for preprocessing, extracting a color spot for each pixel of the hyperspectral classification by taking the pixel as a central point, wherein the label of the color spot is the label of the central pixel and is expressed as Z ∈ RS multiplied by S multiplied by B, the width and height of the color spot are S, and the number of spectral bands is B;

and S4, sending the hyperspectral classified data into a lightweight hybrid convolution model based on global reasoning for processing.

Preferably, the lightweight hybrid convolution model based on global reasoning comprises a two-dimensional convolution neural network, a three-dimensional convolution neural network and a global reasoning module.

Preferably, the three-dimensional convolutional neural network employs 8 convolution kernels of 3 × 3 × 7, the spatial dimension is 3 × 3, and the spectral dimension is 7.

Preferably, the two-dimensional convolutional neural network employs 16 convolution kernels of 3 × 3, and the spatial dimension is 3 × 3.

Preferably, the global inference module is two-dimensional, and the construction method thereof is that the feature aggregated by disjoint areas on the coordinate space of the input data is projected into a potential interaction space, and is represented by a single feature, i.e. a node, a graph convolution mode is applied to model and infer the context relationship between each pair of nodes, and then the context relationship is back projected, and the feature with relationship information is transformed back to the original coordinate space to obtain the related global feature information.

Preferably, step S4 includes:

s4.1, performing combined extraction of spatial features and spectral features on the preprocessed hyperspectral classified data by adopting a three-dimensional convolutional neural network;

s4.2, performing combined extraction of spatial features and spectral features on the hyperspectral classified data processed in the S4.1 by adopting a two-dimensional convolutional neural network;

And S4.3, adopting a global reasoning module to perform global feature extraction on the hyperspectral classified data processed by the S4.2.

Compared with the prior art, the method only uses the lightweight hybrid convolution network model of one layer of 3D-CNN and one layer of 2D-CNN, reduces the complexity of the network model, and can effectively extract the space-spectrum combined characteristics of HSI; a lightweight global reasoning module is added in a designed hybrid convolutional network model, the global characteristics of HSI are effectively extracted by reasoning the context relation among different regions, and the classification performance of HSI is remarkably improved; the proposed model not only reduces the complexity of the existing hybrid network model, but also can obtain very good classification performance when only few samples are required for training.

Drawings

FIG. 1 is a block diagram of a lightweight hybrid convolution model based on global reasoning in accordance with the present invention;

FIG. 2 is a conceptual framework diagram of a global inference module.

Detailed Description

The present invention will be described in further detail with reference to specific embodiments below:

the embodiment firstly introduces the overall structure of a lightweight hybrid convolution model based on global reasoning, then introduces the structure and parameter design of a 3D-CNN and 2D-CNN hybrid convolution model, and finally provides the core idea of a lightweight global reasoning module.

Deep convolutional neural network model: a single deep two-dimensional convolution or three-dimensional convolution network is used as a feature extraction model for hyperspectral classification, and the missing spectrum channel relation or the high model complexity can be caused. The number of layers of 3D-CNN in the existing hybrid convolutional network model for hyperspectral classification is also large, and the complexity, the calculation cost and the labeling cost of samples of the model are increased to a certain extent. According to the invention, a high-performance hyperspectral classification effect can be obtained under the condition of a small number of samples only by combining one layer of 3D-CNN and one layer of 2D-CNN and adding a global reasoning module.

Let the original hyperspectral data be represented as I ∈ R ^M×N×D Wherein, I is the original high spectrum classification input, the width and height of M and N space dimensions, and D is the number of spectral wave bands, which generally has dozens or even hundreds of wave bands. High spectral Classification numerousThe band information not only brings larger calculation cost to subsequent processing tasks, but also reduces the overall classification precision of hyperspectral classification along with the increase of feature dimension when training samples are limited, namely the problem of dimension disaster. Therefore, it is imperative to reduce the spectral dimensions of hyperspectral classification and reduce the redundancy of information. Principal Component Analysis (PCA) is used as a common dimension reduction mode, and the core idea is to calculate the feature similarity between different data and extract main features. The invention adopts a PCA method to reduce the dimension of hyperspectral classified spectral information so as to effectively solve the problems of high calculation cost and low classification precision caused by information redundancy. Expressed as X epsilon R after PCA treatment ^M×N×B The spatial dimension remains unchanged and the number of spectral bands is changed from D to B.

In order to better utilize a subsequent convolutional network for classification processing, after PCA dimension reduction, neighborhood block extraction of hyperspectral classification is preprocessed. Namely, for each pixel of the hyperspectral classification, a patch is extracted by taking the pixel as a central point. The label of each patch is the label of the central pixel and is expressed as Z belonging to R ^S×S×B The width and height of patch is S, and the number of spectral bands is B.

And (3) after preprocessing of the hyperspectral classified data by PCA and neighborhood block extraction, sending the hyperspectral classified data into a lightweight hybrid convolution model based on global reasoning for processing. The single-layer 3D-CNN and the 2D-CNN are mixed to complete the combined extraction of spectral and spatial features and reduce the complexity of a model, and the global reasoning module is used for reasoning the context relationship among different regions to extract global features, so that the problem that a multilayer 3D-CNN with high calculation cost is needed for extracting features with stronger expression capacity is solved.

3D-2D CNN: the hyperspectral classification is volume data including dozens of even hundreds of wave bands and contains spatial information and spectral information. For the traditional 2D-CNN, only the spatial feature information can be extracted; and the 3D-CNN can extract spatial features and spectral features simultaneously. Therefore, in recent years, hyperspectral classification based on 3D-CNN becomes the mainstream, but the depth 3D-CNN model is complex, so that the computation cost of the model is high and a large number of training samples are required. Several existing hybrid convolutions for hyperspectral classification The model has good classification performance and can reduce the number of training samples to a certain extent. However, the problem of increased computation cost caused by multi-layer 3D-CNN stacking is still not negligible, and the problem of large number of training samples cannot be thoroughly solved. In order to ensure the integrity of the spatial information and the spectral information of the hyperspectral classified data, the 3D-CNN is adopted to perform combined extraction of spatial features and spectral features on the preprocessed hyperspectral classified data. In the 3D-CNN layer, 8 convolution kernels of 3 × 3 × 7 are adopted, namely the spatial dimension is 3 × 3 and the spectral dimension is 7. Is represented by K in FIG. 1 ₁ ¹ =3，K ₂ ¹ =3，K ₃ ¹ And = 7. In order to fully extract the spatial feature information of the hyperspectral classified data, after the 3D-CNN is adopted for feature extraction, a layer of 2D-CNN is used for extraction, so that the classification effect of hyperspectral classification is improved. In 2D-CNN, 16 convolution kernels of 3 × 3 are used, i.e., the spatial dimension is 3 × 3. In FIG. 1, denoted K ₁ ³ =3，K ₂ ³ =3。

A single convolutional layer can better capture the local relationship covered by the convolutional kernel, but the relationship between global regions with stronger expression capability needs to stack multiple convolutional layers even deeply, which undoubtedly increases the difficulty and cost of CNN global inference. The existing several hybrid convolution models for hyperspectral classification are designed in such an inefficient multilayer 3D-CNN stacking mode to extract global features. The global reasoning module added in the hybrid convolution model provided by the invention aims to overcome the inherent defect of multilayer convolution operation on global relationship modeling.

The global reasoning module is a lightweight global reasoning module, extracts relevant global and deep features by modeling and reasoning global relations among regions, and a principle framework of the global reasoning module is shown in fig. 2, which represents projection and back projection between an interaction space and a coordinate space. In this way, the relational reasoning is reduced to modeling the interaction between nodes on the smaller graph shown at the top of FIG. 2. Graph convolution is then applied to model and infer the context between each pair of nodes. Finally, the relative global feature information can be obtained by performing back projection on the image and converting the features with the relationship information back to the original coordinate space.

After the spatial-spectral combined features of the hyperspectral classified data are extracted through the 3D-CNN layer, complete spatial information and spectral information exist, and global relationship reasoning is facilitated for hyperspectral classification. Therefore, a module is added after the 3D-CNN layer to extract the global features of hyperspectral classification. Due to the fact that the complexity of the 3D-level module is high, the 2D-level module is adopted to aggregate global features of the hyperspectral classified data, and the global features are matched with the spatial feature information extracted from the subsequent 2D-CNN layer. Full structural combination experiments verify that the model classification performance of the 3D-CNN + +2D-CNN combination is the best. The total number of trainable weight parameters in the structural parameters of the proposed model for the IP dataset is 2, 275, 504, as in table 1.

Table 1 summary of models used by the invention for IP datasets

The examples used a common dataset of three hyperspectral classifications, namely, Salinas Scene (SA), University of Pavia (UP), and Indian Pines (IP), as the experimental dataset. The SA data set has 16 categories with a spatial dimension of 512X 217 and 224 spectral bands in the wavelength range of 360-2500 nm. After 20 water absorption bands are removed, 204 spectral bands are reserved. The UP dataset has 9 categories with spatial dimensions 610X 340 and 103 spectral bands in the wavelength range of 430-860 nm. The IP data set has 16 categories, the spatial dimension is 145 x 145, and in the wavelength range of 400-2500nm, there are 224 spectral bands. After 24 water absorption bands are removed, 200 spectral bands are reserved.

All experiments were performed on a synergetics-Y7000P computer. The computer was equipped with a GTX 1660ti Graphics Processor (GPU) and 16 GB RAM. Using the SGD optimizer, the learning rate was set to 0.01. And regularized using momentum and L2 with parameters set to 0.8 and 0.0005, respectively. Mini-batches of size 32 were used in model training. In addition, in order to accelerate the training process and improve the generalization performance, batch normalization processing is carried out on the model. The number of iterations of the network model training is set to 50. For a fair comparison with other methods, the same neighborhood blocks are extracted from the PCA dimension-reduced data for different datasets, with sizes of 27 × 27 × 20.

The hyperspectral classification performance was judged using a Kappa coefficient (Kappa), Overall Accuracy (OA), and Average Accuracy (AA) as evaluation indexes. Kappa reflects consistency information between information; OA refers to the proportion of correctly classified numbers; AA means the average of the correct classification ratio for each class.

Of the different effects of Window Size and Spectral Dimension on the performance of the LH-CNN model, the best results are obtained for the neighborhood block with 27X 20 Window Size and Spectral Dimension, as shown in tables 2 and 3.

TABLE 2 Effect of spatial Window size on model Performance

TABLE 3 Effect of spectral dimensionality on model Performance

In order to verify the classification performance of the designed model under the condition of a small amount of training samples, the method is compared with a CNN method commonly used for hyperspectral classification and the best mixed convolution method at present, namely 2-D-CNN, 3-D-CNN, multi-scale 3-D deep connected neural network (M3D) -CNN, hybrid SN and 3D-2D CNN. Training samples for each method were randomly selected from the data set at a sample rate of 1%. With only 1% training data, the model proposed by the present invention performed better than the other methods, as shown in table 4.

TABLE 4 Classification accuracy (percentage) for IP, UP and SA datasets (1% sample for training)

Particularly, in the experiment on the IP data set with a small sample size, the performance of the proposed model far exceeds that of the best mixed CNN model in the aspects of three evaluation indexes of Kappa, OA and AA. The complexity of the mixed CNN model and the calculation cost in different data sets are optimized, and the global reasoning module has obvious influence on the high-spectrum classification performance.

In order to further verify the classification performance of the proposed model under the condition of a small number of training samples and reduce the number of the training samples, 0.5% of sample data is randomly selected from UP and SA data sets to serve as the training samples (the IP data set cannot select 0.5% of sample data because of the small number of part of class samples), the designed model has the best performance and far exceeds the classification results of the best two mixed CNNs (CNNs), particularly the classification results on the UP data set, and various evaluation indexes almost exceed 6 percentage points, as shown in Table 5, the superiority of the designed model under the condition of a small number of training samples is fully proved.

TABLE 5 Classification accuracy (percentage) of UP and SA datasets (0.5% sample for training)

Notably, at 0.5% of the training data, the performance of 3D-2D CNN was slightly lower than that of hybrid SN, which is different from the results at 1% of the training data. The analysis reason is probably that the 3D-2D CNN model is more complex and has more parameters, and more sample data is needed to train a model with good classification performance, which is also a defect of the complex network model itself.

It is to be understood that the above description is not intended to limit the present invention, and the present invention is not limited to the above examples, and those skilled in the art may make modifications, alterations, additions or substitutions within the spirit and scope of the present invention.

Claims

1. The hyperspectral classification method of the lightweight hybrid convolution model based on global reasoning is characterized by comprising the following steps of:

s1, setting original hyperspectral data as I epsilon R ^M×N×D I is the original hyperspectral classified input, M and N are the width and height of the spatial dimension, and D is the number of spectral bands;

s2, reducing the dimension of the hyperspectral classification spectral information by adopting principal component analysis, and expressing hyperspectral data subjected to principal component analysis as X epsilon R ^M×N×B The space dimension is kept unchanged, and the number of spectral bands is changed from D to B;

s3, extracting neighborhood blocks of the hyperspectral classification for preprocessing, extracting a color spot for each pixel of the hyperspectral classification by taking the pixel as a central point, wherein the label of the color spot is the label of the central pixel and is expressed as that Z belongs to R ^S×S×B The width and height of the color spot are S, and the number of spectral bands is B;

s4, sending the hyperspectral classified data into a lightweight mixed convolution model based on global reasoning for processing;

the lightweight hybrid convolution model based on the global reasoning comprises a two-dimensional convolution neural network, a three-dimensional convolution neural network and a global reasoning module; the three-dimensional convolution neural network adopts 8 convolution kernels with the size of 3 multiplied by 7, the space dimensionality is 3 multiplied by 3, and the spectrum dimensionality is 7;

The two-dimensional convolution neural network adopts 16 convolution kernels of 3 x 3, and the space dimension is 3 x 3;

the global reasoning module is two-dimensional, and the construction method comprises the steps of projecting the aggregated features of the disjoint areas on the coordinate space of the input data into a potential interaction space, representing the aggregated features by using a single feature, namely a node, modeling and reasoning the context relationship between each pair of nodes by applying a graph convolution mode, then carrying out back projection on the context relationship, and transforming the features with relationship information back to the original coordinate space to obtain related global feature information;

step S4 includes: