CN114373080A - Hyperspectral classification method of lightweight hybrid convolution model based on global reasoning - Google Patents

Abstract

The invention discloses a hyperspectral classification method of a lightweight hybrid convolution model based on global reasoning, belongs to the technical field of information processing, and is mainly used for hyperspectral image classification of small sample data. The lightweight hybrid convolution model based on global inference includes a layer of two-dimensional convolution, a layer of three-dimensional convolution and a global inference module; adding a global inference module can effectively extract hyperspectral images by inferring contextual relationships between different regions To replace the feature extraction of deep 3D convolution, the complexity and computational cost of the model are greatly reduced. The test results in the public data set show that the classification performance of the present invention is better than the current best classification method, and only a small number of training samples can be used to effectively extract the space-spectral joint feature of the hyperspectral image, which solves the problem of using only two-dimensional convolution. The problem of missing channel relationship information and the problem of greatly increased model complexity and computational cost caused by the use of deep three-dimensional convolution.

Description

Hyperspectral classification method based on lightweight hybrid convolution model based on global inference

technical field

The invention discloses a hyperspectral classification method of a lightweight hybrid convolution model based on global reasoning, and belongs to the technical field of information processing.

Background technique

Hyperspectral image (HSI) has brought great help to the extraction of ground object information due to its numerous spectral bands, but a large amount of spectral data also caused the redundancy of ground object information. The correlation between the bands is strengthened, which brings a lot of challenges to the classification of hyperspectral images. Especially when there are few training samples, it will bring greater difficulties to the classification of HSI.

At present, there are mainly two types of HSI classification methods: traditional methods based on spectral information and methods based on deep learning. The traditional method mainly uses the spectral feature information of different objects to classify, among which support vector machine, k-nearest neighbor and random forest are the most representative. In order to obtain better classification performance, methods such as Principal Component Analysis (PCA), Independent Component Analysis (ICA) and Linear Discriminant Analysis (LDA) are applied to effectively perform data dimensionality reduction or feature extraction. Although traditional HSI classification methods have achieved good classification performance, they cannot better adapt to complex HSI classification because they largely rely on hand-crafted features or extracted shallow features. Task.

In recent years, deep learning has been successfully applied in numerous computer vision tasks, such as image classification, image segmentation, object detection, etc., due to its powerful feature representation ability. The huge application potential makes deep learning methods also achieve remarkable results in the field of HSI classification, among which HSI classification methods based on convolutional neural networks (CNN) are the most popular. W. Hu et al. used deep CNN for the first time to classify hyperspectral images directly in the spectral domain, and obtained good classification results. Konstantinos Makantasis et al. used 2D-CNN to extract spatial features from the neighborhood of hyperspectral pixels, and used PCA method to effectively reduce dimensionality and reduce the computational cost of the model. However, this model can only extract spatial feature information, and cannot obtain the relationship information between spectral channels. YushiChen et al. applied the 3D-CNN method to the HSI task, and simultaneously extracted the spectral-spatial joint features through 3D-CNN, which effectively improved the classification accuracy. On this basis, M. He et al. proposed a multi-scale deep 3D-CNN model, which can jointly learn multi-scale spatial features and spectral features from HSI data end-to-end, further improving the classification performance of HSI.

However, utilizing the 3D-CNN model for HSI classification will require deep convolutional layers to effectively extract the joint spatial-spectral features, resulting in a large increase in model complexity and the number of training samples. Based on this, some researchers have begun to combine 2D-CNN with 3D-CNN for HSI classification. In 2020, SwalpaKumarRoy et al. proposed an HSI classification method based on the HybridSN model, which combines 3D-CNN and 2D-CNN to fully extract spatial feature information and spectral feature information, compared to using 3D-CNN alone for HSI classification. , which has achieved great improvements in computational cost and classification performance. In 2021, SaeedGhaderizadeh et al. proposed a 3D-2DCNN model. This model makes the model more robust and efficient by introducing 3D depthwise separable convolution blocks and fast convolution blocks. But it is worth noting that in order to make the model have stronger spatial feature extraction ability, the model often contains multiple or even deep 3D convolution layers. In the literature, up to three layers of 3D convolution are used for feature extraction. extraction. This method of extracting deep features through the stacking of multi-layer 3D convolutions increases the complexity and computational cost of the model on the one hand, and requires a large number of samples for training on the other hand. For HSI, the annotation of a large number of samples It's also quite a challenge.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a hyperspectral classification method based on a lightweight hybrid convolutional model based on global reasoning, so as to solve the problem that in the prior art, the hyperspectral classification method includes multiple three-dimensional convolutional layers, resulting in complex models, high computational costs, and high sample size. high demand problem.

A hyperspectral classification method based on a lightweight hybrid convolutional model based on global inference, including:

S1. Let the original hyperspectral data be represented as I∈RM×N×D, where I is the original hyperspectral classification input, M and N are the width and height of the spatial dimension, and D is the number of spectral bands;

S2. Principal component analysis is used to reduce the dimension of spectral information for hyperspectral classification. The hyperspectral data processed by principal component analysis is expressed as X∈RM×N×B, and its spatial dimension remains unchanged, and the number of spectral bands changes from D to B;

S3. Preprocess the neighborhood block extraction of hyperspectral classification. For each pixel of hyperspectral classification, extract a color spot with the pixel as the center point. The label of the color spot is the label of the center pixel, indicating is Z∈RS×S×B, the width and height of the color spot are S, and the number of spectral bands is B;

S4. The hyperspectral classification data is sent to a lightweight hybrid convolution model based on global inference for processing.

Preferably, the lightweight hybrid convolutional model based on global inference includes a two-dimensional convolutional neural network, a three-dimensional convolutional neural network and a global inference module.

Preferably, the three-dimensional convolutional neural network adopts 8 convolution kernels of 3×3×7, the spatial dimension is 3×3, and the spectral dimension is 7.

Preferably, the two-dimensional convolutional neural network adopts 16 3×3 convolution kernels, and the spatial dimension is 3×3.

Preferably, the global reasoning module is two-dimensional, and its construction method is to project the aggregated features of disjoint regions in the coordinate space of the input data into a potential interaction space, and use a single feature, that is, a node to represent , using graph convolution to model and reason the context relationship between each pair of nodes, and then back-project it to transform the features with relationship information back to the original coordinate space to obtain relevant global feature information.

Preferably, step S4 includes:

S4.1. Use a three-dimensional convolutional neural network to jointly extract the spatial and spectral features of the preprocessed hyperspectral classification data;

S4.2. Use a two-dimensional convolutional neural network to jointly extract spatial features and spectral features from the hyperspectral classification data processed in S4.1;

S4.3. Use the global inference module to extract global features from the hyperspectral classification data processed in S4.2.

Compared with the prior art, the present invention only uses a lightweight hybrid convolutional network model of one layer of 3D-CNN and one layer of 2D-CNN, which can effectively extract the space-spectral joint feature of HSI while reducing the complexity of the network model. ; In the designed hybrid convolutional network model, a lightweight global reasoning module is added to effectively extract the global features of HSI by inferring the contextual relationship between different regions, and the classification performance of HSI is significantly improved; the proposed model not only reduces the The complexity of existing hybrid network models, and very good classification performance can be obtained when training only a few samples.

Description of drawings

Fig. 1 is the structure diagram of the lightweight hybrid convolution model based on global reasoning of the present invention;

Figure 2 is a schematic diagram of the global reasoning module.

Detailed ways

Below in conjunction with specific embodiment, the present invention is described in further detail:

The embodiment first introduces the overall structure of the lightweight hybrid convolution model based on global reasoning, then introduces the structure and parameter design of the 3D-CNN and 2D-CNN hybrid convolution models, and finally gives the core idea of the lightweight global inference module.

Deep convolutional neural network model: A single deep 2D convolutional or 3D convolutional network as a feature extraction model for hyperspectral classification can lead to missing spectral channel relationships or high model complexity. The existing hybrid convolutional network model for hyperspectral classification also has a large number of layers of 3D-CNN, which increases the complexity of the model, the computational cost and the cost of labeling the sample to a certain extent. The present invention only needs a combination of a layer of 3D-CNN and a layer of 2D-CNN, plus a global inference module, to obtain high-performance hyperspectral classification effects with a small number of samples.

Let the original hyperspectral data be represented as I∈R ^M×N×D , where I is the original hyperspectral classification input, M and N spatial dimensions of width and height, D is the number of spectral bands, generally tens or even hundreds of bands . The large number of band information in hyperspectral classification will not only bring a large computational cost to subsequent processing tasks, but also when the training samples are limited, the overall classification accuracy of hyperspectral classification will decrease with the increase of feature dimension, that is, “dimension disaster" issue. Therefore, it is imperative to reduce the spectral dimension of hyperspectral classification and reduce the redundancy of information. Principal component analysis (PCA) is a common dimensionality reduction method, and the core idea is to calculate the feature similarity between different data and extract the main features. The invention adopts the PCA method to reduce the dimension of the hyperspectral classification spectral information, so as to effectively improve the problems of high computational cost and low classification accuracy caused by information redundancy. After PCA processing, it is expressed as X∈R ^M×N×B , the spatial dimension remains unchanged, and the number of spectral bands changes from D to B.

In order to better utilize the subsequent convolutional network for classification processing, after PCA dimensionality reduction, the neighborhood block extraction of hyperspectral classification is preprocessed. That is, for each pixel of hyperspectral classification, a patch is extracted from it as the center point. The label of each patch is the label of the central pixel, expressed as Z∈R ^S×S×B , the width and height of the patch are S, and the number of spectral bands is B.

The hyperspectral classification data is preprocessed by PCA and neighborhood block extraction, and then sent to a lightweight hybrid convolution model based on global inference for processing. The mixture of single-layer 3D-CNN and 2D-CNN completes the joint extraction of spectral and spatial features and reduces the complexity of the model. The global inference module extracts global features by inferring the contextual relationship between different regions, which solves the problem of extracting expressive capabilities. Stronger features require computationally expensive multi-layer 3D-CNN problems.

3D-2D CNN: Hyperspectral classification is a volume data that includes dozens or even hundreds of bands, including spatial information and spectral information. For traditional 2D-CNN, only spatial feature information can be extracted; while 3D-CNN can extract spatial features and spectral features at the same time. Therefore, in recent years, hyperspectral classification based on 3D-CNN has become the mainstream, but due to the complexity of the deep 3D-CNN model, the computational cost of the model is large and a large number of training samples are required. Several existing hybrid convolution models for hyperspectral classification have good classification performance and can reduce the number of training samples to a certain extent. However, the problem of increased computational cost brought about by multi-layer 3D-CNN stacking cannot be ignored, and the problem of a large number of training samples cannot be completely solved. In order to ensure the integrity of the spatial and spectral information of the hyperspectral classification data, 3D-CNN is firstly used to jointly extract the spatial and spectral features of the preprocessed hyperspectral classification data. In the 3D-CNN layer, 8 convolution kernels of 3×3×7 are used, that is, the spatial dimension is 3×3 and the spectral dimension is 7. In FIG. 1 , it is shown that K ₁ ¹ =3, K ₂ ¹ =3, and K ₃ ¹ =7. In order to fully extract the spatial feature information of hyperspectral classification data, after using 3D-CNN for feature extraction, a layer of 2D-CNN extraction is used to improve the classification effect of hyperspectral classification. In 2D-CNN, 16 3×3 convolution kernels are used, that is, the spatial dimension is 3×3. In Figure 1, it is shown that K ₁ ³ =3, K ₂ ³ =3.

A single convolutional layer can better capture the local relationship covered by the convolution kernel, but capturing the relationship between global regions with stronger expressiveness requires stacking multiple or even deep convolutional layers, which undoubtedly increases the global inference ability of CNN. difficulty and cost. Several existing hybrid convolutional models for hyperspectral classification all use this inefficient multi-layer 3D-CNN stacking design to extract global features. The global reasoning module added in the hybrid convolution model proposed by the present invention is to overcome the inherent deficiency of the multi-layer convolution operation for modeling the global relationship.

The global reasoning module is a lightweight global reasoning module. It extracts relevant global and deep-level features by modeling and reasoning the global relationship between regions. The principle framework is shown in Figure 2, which represents the interaction space and coordinates. Projection and back-projection between spaces. First, the features aggregated from disjoint regions in the coordinate space of the input data are projected into a potential interaction space, represented by a single feature, a node. In this way, relational reasoning is reduced to modeling interactions between nodes on the smaller graph shown at the top of Figure 2. Then, a graph convolution approach is applied to model and reason about the contextual relationship between each pair of nodes. Finally, it is back-projected to transform the features with relational information back to the original coordinate space, and then the relevant global feature information can be obtained.

The hyperspectral classification data has relatively complete spatial information and spectral information after the 3D-CNN layer extracts the spatial-spectral joint features, which is conducive to the global relational reasoning for the hyperspectral classification. Therefore, a module is added after the 3D-CNN layer to extract global features for hyperspectral classification. Due to the high complexity of the modules at the 3D level, the present invention adopts the modules at the 2D level to aggregate the global features of the hyperspectral classification data, which exactly matches the spatial feature information extracted by the subsequent 2D-CNN layer. The sufficient structure combination experiment verifies that the "3D-CNN++2D-CNN" combination has the best model classification performance. Among the structural parameters of the proposed model for the IP dataset, the total number of trainable weight parameters is 2,275,504, as shown in Table 1.

Table 1 Overview of the models used in the present invention for the IP dataset

The examples used three public datasets for hyperspectral classification as experimental datasets, namely Salinas Scene (SA), University of Pavia (UP) and Indian Pines (IP). The SA dataset has 16 categories with a spatial dimension of 512 × 217 and 224 spectral bands in the wavelength range of 360-2500 nm. After removing 20 water absorption bands, 204 spectral bands remained. The UP dataset has 9 categories with a spatial dimension of 610 × 340 and 103 spectral bands in the wavelength range of 430-860 nm. The IP dataset has 16 categories with a spatial dimension of 145 × 145 and 224 spectral bands in the 400-2500 nm wavelength range. 200 spectral bands remained after removing 24 water absorption bands.

All experiments are performed on a Lenovo-Y7000P computer. The computer is equipped with a GTX 1660ti Graphics Processing Unit (GPU) and 16 GB of RAM. Using the SGD optimizer, the learning rate is set to 0.01. Momentum and L2 regularization are used, and the parameters are set to 0.8 and 0.0005, respectively. Use mini-batches of size 32 in model training. In addition, in order to speed up the training process and improve the generalization performance, batch normalization is performed on the model. Set the number of iterations for network model training to 50. For a fair comparison with other methods, for different datasets, the same neighborhood blocks of size 27 × 27 × 20 are extracted from the PCA-reduced data.

The hyperspectral classification performance was judged using Kappa coefficient (Kappa), overall accuracy (OA) and average accuracy (AA) as evaluation metrics. Kappa reflects the consistent information between information; OA refers to the proportion of the correct classification; AA refers to the average of the correct classification proportion of each category.

Among the effects of different Window Size and Spectral Dimension on the performance of the LH-CNN model, the window size and spectral dimension of 27 × 27 × 20 neighborhood blocks have the best effect, as shown in Table 2 and Table 3.

Table 2 Influence of spatial window size on model performance

Table 3 Influence of spectral dimension on model performance

In order to verify the classification performance of the designed model under the condition of a small number of training samples, it is compared with the commonly used CNN methods for hyperspectral classification and the current best hybrid convolution methods, namely 2-D-CNN, 3-D-CNN, multi-scale 3-D deepconvolutional neural network (M3D)-CNN, HybridSN and 3D-2D CNN for comparison. The training samples of various methods are randomly selected from the data set, and the sample ratio is 1%. In the case of only 1% training data, the model proposed by the present invention has better performance than other methods, as shown in Table 4.

Table 4 Classification accuracy (percentage) of IP, UP and SA datasets (1% of samples are used for training)

Especially in the experiments on the IP dataset with a small sample size, from the three evaluation indicators of Kappa, OA and AA, the performance of the proposed model far exceeds the best hybrid CNN model. The complexity of the hybrid CNN model and the computational cost in different data sets, the model proposed by the present invention is optimal in terms of model complexity and computational cost, and the global inference module has a significant impact on the performance of hyperspectral classification.

In order to further verify the classification performance of the proposed model in the case of a small number of training samples and reduce the number of training samples, 0.5% of the sample data is randomly selected as training samples in the UP and SA data sets (here, the IP data set has a small number of samples in some categories. 0.5% of the sample data cannot be selected), the designed model has the best performance, and far exceeds the classification results of the best two hybrid CNNs, especially the classification results on the UP data set, all kinds of evaluation indicators almost exceed 6 Percentage points, as shown in Table 5, fully demonstrate the superiority of the designed model in the case of a small number of training samples.

Table 5 Classification accuracy (percentage) of UP and SA datasets (0.5% samples are used for training)

It is worth noting that under 0.5% training data, the performance of 3D-2D CNN is slightly lower than HybridSN, which is different from the results under 1% training data. The reason for the analysis may be that the 3D-2D CNN model is more complex and has more parameters, requiring more sample data to train a model with good classification performance, which is also the deficiency of the complex network model itself.

Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by those skilled in the art within the essential scope of the present invention should also belong to the present invention. the scope of protection of the invention.

Claims (6)

1. The hyperspectral classification method based on the lightweight hybrid convolution model of global reasoning, is characterized in that, comprises: S1. Let the original hyperspectral data be represented as I∈R ^M×N×D , where I is the original hyperspectral classification input, M and N are the width and height of the spatial dimension, and D is the number of spectral bands; S2. Use principal component analysis to reduce the dimension of spectral information for hyperspectral classification. The hyperspectral data processed by principal component analysis is expressed as X∈R ^M×N×B , its spatial dimension remains unchanged, and the number of spectral bands changes from D to D for B; S3. Preprocess the neighborhood block extraction of hyperspectral classification. For each pixel of hyperspectral classification, extract a color spot with the pixel as the center point, and the label of the color spot is the label of the central pixel, indicating is Z∈R ^S×S×B , the width and height of the color spot are S, and the number of spectral bands is B; S4. The hyperspectral classification data is sent to a lightweight hybrid convolution model based on global inference for processing. 2. The hyperspectral classification method of the lightweight hybrid convolution model based on global reasoning according to claim 1, wherein the lightweight hybrid convolution model based on global reasoning comprises a two-dimensional convolutional neural network, a three-dimensional volume Integrating Neural Networks and Global Reasoning Module. 3. The hyperspectral classification method of the lightweight hybrid convolution model based on global reasoning according to claim 2, wherein the three-dimensional convolutional neural network adopts 8 convolution kernels of 3×3×7, and the spatial dimension is 3×3 and the spectral dimension is 7. 4. The hyperspectral classification method of the lightweight hybrid convolution model based on global reasoning according to claim 3, wherein the two-dimensional convolutional neural network adopts 16 convolution kernels of 3×3, and the spatial dimension is 3×3. 5. The hyperspectral classification method of the lightweight hybrid convolution model based on global reasoning according to claim 4, wherein the global reasoning module is two-dimensional, and its construction method is to put the coordinate space of the input data on the coordinate space. The aggregated features of disjoint regions are projected into a potential interaction space, represented by a single feature, that is, a node. The graph convolution method is applied to model and reason the context relationship between each pair of nodes, and then the Backprojection transforms the features with relational information back to the original coordinate space to obtain relevant global feature information. 6. The hyperspectral classification method of the lightweight hybrid convolution model based on global reasoning according to claim 5, wherein step S4 comprises: S4.1. Use a three-dimensional convolutional neural network to jointly extract the spatial and spectral features of the preprocessed hyperspectral classification data; S4.2. Use a two-dimensional convolutional neural network to jointly extract spatial features and spectral features from the hyperspectral classification data processed in S4.1; S4.3. Use the global inference module to extract global features from the hyperspectral classification data processed in S4.2.

Images