CN116310510A - Hyperspectral image classification method based on small sample deep learning - Google Patents

Abstract

The invention relates to a hyperspectral image classification method based on small sample deep learning, which comprises the following steps: acquiring a source domain hyperspectral image set and a target domain hyperspectral image set; extracting a plurality of first image blocks and a plurality of second image blocks by taking each pixel point of the source domain hyperspectral image and the target domain hyperspectral image after filling treatment as a center; in each category, randomly selecting a part of first image blocks to form a source domain support set and a part of first image blocks to form a source domain query set, and randomly selecting a part of second image blocks to form a target domain support set and a part of first image blocks to form a target domain query set; training the small sample spectral space feature extraction convolutional neural network by using a support set and a query set to obtain a trained network; and inputting the hyperspectral image to be classified into a trained small sample spectral space feature extraction convolutional neural network to obtain a classification result. The method and the device can improve the classification precision and extract the spatial information contained in the hyperspectral image in a deeper level.

Description

A hyperspectral image classification method based on small-sample deep learning

technical field

The invention belongs to the technical field of remote sensing information processing, and relates to a hyperspectral image classification method based on small-sample deep learning.

Background technique

Hyperspectral records the continuous spectral characteristics of ground objects with its rich band information, and has the possibility of recognizing more types of ground objects and classifying objects with higher accuracy. Different from ordinary natural images, hyperspectral image data presents a three-dimensional structure with rich spectral information and relatively little spatial information. The key to hyperspectral image classification technology is to use the spatial and spectral features of hyperspectral images to classify sample categories . However, hyperspectral image training samples are few, and overfitting problems are prone to occur for classification methods that require a large number of parameters. How to train an efficient classification model with a small number of samples is very important for hyperspectral image classification. important.

Kun Tan et al. proposed a new semi-supervised HSI classification method in their paper "A novel semi-supervised hyperspectral imageclassification approach based on spatial neighborhood information and classifier combination" (ISPRS journal of photogrammetry and remote sensing, 2015). Combining spatial neighborhood information with classifiers to enhance classification capabilities. In their paper "Semi-supervised hyperspectral imageclassification via spatial-regulated self-training" (Remote Sensing, 2020), Yue Wu et al. proposed a semi-supervised method that uses self-training to cluster highly confident pseudo Labels are gradually assigned to unlabeled samples, and spatial constraints are exploited to regulate the self-training process. However, these methods assume that labeled and unlabeled samples come from the same dataset, which means that the classification performance is still limited by the number of labeled samples in the data to be classified (i.e., the target domain).

In their paper "Deep few-shot learning for hyperspectral image classification" (IEEE Transactions on Geoscience and Remote Sensing, 2018), Bing Liu et al. proposed a small sample deep learning method to solve the small sample problem of HSI classification. Better aid classification by learning a metric space from the training set. In their paper "Deep relation network for hyperspectral image few-shot classification" (Remote Sensing, 2020), Kuiliang Gao et al. designed a new deep relation network-based classification model and trained it with the idea of meta-learning.

In addition to the hyperspectral image classification methods listed above, the current hyperspectral image classification methods based on deep convolutional neural networks are similar to the above methods. The loss of information caused by insufficient utilization, or the redundancy of information caused by retaining too much irrelevant information, cannot make full use of the key information in the hyperspectral image band and obtain more distinguishable spectral-space semantic features, and requires a large number of training methods. Hyperspectral samples are used to train neural networks, which leads to poor performance of these methods on hyperspectral image classification during few-sample training, and does not pay more attention to the differences in information between different spectra. And considering the problem of fewer hyperspectral datasets, although there is a small sample learning method to solve the hyperspectral classification problem, no suitable method is proposed to solve the domain adaptation problem caused by using different hyperspectral datasets. In classification, the loss function adopted by the above methods is too simple to meet the requirements of high-precision classification.

Contents of the invention

The purpose of the present invention is to address the deficiencies of the above-mentioned existing technologies, and propose a hyperspectral image classification method based on small-sample deep learning. By learning the prior knowledge in the source domain hyperspectral data set, the source domain with a large number of label samples is used. Hyperspectral datasets are used to help classify target domain hyperspectral datasets with a small number of label samples, so as to improve the accuracy of object classification in hyperspectral images in the case of few-sample training. The technical problem to be solved in the present invention is realized through the following technical solutions:

An embodiment of the present invention provides a hyperspectral image classification method based on small-sample deep learning. The hyperspectral image classification method includes:

Step 1. Obtain a source domain hyperspectral image set and a target domain hyperspectral image set, the source domain hyperspectral image set includes multiple source domain hyperspectral images, and the target domain hyperspectral image set includes multiple target domain hyperspectral images;

Step 2. Carry out filling processing on the edge parts of the source domain hyperspectral image and the target domain hyperspectral image respectively, with each pixel of the source domain hyperspectral image and the target domain hyperspectral image after filling processing as Centrally extract several first image blocks and several second image blocks;

Step 3. In each category, randomly select some of the first image blocks to form the source domain support set and some of the first image blocks to form the source domain query set, and randomly select some of the second image blocks to form the target The domain support set and part of the first image blocks form a target domain query set;

Step 4. Based on the stochastic gradient descent method, use the source domain support set, the source domain query set, the target domain support set and the target domain query set to alternately extract the convolutional neural network for small-sample spectral-space feature extraction Training to obtain the trained small-sample spectral-space feature extraction convolutional neural network, the small-sample spectral-space feature extraction convolutional neural network extracts features from two aspects of spectrum and space, and the small-sample spectral-space feature extraction convolutional neural network The total loss function of the network consists of a cross-entropy loss function, a correlation alignment loss function, and a maximum meanization difference loss function;

Step 5. Input the hyperspectral image to be classified into the trained small-sample spectral-space feature extraction convolutional neural network to obtain a classification result.

In an embodiment of the invention, the step 2 includes:

Step 2.1, respectively filling pixels with a pixel value of 0 around the source domain hyperspectral image and the target domain hyperspectral image to obtain a filled source domain hyperspectral image and a filled target domain hyperspectral image;

Step 2.2, take each pixel in the filled source hyperspectral image and the filled target hyperspectral image as the center, select a space size of (2t+1)×(2t+1), channel The number of the first image block and the second image block is d, wherein, t is an integer greater than 0.

In an inventive embodiment, the structure of the small-sample spectral-space feature extraction convolutional neural network includes a spectral branch network, a spatial branch network, a domain attention module, a first splicing layer, a fully connected layer, and a softmax classifier. The spectral branch network and the spatial branch network are connected in parallel and then connected in series with the domain attention module, the domain attention module, the first splicing layer, the fully connected layer and the softmax classifier are connected in series, and the spectral The branch network includes 2 3D deformable convolution blocks and 2 first maximum pooling layers, the first 3D deformable convolution block, the first first maximum pooling layer, and the second The 3D deformable convolution block and the second first maximum pooling layer are connected in sequence, and the spatial branch network domain attention module includes a multi-scale spatial feature extraction module, a second splicing layer and a second maximum pooling layer connected in sequence .

In an inventive embodiment, the 3D deformable convolution block includes three 3D deformable convolution layers and three first activation function layers, the first 3D deformable convolution layer, the first one The first activation function layer, the second 3D deformable convolution layer, the second first activation function layer, the third 3D deformable convolution layer, the third first activation function The layers are connected in sequence, and the output of the first activation function layer is added to the output of the third 3D deformable convolution layer to form a residual structure.

In an inventive embodiment, the multi-scale spatial feature extraction module includes 2 scale operation layers, 3 convolution layers, 3 normalization layers, and 3 second activation function layers;

The first convolutional layer, the first normalization layer, and the first second activation function layer are sequentially connected in series;

The first scale operation layer, the second convolution layer, the second normalization layer, and the second second activation function layer are connected in sequence;

The second scale operation layer, the third convolution layer, the third normalization layer, and the third second activation function layer are connected in series in sequence;

The first described second activation function layer, the second described second activation function layer, and the third described second activation function layer are connected in parallel and then combined with the second stitching layer and the second maximum The pooling layers are sequentially connected in series.

In an inventive embodiment, the total loss function of the small-sample spectral-space feature extraction convolutional neural network is:

L _total ＝L _fsl +L _coral +L _MMD

Among them, L _total represents the total loss function of the convolutional neural network for small-sample spectral-space feature extraction, L _fsl represents the cross-entropy loss, L _coral represents the correlation alignment loss, and L _MMD represents the maximum mean difference loss;

The cross-entropy loss L _fsl is expressed as:

Among them, d(·) represents the Euclidean distance, F _ω (·) represents the feature extraction function of parameter ω, f _l represents the feature of the lth category in the source domain support set or target domain support set, C represents the number of categories, x _j represents a sample in the source domain query set or the target domain query set, y _j represents the label of the sample x _j , and Q represents the source domain query set or the target domain query set;

The correlation alignment loss L _coral is expressed as:

表示矩阵的Frobenius范数，C_S表示源域特征的协方差矩阵，C_T表示目标域特征的协方差矩阵，d表示特征的维度；in,

Represents the Frobenius norm of the matrix, C _S represents the covariance matrix of the source domain features, C _T represents the covariance matrix of the target domain features, and d represents the dimension of the features;

The maximum meanization difference loss L _MMD is expressed as:

表示空间距离，φ(·)表示映射函数，/>

表示源域特征，/>

表示目标域特征，X_s表示源域数据集，X_t表示目标域数据集，n_s表示源域数据集中数据的个数，n_t表示目标域数据集中数据的个数。in,

represents the spatial distance, φ( ) represents the mapping function, />

Indicates source domain characteristics, />

Represents the characteristics of the target domain, X _s represents the source domain data set, X _t represents the target domain data set, n _s represents the number of data in the source domain data set, n _t represents the number of data in the target domain data set.

In an embodiment of the invention, the step 4 includes:

Set the initial learning rate of training to α, and the number of iterations is T. When an odd number of iterations is used, the source domain support set and the source domain query set are sent to the small-sample spectral-space feature extraction convolutional neural network for training , using the total loss function to calculate the loss value between the features of the source domain support set and the features of the source domain query set, so as to update the parameters in the small-sample spectral-space feature extraction convolutional neural network; in During an even number of iterations, the target domain support set and the target domain query set are sent to the small-sample spectral-space feature extraction convolutional neural network for training, and the total loss function is used to calculate the target domain support set. The loss value between the feature and the feature of the target domain query set to update the parameters in the small-sample spectral-space feature extraction convolutional neural network until the loss value of the small-sample spectral-space feature extraction convolutional neural network No longer decline and the current number of training rounds is less than the number of iterations T or the number of training rounds reaches the number of iterations T, then stop the training of the convolutional neural network for extracting the small-sample spectral-space features, and obtain the trained small-sample spectral-space features Extract Convolutional Neural Networks.

In an inventive embodiment, the updated weight vector W _new of the small-sample spectral-space feature extraction convolutional neural network is:

Among them, L _total represents the total loss function of the small-sample spectral-space feature extraction convolutional neural network, W represents the weight vector of the small-sample spectral-space feature extraction convolutional neural network before updating, and R represents the learning rate.

Compared with prior art, the beneficial effect of the present invention:

Aiming at rich spectral feature information and spatial feature information of hyperspectral images, the present invention utilizes a small-sample spectral-space feature extraction convolutional neural network to classify hyperspectral images. Extracting features from two aspects of space can improve the classification accuracy and extract the spatial information contained in the hyperspectral image in a deeper level.

In the spectral branch, the present invention adopts 3D deformable convolution and residual structure, so that the convolution kernel can better extract deep information and retain shallow network information for irregularly shaped hyperspectral images, improving classification accuracy ; the spatial branch adopts multi-scale operation. For the input hyperspectral samples, two copies are first copied, and then the edge pixels are discarded in turn. In this way, three input samples with different spatial resolutions are obtained, which can be extracted at a deeper level. Spatial information contained in hyperspectral images.

Other aspects and features of the present invention will become apparent from the following detailed description with reference to the accompanying drawings. It should be understood, however, that the drawings are designed for purposes of illustration only and not as a limitation of the scope of the invention since reference should be made to the appended claims. It should also be understood that, unless otherwise indicated, the drawings are not necessarily drawn to scale and are merely intended to conceptually illustrate the structures and processes described herein.

Description of drawings

Fig. 1 is a schematic flow chart of a hyperspectral image classification method based on small-sample deep learning provided by an embodiment of the present invention;

2 is a schematic diagram of a model structure of a small-sample spectral-space feature extraction convolutional neural network provided by an embodiment of the present invention;

Fig. 3 is a schematic structural diagram of a 3D deformable convolution module in a spectrum branch provided by an embodiment of the present invention;

Fig. 4 is a schematic structural diagram of a multi-scale spatial feature extraction module provided by an embodiment of the present invention;

Fig. 5 is respectively with the present invention and existing two kinds of network classification result simulation figure on the University of Pavia data set;

Fig. 6 is a simulation diagram of classification results on the Indian Pines data set using the present invention and the existing two networks respectively.

Detailed ways

The present invention will be described in further detail below in conjunction with specific examples, but the embodiments of the present invention are not limited thereto.

Embodiment one

At present, hyperspectral image classification methods based on deep convolutional neural networks are deficient. The commonality of these methods is that information is lost due to insufficient utilization of extracted features during spectral and spatial feature extraction, or too much irrelevant information is retained. The information causes information redundancy, which cannot make full use of the key information in the hyperspectral image band and obtain more distinguishable spectral-space semantic features, and requires a large number of hyperspectral samples to train the neural network during training, which leads to These methods do not perform well in hyperspectral image classification during training, and do not pay more attention to the differences in information between different spectra.

Based on this, the present invention proposes a hyperspectral image classification method based on small-sample deep learning, please refer to Figure 1 for details, which is a flow chart of a hyperspectral image classification method based on small-sample deep learning provided by an embodiment of the present invention Schematic diagram, the hyperspectral image classification method based on small-sample deep learning provided by the embodiment of the present invention may specifically include steps 1 to 4, wherein:

Step 1. Obtain a source domain hyperspectral image set and a target domain hyperspectral image set. The source domain hyperspectral image set includes multiple source domain hyperspectral images, and the target domain hyperspectral image set includes multiple target domain hyperspectral images.

Specifically, the hyperspectral image is a three-dimensional data S∈R ^h×w×c , and each band in the hyperspectral image corresponds to a two-dimensional matrix S _i ∈R ^h×w in the three-dimensional data, where ∈ represents the symbol , R represents the real number domain symbol, h represents the length of the hyperspectral image, w represents the width of the hyperspectral image, c represents the number of spectral bands in the hyperspectral image, i represents the serial number of the spectral band in the hyperspectral image, i=1,2, ..., c.

In this embodiment, the source domain hyperspectral image set adopts the Chikusei dataset, and the target domain hyperspectral image set adopts UP and other datasets.

Step 2. Carry out filling processing on the edge parts of the source domain hyperspectral image and the target domain hyperspectral image respectively, and extract several pixel points of the filled source domain hyperspectral image and target domain hyperspectral image One image block and several second image blocks.

Specifically, this embodiment performs filling processing on the source domain hyperspectral image and the target domain hyperspectral image respectively, so that the first image block and the second image block at the edge can also be obtained, so that the first image at the edge The block and the second image block contain the required information.

Step 2.1. Fill the surroundings of the source-domain hyperspectral image and the target-domain hyperspectral image with pixels with a pixel value of 0 to obtain the filled source-domain hyperspectral image and the filled target-domain hyperspectral image.

Step 2.2. Taking each pixel in the filled source hyperspectral image and the filled target hyperspectral image as the center, select the image with a space size of (2t+1)×(2t+1) and a channel number of d For the first image block and the second image block, t is an integer greater than 0, for example, the size of the image block is 9×9, so t=4.

Specifically, the first image block is a pixel block extracted centering on the pixel point in the filled source domain hyperspectral image, and the second image block is a pixel block extracted centering on the pixel point in the filled target domain hyperspectral image . Among them, the number of channels d is the same as the number of spectral bands of the hyperspectral image.

Step 3. In each category, randomly select some of the first image blocks to form the source domain support set and some of the first image blocks to form the source domain query set, randomly select some of the second image blocks to form the target domain support set and some of the first image blocks to form the target domain support set and some of the first image blocks to form the source domain support set. An image patch constitutes the target domain query set.

Specifically, the first image block and the second image block are allocated to the set to which the category belongs according to the category of the central pixel point, such as water, glass, and so on. Select a part from the first image block of each category to form the source domain support set, and then select a part to form the source domain query set; select a part from the second image block of each category to form the target domain support set, and then select The output part constitutes the query set of the target domain.

For example, the source domain hyperspectral image set uses the Chikusei dataset, and 200 first image blocks are selected for each category to form the source domain dataset, and then an image block is randomly selected from the source domain dataset for each category to form the source domain support Each category randomly selects 19 first image blocks to form the source domain query set; the target domain hyperspectral image set adopts UP and other data sets, and each category selects 5 second image blocks to form the target domain data set, and performs After data enhancement (that is, copying the target domain dataset), for each category, an image block is randomly selected from the target domain dataset to form the target domain support set, and 19 image blocks are randomly selected for each category to form the target domain query set .

Step 4. Based on the stochastic gradient descent method, use the source domain support set, source domain query set, target domain support set and target domain query set to alternately train the small-sample spectral-space feature extraction convolutional neural network to obtain trained small samples Spectral-space feature extraction Convolutional neural network, small-sample spectral-space feature extraction Convolutional neural network extracts features from two aspects of spectrum and space, small-sample spectral-space feature extraction The total loss function of convolutional neural network is composed of cross-entropy loss function, correlation Consistency alignment loss function and maximum meanization difference loss function.

Specifically, please refer to Figure 2. The structure of the small-sample spectral-space feature extraction convolutional neural network includes a spectral branch network, a spatial branch network, a domain attention module, the first splicing layer, a fully connected layer and a softmax classifier, and a spectral branch network. , the spatial branch network is connected in parallel and then connected in series with the domain attention module, the domain attention module, the first splicing layer, the fully connected layer and the softmax classifier are connected in series in sequence, and the spectral branch network includes 2 3D deformable convolution blocks and 2 first A maximum pooling layer, the first 3D deformable convolutional block, the first first maximum pooling layer, the second 3D deformable convolutional block, and the second first maximum pooling layer are connected in series, and the spatial branch The network domain attention module includes a multi-scale spatial feature extraction module, a second concatenation layer and a second maximum pooling layer in series.

The convolution kernel size of the first first maximum pooling layer is set to 2*2*4, the number of convolution kernels is set to 8, and the convolution kernel size of the second first maximum pooling layer is set to 2*2* 4. The number of convolution kernels is set to 16.

Please refer to Figure 3, the 3D deformable convolution block includes 3 3D deformable convolution layers and 3 first activation function layers, the first 3D deformable convolution layer, the first first activation function layer, the second A 3D deformable convolutional layer, the second first activation function layer, the third 3D deformable convolutional layer, and the third first activation function layer are connected in series, and the output of the first first activation function layer is connected to the first The outputs of three 3D deformable convolutional layers are added to form a residual structure. The 3D deformable convolutional layer is a structure formed by adding a deformable convolution kernel to the 3D convolution.

The convolution kernel size of the 3D deformable convolution layer is set to 3*3*3; the activation function of each first activation function layer is set to the ReLU activation function, expressed as follows:

ReLU(x)=max(0,x)

Among them, x represents the input of the activation function.

Please refer to Figure 4. The multi-scale spatial feature extraction module includes 2 scale operation layers, 3 convolution layers, 3 normalization layers, and 3 second activation function layers, where:

The first convolutional layer, the first normalization layer, and the first second activation function layer are connected in series;

The first scale operation layer, the second convolutional layer, the second normalization layer, and the second second activation function layer are connected in sequence;

The second scale operation layer, the third convolutional layer, the third normalization layer, and the third second activation function layer are connected in sequence;

The first second activation function layer, the second second activation function layer, and the third second activation function layer are connected in parallel and then connected in series with the second splicing layer and the second maximum pooling layer.

The first scale operation layer in the multi-scale spatial feature extraction module reduces one pixel point around the edge of the selected image block, and the second scale operation layer reduces two pixels around the edge of the selected image block. The convolution kernel size of the multilayer is set to 5*5*4, the convolution kernel size of the second convolution layer is set to 3*3*4, and the convolution kernel size of the third convolution layer is set to 1*1 *4, the number is set to 16, and the activation function of each second activation function layer is set to the ReLU activation function;

The outputs of the three second activation function layers are all 16 features with a size of 5*5*25. After the splicing operation of the second splicing layer, 16 features with a size of 5*5*75 are obtained, and then through the second largest The pooling layer performs a pooling operation, the convolution kernel of the second maximum pooling layer is set to 2*2*8, and the number of convolution kernels is set to 16.

The domain attention module uses a 2D convolutional layer, including 1 spectral attention module and 2 spatial attention modules, the spectral attention module is 1 2D convolutional layer, and the 2 spatial attention modules are 2 2D convolutional layers Layer, three 2D convolutional layers are connected in series in sequence, the spectral attention module has its convolution kernel size set to 9*9, the number of convolution kernels is set to 1, and the domain attention module includes spatial attention and interspectral attention;

In this embodiment, the spectral branch network and the spatial branch network are connected in parallel and then connected in series with the domain attention module, the first splicing layer, the fully connected layer and the softmax classifier to form a small-sample spectral-space feature extraction convolutional neural network. The domain attention module is Convolution layer, domain attention module performs convolution operation to obtain weighting coefficients, small-sample spectral space feature extraction convolutional neural network selects cross-entropy loss function, correlation alignment loss function and maximum meanization difference loss function as the small-sample spectrum Loss functions for convolutional neural networks for empty feature extraction.

In this embodiment, the total loss function of the convolutional neural network for small-sample spectral-space feature extraction is:

L _total ＝L _fsl +L _coral +L _MMD

Among them, L _total represents the total loss function of the small-sample spectral space feature extraction convolutional neural network, L _fsl represents the cross-entropy loss function, L _coral represents the correlation alignment loss function, and L _MMD represents the maximum mean difference loss function;

The cross-entropy loss function L _fsl is expressed as:

where L _fsl represents the loss value between the predicted label vector and the true label vector, d( ) represents the Euclidean distance, F _ω ( ) represents the feature extraction function with parameter ω, f _l represents the source domain support set or the target domain support The feature of the lth category in the set, C represents the number of categories, x _j represents a sample in the source domain query set or target domain query set, y _j represents the label of sample x _j , Q represents the source domain query set or target domain queryset.

The correlation alignment loss L _coral is expressed as:

表示矩阵的Frobenius范数，C_S表示源域特征的协方差矩阵，C_T表示目标域特征的协方差矩阵，源域特征为源域支持集和源域查询集经谱分支网络、空间分支网络提取特征后，再经域注意力模块、第一拼接层后拼接的特征，目标域特征为目标域支持集和目标域查询集经谱分支网络、空间分支网络提取特征后，再经域注意力模块、第一拼接层后拼接的特征，d表示特征的维度；in,

Represents the Frobenius norm of the matrix, C _S represents the covariance matrix of the source domain features, C _T represents the covariance matrix of the target domain features, and the source domain features are the source domain support set and the source domain query set through the spectral branch network and the spatial branch network After the features are extracted, the features spliced after the domain attention module and the first splicing layer. The target domain features are the target domain support set and the target domain query set. The features spliced after the module and the first splicing layer, d represents the dimension of the feature;

The maximum meanization difference loss L _MMD is expressed as:

表示空间距离，其是由φ(·)将数据映射到再生希尔伯特空间(RKHS)中进行度量的，φ(·)表示映射函数，/>

表示源域特征，/>

表示目标域特征，X_s表示源域数据集，X_t表示目标域数据集，n_s表示源域数据集中数据的个数，n_t表示目标域数据集中数据的个数。in,

Indicates the spatial distance, which is measured by φ( ) mapping the data into the regenerative Hilbert space (RKHS), φ( ) represents the mapping function, />

Indicates source domain characteristics, />

Represents the characteristics of the target domain, X _s represents the source domain data set, X _t represents the target domain data set, n _s represents the number of data in the source domain data set, n _t represents the number of data in the target domain data set.

Based on the small-sample spectral-space feature extraction convolutional neural network and its total loss function described above, the training method for the small-sample spectral-space feature extraction convolutional neural network is:

Set the initial learning rate of the training to α, and the number of iterations is T. In odd number of iterations, the source domain support set and the source domain query set are sent to the small-sample spectral-space feature extraction convolutional neural network for training, and the total loss function is used to calculate The loss value between the features of the source domain support set and the features of the source domain query set is used to update the parameters in the small-sample spectral-space feature extraction convolutional neural network; at even iterations, the target domain support set and the target domain query set The set is sent to the small-sample spectral-space feature extraction convolutional neural network for training, and the total loss function is used to calculate the loss value between the features of the target domain support set and the features of the target domain query set to update the small-sample spectral-space feature extraction convolution The parameters in the neural network, until the loss value of the small-sample spectral-space feature extraction convolutional neural network no longer decreases and the current number of training rounds is less than the number of iterations T or the number of training rounds reaches the number of iterations T, then stop extracting small-sample spectral-space features The convolutional neural network is trained to obtain a trained small-sample spectral-space feature extraction convolutional neural network.

Specifically, at odd number of iterations, the source domain support set and source domain query set in the source domain data set are sent to the small-sample spectral-space feature extraction convolutional neural network for training, and the features of the source domain support set and the source domain The loss value between the features of the query set; at an even number of iterations, the target domain support set and the target domain query set in the target domain dataset are sent to the small-sample spectral-space feature extraction convolutional neural network for training, and the support set is also calculated Loss value between features and queryset features. Alternately, input the source domain data set and the target domain data set respectively to train the small sample spectral space feature extraction convolutional neural network, and continuously iterate to update the parameters in the network.

The learning rate R of each input hyperspectral image block is set as: R=α.

Perform T weight updates on the small-sample spectral-space feature extraction convolutional neural network to obtain an updated weight vector W _new :

Among them, L _total represents the total loss function, W represents the weight vector of the small-sample spectral-space feature extraction convolutional neural network before updating, and R represents the learning rate.

Input the next training sample set into the small-sample spectral-space feature extraction convolutional neural network, and update the loss function value of the total loss function, so that the loss function value L _total continues to decrease until the loss function value L _total no longer decreases, and the current If the number of training rounds is less than the set number of iterations T, the training of the network is stopped, and the trained small-sample spectral-space feature extraction convolutional neural network is obtained; otherwise, when the number of training rounds reaches T, the training of the network is stopped, and the training of the network is obtained. Trained Convolutional Neural Networks for Few-Sample Spectral-Space Feature Extraction.

Step 5. Input the hyperspectral image to be classified into the trained small-sample spectral-space feature extraction convolutional neural network to obtain the classification result.

In this embodiment, in order to test the trained small-sample spectral-space feature extraction convolutional neural network, the test sample can be input into the trained small-sample spectral-space feature extraction convolutional neural network to obtain the category of the test sample and complete the high-level Classification of spectral images.

First, the spectral branch network constructed by the present invention can extract rich inter-spectral features through the 3D deformable convolution block, and the 3D deformable convolution block can be used to focus on and filter these inter-spectral features to extract more distinguishable features. It overcomes the inability to extract more useful information due to the fixed convolution kernel when extracting inter-spectral features in the prior art, or retains too much irrelevant information to cause information redundancy, and improves the accuracy of ground objects in hyperspectral images. classification accuracy.

Second, through the multi-scale spatial feature extraction module of the spatial branch network constructed in the present invention, the small-sample spectral-space feature extraction convolutional neural network can pay attention to the spatial features of different scales, which overcomes the high cost of single-scale extraction in the prior art. The shortcomings of the spatial features of the spectral image block, through the multi-channel spatial attention mechanism module, can focus on and screen these multi-scale spatial features, extract more resolving spatial features, and overcome the shortcomings of the existing technology in the extraction of spatial features. The loss of information caused by insufficient utilization of the extracted features, or the redundancy of information caused by retaining too much irrelevant information, improves the classification ability of the convolutional neural network during training with few samples.

Third, the small-sample spectral-space feature extraction convolutional neural network of the present invention uses a domain attention module, which is mainly aimed at the problem of excessive redundant information between bands due to the large number of spectral bands in hyperspectral images. Using spectral attention to extract useful inter-spectral features and spatial attention to extract useful spatial features makes the neural network pay more attention to the useful information in feature information. In order to reduce the domain transfer problem caused by training with different hyperspectral data sets, the present invention uses a cross-entropy loss function, a correlation alignment loss function and a maximum mean difference loss function as the loss function of the network. The small-sample spectral-space feature extraction convolutional neural network pays more attention to the category of ground objects whose sample distribution is not concentrated or the sample size is small.

The effects of the present invention will be further described below in combination with simulation experiments.

Simulation experiment conditions:

The hardware platform of the emulation experiment of the present invention is: Inter core i7-6700, frequency is 3.4GHz, NvidiaGeForce RTX3090. The software of the simulation experiment of the present invention uses pytorch.

The simulation experiment of the present invention is to use the present invention and two existing RN-FSC and DCFSL methods to classify the ground objects in the University of Pavia and Indian Pines hyperspectral data sets respectively.

The RN-FSC method refers to a hyperspectral classification method proposed by Kuiliang Gao et al. in "Deep relation network for hyperspectral image few-shot classification" (Remote Sensing, 2020), and its feature learning module and relation learning module can be Make full use of the spatial-spectral information in hyperspectral images to achieve accurate classification of new hyperspectral images with a small number of labeled samples.

The DCFSL method refers to: Rui Li et al. proposed in "Deep crossdomain few-shot learning for hyperspectral image classification" (Remote Sensing, 2021) that uses source class data to help classify target classes. Spectral Classification Method.

The target domain data sets used in the present invention are University of Pavia and Indian Pines hyperspectral data sets, which are collected by AVIRIS sensor at University of Pavia in California and an Indian pine tree in Indiana, USA, respectively. Indian Pines is the earliest test data for hyperspectral image classification. The Airborne Visible Infrared Imaging Spectrometer (AVIRIS) imaged an Indian pine tree in Indiana, USA in 1992, and then cut and marked it with a size of 145×145 It is used as a hyperspectral image classification test. Among them, the image size of the University of Pavia hyperspectral dataset is 610×340, with 103 bands, including 9 types of ground objects, and the categories and quantities of each type of ground objects are shown in Table 1.

Table 1 University of Pavia sample category and quantity

The image size of the Indian Pines hyperspectral dataset is 145×145, with 200 bands, and contains 16 types of ground objects. The categories and quantities of each type of ground objects are shown in Table 2.

Table 2 Indian Pines sample category and quantity

class label Feature category quantity 1 Alfalfa 46 2 Corn-notill 1428 3 Corn-mintill 830 4 corn 237 5 Grass-pasture 483 6 Grass-tree 730 7 Grass-pasture-mowed 28 8 Hay-windowed 478 9 Oats 20 10 Soybean-notill 972 11 Soybean-mintill 2455 12 Soybean-clean 593 13 Wheat 205 14 Woods 1265 15 Buildings-Grass-Trees-Dribes 386 16 Stone-steel-Towers 93

The source domain data set used in the present invention is the Chikusei data set, which is the hyperspectral image of Chikusai, Ibaraki, Japan, obtained by the Hyperspec-VNIR-CIRIS spectrometer. The ground sampling distance is 2.5m, the image size is 2517×2335 pixels, and there are 512 bands in total. There are 128 bands in the spectral range from 363nm to 1018nm, including 19 categories in total. The types and quantities of each type of ground objects are shown in Table 3.

In order to verify the high efficiency and good classification performance of the present invention, three evaluation indexes including overall classification accuracy OA, average accuracy AA and Kappa coefficient are used.

The overall classification accuracy OA refers to the ratio of the number of correctly classified pixels on the test set divided by the total number of pixels, and its value is between 0 and 100%. The larger the value, the better the classification effect .

The average accuracy AA refers to dividing the number of correctly classified pixels of each class on the test set by the total number of all pixels of this class to obtain the correct classification accuracy of this class, and taking the average of the accuracy of all classes, its value Between 0 and 100%, the larger the value, the better the classification effect.

The Kappa coefficient is an evaluation index defined on the confusion matrix, which comprehensively considers the elements on the diagonal of the confusion matrix and the elements off the diagonal, and more objectively reflects the classification performance of the algorithm. The value of the Kappa coefficient is in Between -1 and 1, the larger the value, the better the classification effect.

2. Simulation experiment content and result analysis:

Simulation 1, the present invention and two prior art are respectively carried out classification test in University of Pavia hyperspectral data set, and result figure is as shown in Figure 5, wherein:

Figure 5(a) is the classification result of the existing RN-FSC method on the University of Pavia hyperspectral dataset;

Figure 5(b) is the classification result of the existing DCFSL method on the University of Pavia hyperspectral dataset;

Fig. 5(c) is the classification result on the University of Pavia hyperspectral data set using the method of the present invention.

It can be seen from Fig. 5(c) that the classification result map of the present invention on the University of Pavia dataset is obviously smoother and the edges are clearer than Fig. 5(a) and Fig. 5(b).

Simulation 2, test the Indian Pines hyperspectral data set with the present invention and two prior art respectively, its simulation result figure is as shown in Figure 6, wherein:

Figure 6(a) is the classification result of the existing RN-FSC method on the Indian Pines hyperspectral dataset;

Figure 6(b) is the classification result of the existing DCFSL method on the Indian Pines hyperspectral dataset;

Fig. 6 (c) is the classification result on the Indian Pines hyperspectral data set with the method of the present invention;

In the above two simulations, the classification accuracy of the present invention and the prior art in the University of Pavia hyperspectral data set and the Indian Pines hyperspectral data set were compared, and the results are shown in Table 4.

Table 4 Comparison of classification accuracy of the three networks under two different data sets

It can be seen from Table 4 that under the University of Pavia and Indian Pines data sets, the method of the present invention has achieved higher classification accuracy than the prior art RN-FSC method and DCFSL method, indicating that the present invention can more accurately predict Classes of hyperspectral image samples.

The above simulation experiments show that the method of the present invention can more fully extract inter-spectral features by using the constructed inter-spectral 3D deformable convolution block, and the constructed multi-scale spatial feature extraction block can more fully extract spatial features. The spatial features and inter-spectral features are concatenated, and more distinguishable spectral-spatial features can be obtained through the fully connected layer, and finally the hyperspectral image classification result is obtained through the softmax classifier. The present invention adopts the total loss function composed of cross-entropy loss function, correlation alignment loss function and maximum meanization difference loss function to train the neural network, so that the small-sample spectrum-space feature extraction convolutional neural network pays more attention to the non-concentrated sample distribution or sample size Very few feature categories. It solves the problem of low classification accuracy in the case of few training samples due to the loss of information caused by insufficient utilization of the extracted features in the existing technology in the extraction of spatial features, or the redundancy of information caused by the retention of too much irrelevant information , is a very practical method for hyperspectral image classification with few training samples.

In the description of the invention, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, a feature defined as "first" and "second" may explicitly or implicitly include one or more of these features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic data points are included in at least one embodiment or example of the invention. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristic data points described may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification. The above content is a further detailed description of the present invention in conjunction with specific preferred embodiments, and it cannot be assumed that the specific implementation of the present invention is limited to these descriptions. For those of ordinary skill in the technical field of the present invention, without departing from the concept of the present invention, some simple deduction or replacement can be made, which should be regarded as belonging to the protection scope of the present invention.

Claims (8)

1. The hyperspectral image classification method based on the deep learning of the small sample is characterized by comprising the following steps of:

step 1, acquiring a source domain hyperspectral image set and a target domain hyperspectral image set, wherein the source domain hyperspectral image set comprises a plurality of Zhang Yuanyu hyperspectral images, and the target domain hyperspectral image set comprises a plurality of target domain hyperspectral images;

step 2, respectively filling the edge parts of the source domain hyperspectral image and the target domain hyperspectral image, and extracting a plurality of first image blocks and a plurality of second image blocks by taking each pixel point of the filled source domain hyperspectral image and target domain hyperspectral image as a center;

step 3, in each category, randomly selecting part of the first image blocks to form a source domain support set and part of the first image blocks to form a source domain query set, and randomly selecting part of the second image blocks to form a target domain support set and part of the first image blocks to form a target domain query set;

step 4, based on a random gradient descent method, carrying out alternate training on a small sample spectral space feature extraction convolutional neural network by using the source domain support set, the source domain query set, the target domain support set and the target domain query set to obtain a trained small sample spectral space feature extraction convolutional neural network, wherein the small sample spectral space feature extraction convolutional neural network extracts features from two aspects of spectrum and space, and a total loss function of the small sample spectral space feature extraction convolutional neural network consists of a cross entropy loss function, a correlation alignment loss function and a maximum average difference loss function;

And step 5, inputting the hyperspectral image to be classified into the trained small sample spectral space feature extraction convolutional neural network to obtain a classification result.

2. The hyperspectral image classification method based on small sample deep learning as claimed in claim 1, wherein the step 2 includes:

step 2.1, filling pixels with pixel values of 0 into the peripheries of the source domain hyperspectral image and the target domain hyperspectral image respectively to obtain a filled source domain hyperspectral image and a filled target domain hyperspectral image;

and 2.2, selecting the first image block and the second image block with the space size of (2t+1) x (2t+1) and the channel number of d by taking each pixel point in the filled source domain hyperspectral image and the filled target domain hyperspectral image as a center, wherein t is an integer greater than 0.

3. The hyperspectral image classification method based on small sample deep learning as claimed in claim 1, wherein the structure of the small sample spectral space feature extraction convolutional neural network comprises a spectral branch network, a spatial branch network, a domain attention module, a first splicing layer, a full connection layer and a softmax classifier, the spectral branch network, the spatial branch network and the domain attention module are connected in parallel and then sequentially connected in series, the domain attention module, the first splicing layer, the full connection layer and the softmax classifier are connected in series, the spectral branch network comprises 2 3D deformable convolution blocks and 2 first maximum pooling layers, the 1 st 3D deformable convolution block, the 1 st first maximum pooling layer, the 2 nd 3D deformable convolution block and the 2 nd first maximum pooling layer are sequentially connected in series, and the spatial branch network domain attention module comprises a multi-scale spatial feature extraction module, a second splicing layer and a second maximum pooling layer which are sequentially connected in series.

4. The hyperspectral image classification method based on small sample deep learning as claimed in claim 3, wherein the 3D deformable convolution block includes 3D deformable convolution layers and 3 first activation function layers, the 1 st 3D deformable convolution layer, the 1 st first activation function layer, the 2 nd 3D deformable convolution layer, the 2 nd first activation function layer, the 3 rd 3D deformable convolution layer, and the 3 rd first activation function layer are sequentially connected in series, and the output of the 1 st first activation function layer and the output of the 3 rd 3D deformable convolution layer are added to form a residual structure.

5. The hyperspectral image classification method based on small sample deep learning as claimed in claim 3, wherein the multi-scale spatial feature extraction module comprises 2 scale operation layers, 3 convolution layers, 3 normalization layers and 3 second activation function layers;

the 1 st convolution layer, the 1 st normalization layer and the 1 st second activation function layer are sequentially connected in series;

the 1 st scale operation layer, the 2 nd convolution layer, the 2 nd normalization layer and the 2 nd second activation function layer are sequentially connected in series;

the 2 nd scale operation layer, the 3 rd convolution layer, the 3 rd normalization layer and the 3 rd second activation function layer are sequentially connected in series;

The 1 st second activation function layer, the 2 nd second activation function layer and the 3 rd second activation function layer are connected in parallel and then connected in series with the second splicing layer and the second maximum pooling layer in sequence.

6. The hyperspectral image classification method based on small sample deep learning as claimed in claim 1, wherein the total loss function of the small sample spectral null feature extraction convolutional neural network is:

L _total ＝L _fsl +L _coral +L _MMD

wherein ,L_total Representing the total loss function L of the small sample spectral space feature extraction convolutional neural network _fsl Represents cross entropy loss, L _coral Representing dependency alignment loss, L _MMD Representing the maximum averaged delta loss;

cross entropy loss L _fsl Expressed as:

wherein d (·) represents the Euclidean distance, F _ω (. Cndot.) the feature extraction function of parameter ω, f _l Features representing the first class in the source domain support set or the target domain support set, C being the number of classes, x _j Representing one sample in a source domain query set or a target domain query set, y _j Representing sample x _j Q represents a source domain query set or a target domain query set;

correlation alignment loss L _coral Expressed as:

wherein ,

frobenius norms, C representing matrix _S Covariance matrix representing source domain features, C _T A covariance matrix representing the characteristics of the target domain, and d represents the dimension of the characteristics;

maximum averaged difference loss L _MMD Expressed as:

wherein ,

represents the spatial distance, phi (·) represents the mapping function, +.>

Representing source domain features, ++>

Representing the characteristics of the target domain, X _s Representing a source domain dataset, X _t Representing a target domain dataset, n _s Representing the number of data in the source domain dataset, n _t Representing the number of data in the target domain dataset.

7. The hyperspectral image classification method based on small sample deep learning as claimed in claim 6, wherein the step 4 includes:

setting the initial learning rate of training as alpha and the iteration times as T, and sending the source domain support set and the source domain query set into the small sample spectral space feature extraction convolutional neural network for training when the iteration is performed for odd times, and calculating the loss value between the features of the source domain support set and the features of the source domain query set by using the total loss function so as to update the parameters in the small sample spectral space feature extraction convolutional neural network; and when the number of iterations is even, the target domain support set and the target domain query set are sent into the small sample spectral space feature extraction convolutional neural network to train, the total loss function is utilized to calculate the loss value between the features of the target domain support set and the features of the target domain query set so as to update the parameters in the small sample spectral space feature extraction convolutional neural network until the loss value of the small sample spectral space feature extraction convolutional neural network is not reduced any more and the current training round number is smaller than the iteration number T or the training round number reaches the iteration number T, and training of the small sample spectral space feature extraction convolutional neural network is stopped to obtain the trained small sample spectral space feature extraction convolutional neural network.

8. The hyperspectral image classification method based on small sample deep learning as claimed in claim 7, wherein the small sample spectral null feature extraction convolutional neural network updated weight vector W _new The method comprises the following steps:

wherein ,L_total The method is characterized in that the method comprises the steps of representing a total loss function of a small sample spectral null feature extraction convolutional neural network, W represents a weight vector before updating the small sample spectral null feature extraction convolutional neural network, and R represents a learning rate.

Images