CN111563520B - Hyperspectral image classification method based on space-spectrum combined attention mechanism - Google Patents

Abstract

The algorithm of the present invention is dedicated to solving the problem of insufficient performance of the traditional convolutional neural network for fine-grained image classification tasks represented by hyperspectral images, and proposes a hyperspectral image classification algorithm based on the space-spectral joint attention mechanism. The network can effectively capture the global features of the image, and adaptively focus on the spatial local features with large differences between similar images; at the same time, it can evaluate the contribution of different bands to the task, so that the neural network can pay more attention to the spectral bands with large contributions and extract the image spectrum. Local difference features improve the classification accuracy of hyperspectral images, and have a wide range of applications in the field of fine-grained image classification represented by hyperspectral images.

Description

A Hyperspectral Image Classification Method Based on Spatial-Spectral Joint Attention Mechanism

technical field

The present invention is a hyperspectral image classification method based on the space-spectrum joint attention mechanism. By embedding the space-spectrum joint attention mechanism module provided by the method into the convolutional neural network, the global features of the image can be effectively captured, and Adaptively focus on local areas with large differences between images, greatly improve the feature extraction and expression capabilities of convolutional neural networks, and realize the classification of hyperspectral images. This method can be used in the field of remote sensing image processing.

Background technique

Hyperspectral remote sensing technology is one of the most important technological breakthroughs in the field of airborne observation systems and spaceborne observation systems at the end of the 20th century. Hyperspectral images overcome traditional single-band and multi-spectral remote sensing in terms of band range, number of bands, and fine ground targets. It has its unique advantages in the field of remote sensing and earth observation due to limitations in observation and identification. Hyperspectral image classification is an important and meaningful task in practice. Specifically, hyperspectral image classification is the task of identifying a given image and labeling each pixel in the image according to different spectral features or spatial features. .

Compared with ordinary image classification tasks, hyperspectral images make classification tasks more difficult due to the "curse of dimensionality" and "same spectrum foreign objects" in the spectral domain. In this case, traditional hyperspectral image classification algorithms relying solely on spectral information have limited performance, and classification algorithms based on joint spatial-spectral information have been a research hotspot in recent years.

Since 2012, the deep learning technology represented by Convolution Neural Network (CNN) has sprung up and made great achievements in computer vision tasks. Convolutional neural networks are very suitable for processing image spatial domain information and have achieved great success in common image classification tasks. Convolutional neural networks were first used in hyperspectral image classification tasks in 2016. Subsequently, various convolutional neural network algorithms for hyperspectral image classification tasks emerged in an endless stream, but these algorithms are difficult to extract global image features due to the limited size of the convolutional network "receptive field". To make matters worse, due to the particularity of hyperspectral image data, it is necessary to preprocess the hyperspectral data before classification, that is, divide each pixel into cubes (the general size is 27*27), and use the middle pixel label Classify labels for each cube, so that similar and heterogeneous pixel cubes are very similar in spatial characteristics, and images with redundant overall spatial characteristics and small differences in local characteristics are usually called fine-grained images. However, the ability of traditional convolutional neural networks to process such fine-grained images with spatial redundancy features is very weak, which seriously restricts the further improvement of the performance of convolutional neural networks in fine-grained image classification tasks such as hyperspectral.

In addition, unlike ordinary images, hyperspectral images have very rich spectral information. Most traditional classification algorithms believe that different spectral bands contribute equally to algorithm tasks, but in fact, due to physical factors such as illumination and atmosphere, some bands tend to be noisy. It basically does not contribute to the current task, and even causes interference.

Based on this, we design a method that can effectively capture the global features of the image, and adaptively focus on the spatial local features with large differences between images of similar fineness; at the same time, evaluate the contribution of different bands to the task, so that the neural network can pay more attention to the large contribution. Spectral bands, extracting image spectral local difference features, and improving the classification accuracy of hyperspectral images are very worthy of research.

Contents of the invention

This algorithm is dedicated to solving the problem of insufficient classification performance of traditional convolutional neural networks for fine-grained images represented by hyperspectral images. We propose a hyperspectral image classification method based on a spatial-spectral joint attention mechanism, combined with convolutional neural networks. It can effectively capture the global features of the image, and adaptively focus on the spatial local features with large differences between similar images; at the same time, it can evaluate the contribution of different bands to the task, so that the neural network can pay more attention to the spectral bands with large contributions, and extract the local differences in the image Features, improve the accuracy of hyperspectral image classification, and have a wide range of applications in the field of hyperspectral and other fine-grained image classification.

The algorithm of the present invention proposes a spatial-spectral joint attention mechanism module, which has the following three advantages:

(1) The algorithm is highly portable and can be embedded in various existing convolutional neural networks at will.

(2) The algorithm has good versatility, and the attention mechanism module can be flexibly selected according to the task requirements. For example, when facing ordinary fine-grained image classification tasks without spectral features, the spatial attention mechanism module can be flexibly selected.

(3) The performance of the algorithm is powerful, which can effectively improve the performance of the convolutional neural network;

Description of drawings

Figure 1 is a block diagram of the spatial-spectral joint attention mechanism;

Fig. 2 is three structural block diagrams of embedded space-spectral joint attention mechanism module convolutional neural network;

Figure 3 is a comparison of experimental results of different algorithms in hyperspectral datasets. Note: In the experiment, the spatial-spectral joint attention mechanism module is referred to as the Joint Spatial-Spectral Attention Module, referred to as JSSAM, and the convolutional neural network using the serial embedding method is called CNN-JSSAM-A, and the convolutional neural network using the parallel embedding method is called CNN-JSSAM-A. The product neural network is called CNN-JSSAM-B, and the convolutional neural network using serial-parallel embedding is called CNN-JSSAM-C. Taking the Indian Pine data as the hyperspectral data set in the experiment, and taking 10% as the training set, the network parameters and the number of layers of the above CNNs are all consistent, the difference is whether to embed the spatial-spectral joint attention mechanism module JSSAM.

detailed description

As shown in Figure 1, the spatial-spectral joint attention mechanism module is mainly composed of three submodules: the spatial attention score extraction submodule, the spectral attention score extraction submodule and the attention score distribution submodule. Among them, the spatial attention score extraction sub-module mainly extracts the similarity features between any two pixels in the space, and obtains the spatial attention score map; the spectral attention score extraction sub-module mainly extracts the correlation dependence in different spectral bands, and obtains the Attention score map; the attention score assignment branch assigns the extracted spatial attention scores and spectral attention scores to the original feature space, and obtains an attention score cube containing attention features of different spatial domains and different bands.

(1) Spatial attention score extraction sub-module

The hyperspectral cube that will be fed into the network is denoted by X below:

where H is the length of the input hyperspectral cube;

W is the width of the input hyperspectral cube;

C is the spectral dimension of the input hyperspectral cube;

And N=H×W;

Step 1: Map the input image X according to formula (1) into the embedded spectral feature space to obtain two new feature maps θ(X) and φ(X);

Among them, i and j are the numbers of pixels in the feature map;

和

是线性映射矩阵，它们都是神经网络中可以学习的 参数；

and

is a linear mapping matrix, they are all parameters that can be learned in the neural network;

D is the spectral dimensionality mapped to new feature maps θ(X) and φ(X) in the embedded spectral space;

Step 2: Use the Gaussian function in the embedded space to calculate the similarity s _ij of any two pixels, and obtain the spatial attention score map S. The specific calculation process is shown in formula (2) and Figure 1:

where s _ij represents the similarity between the i-th and j-th pixels;

In the program, W _θ and W _φ are learnable network parameters, implemented by 1*1 convolutional layer; in formula (2), first transpose _θ(xi) to get θ( _xi ) ^T , and then convert θ( x _i ) ^T and φ(x _j ) perform matrix multiplication, and finally use the softmax layer of the neural network for normalization.

(2) Spectral attention score extraction sub-module

The hyperspectral cube that will be input to the network is represented by X below

where H is the length of the input hyperspectral cube;

W is the width of the input hyperspectral cube;

C is the spectral dimension of the input hyperspectral cube;

Step 1: Map the input image X according to the formula (4) into the embedding space feature space to obtain two new feature maps υ(X) and ω(X);

Among them, i and j are the numbers of the spectral bands corresponding to the feature map;

W _υ and W _ω are linear mapping matrices, which are parameters that can be learned in neural networks;

Step 2: Use the Gaussian function embedded in the space to calculate the similarity q _ij of the feature maps corresponding to any two spectral bands, and obtain the spatial attention score map Q. The specific calculation process is shown in formula (5) and Figure 1:

Among them, q _ij represents the similarity between the feature maps corresponding to the i-th and j-th spectral bands;

In the program, W _υ and W _ω are learnable network parameters, which are realized by 3*3Depth-wise convolutional layer; in formula (5), first transpose υ( _xi ) to get υ( _xi ) ^T , and then Perform matrix multiplication between υ( _xi ) ^T and ω( _xi ), and finally use the softmax layer of the neural network for normalization.

(3) Attention score allocation sub-module

The main function of the attention score allocation sub-module is to assign the attention score allocation branch to the extracted spatial attention score and spectral attention score into the original feature space, and obtain attention features that include different spatial domains and different bands. Force Fraction Cube.

The input image X is represented as follows:

如公式(7)；程序中，公式(7)是利用3*3 卷积层实现，其中W_ζ是3*3卷积核参数。Step 1: In order to ensure that the attention mechanism module can adaptively focus on the local space and local spectral bands of the feature map according to the task requirements, first map in the feature space to obtain a new feature map

Such as formula (7); in the program, formula (7) is implemented using a 3*3 convolutional layer, where W _ζ is a 3*3 convolution kernel parameter.

A＝S·ζ(X)·Q (8)

Step 2: Distribute the spatial attention score S and the spectral attention score Q into the original feature space through formula (8), and obtain the attention mechanism score cube A.

In addition, the algorithm also designs a set of spatial-spectral joint attention mechanism modules and convolutional neural network embedding methods. There are mainly three embedding methods as follows:

(1) Embedded in series

(2) Parallel embedded mode

(3) Embedded in series and parallel

See Figure 2 for details, which are three structural block diagrams of the convolutional neural network embedded in the spatial-spectral joint attention mechanism module.

Claims (1)

1. A hyperspectral image classification method based on the spatial-spectral joint attention mechanism is mainly composed of two parts: the spatial-spectral joint attention mechanism module and the embedded convolutional neural network method: 1. The spatial-spectral joint attention mechanism module consists of three submodules, the spatial attention score extraction submodule, the spectral attention score extraction submodule and the attention score allocation submodule; among them, the spatial attention score extraction branch extracts the spatial The similarity feature between any two pixels obtains the spatial attention score map; the spectral attention score extraction branch extracts the correlation dependence in different spectral bands, and obtains the attention score map of the spectral band; then extracts the attention score distribution sub-module separately The obtained spatial attention score map and spectral attention score map are assigned to the original feature space pixel by pixel and spectrum, and the attention score cube containing different pixel points and different band attention features is obtained; the details are as follows: (1) Spatial attention score extraction sub-module Step 1: Map the input image X to the embedded spectral feature space to obtain two new feature maps θ(X) and φ(X); Step 2: Use the Gaussian function in the embedded space to calculate the similarity s _ij of any two pixels, obtain the spatial attention score map S, and finally use the softmax layer of the neural network to perform normalization; (2) Spectral attention score extraction sub-module Step 3: Map the input image X to the embedding space feature space to obtain two new feature maps u(X) and ω(X); Step 4: Use the Gaussian function embedded in the space to calculate the similarity q _ij of the feature maps corresponding to any two spectral bands, and obtain the spectral attention score map Q. In the experiment, a 3*3 layered (Depth-wise) convolutional layer is used Realize; finally use the neural network Softmax layer to perform normalization operation; (3) Attention score allocation sub-module The role of the attention score assignment sub-module is to assign the extracted spatial attention scores and spectral attention scores to the original feature space, and obtain attention score cubes containing different spatial domains and different band attention features; Step 5: In order to ensure that the attention mechanism module can adaptively focus on the local space and local spectral bands of the feature map according to the task requirements, first map in the feature space, and use 3*3 convolution operation to obtain a new feature map

Step 6: Assign the spatial attention score S and the spectral attention score Q to the original feature space, and obtain the attention mechanism score cube A 2. There are three ways to embed the spatial-spectral joint attention mechanism module into the convolutional neural network: (1) Embedded in series; (2) Parallel embedded mode; (3) Embedded in series and parallel.

Images