CN111563520B - Hyperspectral image classification method based on space-spectrum combined attention mechanism - Google Patents
Hyperspectral image classification method based on space-spectrum combined attention mechanism Download PDFInfo
- Publication number
- CN111563520B CN111563520B CN202010044989.1A CN202010044989A CN111563520B CN 111563520 B CN111563520 B CN 111563520B CN 202010044989 A CN202010044989 A CN 202010044989A CN 111563520 B CN111563520 B CN 111563520B
- Authority
- CN
- China
- Prior art keywords
- attention
- space
- spectral
- spectrum
- attention score
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The algorithm is used for solving the problem that the performance of a traditional convolutional neural network on a fine-grained image classification task represented by a hyperspectral image is insufficient, and the hyperspectral image classification algorithm based on a space-spectrum combined attention mechanism is provided, can effectively capture image global features by matching with the convolutional neural network, and adaptively focuses on spatial local features with large differences among similar images; meanwhile, contributions of different wave bands to a task are evaluated, so that the neural network pays more attention to the spectrum wave bands with large contributions, local difference characteristics of image spectrums are extracted, the hyperspectral image classification precision is improved, and the method has wide application in the field of classification of fine-grained images represented by hyperspectral images.
Description
Technical Field
The invention relates to a hyperspectral image classification method based on a space-spectrum combined attention mechanism. The method can be used in the field of remote sensing image processing.
Background
The hyperspectral remote sensing technology is one of the most important technical breakthroughs in the field of airborne observation systems and satellite-borne observation systems for human beings at the end of the twentieth century, the hyperspectral image overcomes the limitations of the traditional single-waveband and multispectral remote sensing in the aspects of waveband range, waveband quantity, fine ground target observation and identification and the like, and has unique advantages in the field of remote sensing ground observation. The hyperspectral image classification is an important and meaningful task in practice, and particularly, the hyperspectral image classification is a task of identifying a given image according to different spectral features or spatial features and marking each pixel point type in the image.
Compared with the common image classification task, the hyperspectral image has the characteristic of dimensionality disaster and homospectral foreign matter in a spectral domain, so that the classification task is more difficult. Under the circumstances, the performance of the traditional hyperspectral image classification algorithm which solely depends on spectral information is limited, and the classification algorithm based on the joint space-spectral information is a research hotspot in recent years.
Since 2012, the deep learning technique, represented by Convolutional Neural Network (CNN), was a great achievement in computer vision tasks. The convolutional neural network is very suitable for processing image space domain information and has achieved great success in common image classification tasks, and the convolutional neural network is used for hyperspectral image classification tasks at the earliest years. Subsequently, various convolutional neural network algorithms for the hyperspectral image classification task are developed, but the algorithms have difficulty in extracting image global features due to the limited size of the 'receptive field' of the convolutional network. What is worse, due to the particularity of the hyperspectral image data, the hyperspectral image data needs to be preprocessed before being classified, that is, each pixel is divided into cubes (the general size is 27 × 27) as the center, and the middle pixel label is used as each cube classification label, so that similar and heterogeneous pixel cubes are very similar in spatial features, which is generally called as the redundancy of the overall spatial features, and the image with slight difference of local features is a fine-grained image. The ability of the traditional convolutional neural network to process the fine-grained image with the spatial redundancy characteristic is very weak, and the performance of the convolutional neural network in the fine-grained image classification tasks such as hyperspectral and the like is seriously improved.
In addition, different from common images, hyperspectral images have very rich spectral information, most of the traditional classification algorithms consider that different spectral bands contribute the same to the algorithm task, but actually, due to the influence of physical factors such as illumination, atmosphere and the like, some bands tend to be noisy, and basically do not contribute to the current task, or even cause interference.
Based on the method, a spatial local feature which can effectively capture the global feature of the image and adaptively focus the images with similar fineness and with larger difference is designed; meanwhile, the contributions of different wave bands to the task are evaluated, so that the neural network pays more attention to the spectrum wave bands with large contributions, the local difference characteristics of the image spectrum are extracted, the high-spectrum image classification precision is improved, and the method is a very worthy of research.
Disclosure of Invention
The algorithm aims at solving the problem that the traditional convolutional neural network has insufficient classification performance on the fine images represented by the hyperspectral images, and provides a hyperspectral image classification method based on a space-spectrum combined attention mechanism, which can be matched with the convolutional neural network to effectively capture the global features of the images and adaptively focus the spatial local features with larger differences among similar images; meanwhile, different wave bands are evaluated to contribute to tasks, so that the neural network pays more attention to the spectrum wave bands contributing to large, local difference features of the images are extracted, the classification precision of the hyperspectral images is improved, and the method has wide application in the field of classification of fine images such as hyperspectrum.
The algorithm of the invention provides a space-spectrum combined attention mechanism module, which has the following three advantages:
(1) The algorithm has strong portability and can be randomly embedded into various conventional convolutional neural networks.
(2) The algorithm has good universality, and attention mechanism modules are flexibly selected according to task requirements. For example, when a common fine image classification task without spectral features is performed, a spatial attention machine module can be flexibly selected.
(3) The performance of the algorithm is strong, and the performance of the convolutional neural network can be effectively improved;
drawings
FIG. 1 is a block diagram of a spatial-spectral combined attention mechanism;
FIG. 2 is a block diagram of three structures of a convolutional neural network embedded with a spatial-spectral combined attention mechanism module;
FIG. 3 is a comparison of experimental results of different algorithms on a hyperspectral dataset. Note: in the experiment, a space-spectrum combined Attention mechanism Module is called Joint Spatial-Spectral Attention Module for short, JSAM for short, a convolutional neural network adopting a series embedding mode is called CNN-JSAM-A, a convolutional neural network adopting a parallel embedding mode is called CNN-JSAM-B, and a convolutional neural network adopting a series parallel embedding mode is called CNN-JSAM-C. Indian Pine data is taken as a high-spectrum data set in an experiment, 10% of the Indian Pine data is taken as a training set, and all network parameters and layer numbers of the CNNs are kept consistent, so that the difference is whether a space-spectrum combined attention mechanism module JSAM is embedded.
Detailed Description
As shown in fig. 1, the spatial-spectral combined attention mechanism module mainly comprises three sub-modules: a spatial attention score extraction sub-module, a spectral attention score extraction sub-module, and an attention score assignment sub-module. The spatial attention score extraction submodule mainly extracts similarity characteristics between any two pixels in a space to obtain a spatial attention score map; the spectral attention fraction extraction submodule mainly extracts correlation dependencies in different spectral bands to obtain an attention fraction graph of the spectral bands; and the attention score distribution branch distributes the respectively extracted spatial attention scores and the spectral attention scores to the original feature space to obtain an attention score cube containing attention features of different spatial domains and different wave bands.
(1) Spatial attention score extraction submodule
The hyperspectral cube of the input network is denoted by X as follows:
wherein H is the length of the input hyperspectral cube;
w is the width of the input hyperspectral cube;
c is the spectral dimension of the input hyperspectral cube;
and N = H × W;
the method comprises the following steps: respectively mapping an input image X according to a formula (1) into an embedded spectral feature space to obtain two new feature maps theta (X) and phi (X);
wherein i and j are the numbers of pixels in the feature map;
d is the spectral dimension mapped to new feature maps θ (X) and φ (X) in the embedded spectral space;
step two: calculating the similarity s of any two pixels by using a Gaussian function embedded in space ij Obtaining a spatial attention point map S, and specifically calculating a process map formula (2) and shown in FIG. 1:
wherein s is ij Representing the similarity between the ith and jth pixels;
in the procedure, W θ And W φ The network parameters are learnable and are realized by adopting 1 × 1 convolution layers; first in formula (2) θ(xi) Transposing to obtain theta (x) i ) T Then, theta (x) is added i ) T Phi (x) j ) And performing matrix multiplication operation, and finally performing normalization operation by using a neural network softmax layer.
(2) Spectral attention score extraction submodule
The hyperspectral cube of the input network is denoted by X below
Wherein H is the length of the input hyperspectral cube;
w is the width of the input hyperspectral cube;
c is the spectral dimension of the input hyperspectral cube;
the method comprises the following steps: respectively mapping an input image X according to a formula (4) into an embedding space feature space to obtain two new feature maps upsilon (X) and omega (X);
wherein i and j are numbers of spectral bands corresponding to the characteristic diagram;
W υ and W ω Linear mapping matrixes are adopted, and the linear mapping matrixes are parameters which can be learned in the neural network;
step two: calculating the similarity q of the corresponding characteristic graphs of any two spectral bands by using a Gaussian function embedded in space ij Obtaining a spatial attention point map Q, and specifically calculating a process map formula (5) and shown in FIG. 1:
wherein q is ij Representing the similarity between corresponding signatures in the ith and jth spectral bands;
in the procedure, W υ And W ω The method is a learnable network parameter and is realized by adopting a 3X 3Depth-wise convolution layer; in the formula (5), v (x) is first measured i ) Transposing the resulting product to give v (x) i ) T Then v (x) i ) T And ω (x) i ) And performing matrix multiplication operation, and finally performing normalization operation by using a neural network softmax layer.
(3) Attention score assignment submodule
The attention score distribution submodule has the main function of distributing the spatial attention scores and the spectral attention scores extracted respectively to the original feature space by the attention score distribution branch to obtain the attention score cube containing the attention features of different spatial domains and different wave bands.
The input image X is represented as follows:
the method comprises the following steps: in order to ensure that the attention mechanism module can adaptively focus on the local space and the local spectral band of the feature map according to task requirements, firstly mapping is carried out in the feature space to obtain a brand new feature map
As in equation (7); in the program, formula (7) is implemented by using 3 × 3 convolution layers, where W is ζ Is the 3 x 3 convolution kernel parameter.
A＝S·ζ(X)·Q (8)
Step two: the attention mechanism score cube a is obtained by assigning the spatial attention score S and the spectral attention score Q to the original feature space by formula (8).
In addition, the algorithm also designs a set of spatial-spectral combined attention mechanism module and convolution neural network embedding modes, which mainly comprise the following three embedding modes:
(1) Series embedded mode
(2) Parallel embedded mode
(3) Series-parallel connection embedding mode
Detailed diagrams of three structures of the convolutional neural network embedded with the spatial-spectral combined attention mechanism module are shown in FIG. 2.
Claims (1)
1. A hyperspectral image classification method based on a space-spectrum combined attention mechanism mainly comprises a space-spectrum combined attention mechanism module and a convolution neural network embedded mode:
1. the space-spectrum combined attention mechanism module consists of three sub-modules, namely a space attention score extraction sub-module, a spectrum attention score extraction sub-module and an attention score distribution sub-module; the spatial attention score extraction branch extracts similarity characteristics between any two pixels in a space to obtain a spatial attention score map; the spectral attention fraction extraction branch extracts the related dependencies in different spectral bands to obtain an attention fraction graph of the spectral bands; distributing the spatial attention score map and the spectral attention score map respectively extracted by the attention score distribution submodule into the original feature space pixel by pixel and spectrum by spectrum to obtain an attention score cube containing different pixel points and attention features of different wave bands; the method comprises the following specific steps:
(1) Spatial attention score extraction submodule
The method comprises the following steps: mapping the input image X into an embedded spectral feature space respectively to obtain two new feature maps theta (X) and phi (X);
step two: calculating the similarity s of any two pixels by using a Gaussian function embedded in space ij Obtaining a spatial attention score map S, and finally performing normalization operation by using a neural network softmax layer;
(2) Spectral attention score extraction submodule
Step three: mapping an input image X into an embedding space feature space respectively to obtain two new feature maps u (X) and omega (X);
step four: calculating the similarity q of the corresponding characteristic graphs of any two spectral bands by using a Gaussian function embedded in space ij Obtaining a spectrum attention point diagram Q, wherein in the experiment, the spectrum attention point diagram Q is realized by adopting a 3 × 3 layered (Depth-wise) convolution layer; finally, carrying out normalization operation by utilizing a neural network Softmax layer;
(3) Attention score assignment submodule
The attention score distribution submodule is used for distributing the extracted spatial attention score and the spectral attention score to an original feature space to obtain an attention score cube containing attention features of different spatial domains and different wave bands;
step five: in order to ensure that the attention mechanism module can adaptively focus on the local space and the local spectral band of the feature map according to task requirements, firstly mapping is carried out in the feature space, and a brand new feature map is obtained by adopting 3-by-3 convolution operation
Step six: distributing the space attention score S and the spectrum attention score Q to the original feature space to obtain an attention mechanism score cube A
2. The spatial-spectral combined attention mechanism module is embedded into the convolutional neural network in three ways:
(1) A serial embedding mode;
(2) A parallel embedding mode;
(3) And a series-parallel embedding mode.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010044989.1A CN111563520B (en) | 2020-01-16 | 2020-01-16 | Hyperspectral image classification method based on space-spectrum combined attention mechanism |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010044989.1A CN111563520B (en) | 2020-01-16 | 2020-01-16 | Hyperspectral image classification method based on space-spectrum combined attention mechanism |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111563520A CN111563520A (en) | 2020-08-21 |
| CN111563520B true CN111563520B (en) | 2023-01-13 |
Family
ID=72071383
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010044989.1A Active CN111563520B (en) | 2020-01-16 | 2020-01-16 | Hyperspectral image classification method based on space-spectrum combined attention mechanism |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111563520B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112052755B (en) * | 2020-08-24 | 2023-06-02 | 西安电子科技大学 | Semantic convolution hyperspectral image classification method based on multipath attention mechanism |
| CN112232343B (en) * | 2020-09-03 | 2023-11-21 | 国家粮食和物资储备局科学研究院 | Grain mildew grain identification neural network and identification method |
| CN112287989B (en) * | 2020-10-20 | 2022-06-07 | 武汉大学 | Aerial image ground object classification method based on self-attention mechanism |
| CN113537239B (en) * | 2021-07-08 | 2022-02-18 | 宁波大学 | Hyperspectral image band selection method based on global relationship perception attention |
| CN114462596B (en) * | 2022-02-10 | 2023-04-07 | 黑龙江省农业科学院 | Disease and insect pest monitoring method and monitoring system for industrial hemp growth period |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109376804A (en) * | 2018-12-19 | 2019-02-22 | 中国地质大学（武汉） | Based on attention mechanism and convolutional neural networks Classification of hyperspectral remote sensing image method |
| CN109993220A (en) * | 2019-03-23 | 2019-07-09 | 西安电子科技大学 | Multi-source Remote Sensing Images Classification method based on two-way attention fused neural network |
| CN110516596A (en) * | 2019-08-27 | 2019-11-29 | 西安电子科技大学 | Empty spectrum attention hyperspectral image classification method based on Octave convolution |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| SG172262A1 (en) * | 2008-12-19 | 2011-07-28 | Agency Science Tech & Res | Device and method for generating a representation of a subject's attention level |
| CN110458192B (en) * | 2019-07-05 | 2022-06-14 | 中国地质大学（武汉） | Hyperspectral remote sensing image classification method and system based on visual saliency |
-
2020
- 2020-01-16 CN CN202010044989.1A patent/CN111563520B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109376804A (en) * | 2018-12-19 | 2019-02-22 | 中国地质大学（武汉） | Based on attention mechanism and convolutional neural networks Classification of hyperspectral remote sensing image method |
| CN109993220A (en) * | 2019-03-23 | 2019-07-09 | 西安电子科技大学 | Multi-source Remote Sensing Images Classification method based on two-way attention fused neural network |
| CN110516596A (en) * | 2019-08-27 | 2019-11-29 | 西安电子科技大学 | Empty spectrum attention hyperspectral image classification method based on Octave convolution |
Non-Patent Citations (1)
| Title |
|---|
| Spectral-Spatial Attention Networks for Hyperspectral Image Classification;Xiaoguang Mei 等;《Remote Sensing》;20190423;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111563520A (en) | 2020-08-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111563520B (en) | Hyperspectral image classification method based on space-spectrum combined attention mechanism | |
| Chen et al. | Land-use scene classification using multi-scale completed local binary patterns | |
| Lin et al. | Multispectral change detection with bilinear convolutional neural networks | |
| CN111160273B (en) | Hyperspectral image spatial spectrum joint classification method and device | |
| Zhu et al. | Plant identification based on very deep convolutional neural networks | |
| CN109961096B (en) | Multimode hyperspectral image migration classification method | |
| Thoonen et al. | Multisource classification of color and hyperspectral images using color attribute profiles and composite decision fusion | |
| Hang et al. | Graph regularized nonlinear ridge regression for remote sensing data analysis | |
| Chen et al. | Convolutional neural network for classification of solar radio spectrum | |
| Ye et al. | Hyperspectral image classification using principal components-based smooth ordering and multiple 1-D interpolation | |
| El-Rahiem et al. | An efficient multi-biometric cancellable biometric scheme based on deep fusion and deep dream | |
| Shi et al. | F 3 Net: Fast Fourier filter network for hyperspectral image classification | |
| Rahiche et al. | Blind decomposition of multispectral document images using orthogonal nonnegative matrix factorization | |
| Su et al. | Probabilistic collaborative representation based ensemble learning for classification of wetland hyperspectral imagery | |
| Jafarzadeh et al. | Wet-GC: A Novel Multimodel Graph Convolutional Approach for Wetland Classification Using Sentinel-1 and 2 Imagery with Limited Training Samples | |
| Hamam et al. | Single-band infrared texture-based image colorization | |
| CN116486238A (en) | Target fine granularity identification method combining point set representation and graph classification | |
| Giri et al. | Enhanced Hyperspectral Image Classification Through Pretrained CNN Model for Robust Spatial Feature Extraction | |
| Qayyum et al. | Optimal feature extraction technique for crop classification using aerial imagery | |
| Ipe et al. | Periocular recognition under unconstrained conditions using CNN-based super-resolution | |
| Chhapariya et al. | A Deep Spectral-Spatial Residual Attention Network for Hyperspectral Image Classification | |
| CN113850316A (en) | Hyperspectral image classification method and device of combined spectrum space multilayer perceptron | |
| CN114037922A (en) | Aerial image segmentation method based on hierarchical context network | |
| Ghoshal et al. | Improving performance of convolutional neural networks via feature embedding | |
| Aptoula et al. | Segmentation networks reinforced with attribute profiles for large scale land-cover map production |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |