CN111563520A - Hyperspectral image classification method based on space-spectrum combined attention mechanism - Google Patents

Hyperspectral image classification method based on space-spectrum combined attention mechanism Download PDF

Info

Publication number
CN111563520A
CN111563520A CN202010044989.1A CN202010044989A CN111563520A CN 111563520 A CN111563520 A CN 111563520A CN 202010044989 A CN202010044989 A CN 202010044989A CN 111563520 A CN111563520 A CN 111563520A
Authority
CN
China
Prior art keywords
attention
space
spectral
spatial
score
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010044989.1A
Other languages
Chinese (zh)
Other versions
CN111563520B (en
Inventor
尹继豪
李磊
刘雨晨
黄浦
王麒雄
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beihang University
Original Assignee
Beihang University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beihang University filed Critical Beihang University
Priority to CN202010044989.1A priority Critical patent/CN111563520B/en
Publication of CN111563520A publication Critical patent/CN111563520A/en
Application granted granted Critical
Publication of CN111563520B publication Critical patent/CN111563520B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks

Abstract

The algorithm is used for solving the problem that the performance of a traditional convolutional neural network on a fine-grained image classification task represented by a hyperspectral image is insufficient, and the hyperspectral image classification algorithm based on a space-spectrum combined attention mechanism is provided, can effectively capture image global features by matching with the convolutional neural network, and adaptively focuses on spatial local features with large differences among similar images; meanwhile, contributions of different wave bands to the task are evaluated, so that the neural network pays more attention to the spectrum wave bands with large contributions, local difference characteristics of image spectrums are extracted, the hyperspectral image classification precision is improved, and the method has wide application in the field of classification of fine-grained images represented by hyperspectral images.

Description

Hyperspectral image classification method based on space-spectrum combined attention mechanism
Technical Field
The invention relates to a hyperspectral image classification method based on a space-spectrum combined attention mechanism. The method can be used in the field of remote sensing image processing.
Background
The hyperspectral remote sensing technology is one of the most important technical breakthroughs in the field of airborne observation systems and satellite-borne observation systems for human beings at the end of the twentieth century, the hyperspectral image overcomes the limitations of the traditional single-waveband and multispectral remote sensing in the aspects of waveband range, waveband quantity, fine ground target observation and identification and the like, and has unique advantages in the field of remote sensing ground observation. The hyperspectral image classification is an important and meaningful task in practice, and specifically, the hyperspectral image classification is a task of identifying a given image according to different spectral features or spatial features and marking the type of each pixel point in the image.
Compared with the common image classification task, the hyperspectral image has the characteristic of dimension disaster and same-spectrum foreign matter in the spectrum domain, so that the classification task becomes more difficult. Under the circumstance, the traditional hyperspectral image classification algorithm which solely depends on spectral information has limited performance, and the classification algorithm based on the combined space-spectral information is a research hotspot in recent years.
Since 2012, the deep learning technique, represented by Convolutional Neural Network (CNN), was a great achievement in computer vision tasks. The convolutional neural network is very suitable for processing image space domain information and has achieved great success in common image classification tasks, and the convolutional neural network is used for hyperspectral image classification tasks at the earliest in 2016. Subsequently, various convolutional neural network algorithms for the hyperspectral image classification task are developed, but the algorithms have difficulty in extracting image global features due to the limited size of the convolutional network 'receptive field'. What is worse, due to the particularity of the hyperspectral image data, the hyperspectral image data needs to be preprocessed before being classified, that is, each pixel is divided into cubes (the general size is 27 × 27) as the center, and the middle pixel label is used as each cube classification label, so that similar and heterogeneous pixel cubes are very similar in spatial features, which is generally called as integral spatial feature redundancy, and an image with slight difference in local features is a fine-grained image. The ability of the traditional convolutional neural network to process the fine-grained image with the spatial redundancy characteristic is very weak, and the performance of the convolutional neural network in the task of classifying the high-spectrum and other fine-grained images is further improved seriously.
In addition, different from a common image, a hyperspectral image has very rich spectrum information, most of traditional classification algorithms consider that different spectrum bands contribute the same to an algorithm task, but actually, due to the influence of physical factors such as illumination, atmosphere and the like, some bands tend to be noisy, and basically do not contribute to the current task, or even cause interference.
Based on the method, a spatial local feature which can effectively capture the global feature of the image and has larger difference between similar fine degree images is designed; meanwhile, the contributions of different wave bands to the task are evaluated, so that the neural network pays more attention to the spectrum wave bands with large contributions, the local difference characteristics of the image spectrum are extracted, the high-spectrum image classification precision is improved, and the method is a very worthy of research.
Disclosure of Invention
The algorithm aims at solving the problem that the classification performance of the traditional convolutional neural network on the fine and smooth images represented by the hyperspectral images is insufficient, and provides a hyperspectral image classification method based on a space-spectrum combined attention mechanism, which can be matched with the convolutional neural network to effectively capture the global features of the images and adaptively focus the spatial local features with larger differences among similar images; meanwhile, different wave bands are evaluated to contribute to tasks, so that the neural network pays more attention to the spectrum wave bands contributing to large, local difference features of the images are extracted, the classification precision of the hyperspectral images is improved, and the method has wide application in the field of classification of fine images such as hyperspectrum.
The algorithm of the invention provides a space-spectrum combined attention mechanism module, which has the following three advantages:
(1) the algorithm has strong portability and can be randomly embedded into various conventional convolutional neural networks.
(2) The algorithm has good universality, and attention mechanism modules are flexibly selected according to task requirements. For example, when the ordinary fine image classification task without spectral features is faced, the spatial attention machine modeling module is flexibly selected.
(3) The performance of the algorithm is strong, and the performance of the convolutional neural network can be effectively improved;
drawings
FIG. 1 is a block diagram of a spatial-spectral combined attention mechanism;
FIG. 2 is a block diagram of three structures of a convolutional neural network embedded with a spatial-spectral combined attention mechanism module;
FIG. 3 is a comparison of experimental results of different algorithms on a hyperspectral dataset. Note: in the experiment, a space-spectrum combined Attention mechanism Module is called Joint Spatial-Spectral Attention Module for short and JSAM for short, a convolutional neural network adopting a series embedding mode is called CNN-JSAM-A, a convolutional neural network adopting a parallel embedding mode is called CNN-JSAM-B, and a convolutional neural network adopting a series embedding mode is called CNN-JSAM-C. Taking Indian Pine data as a high-spectrum data set in an experiment, taking 10% as a training set, and keeping the network parameters and the layer number of the CNNs consistent, wherein the difference is whether a space-spectrum combined attention mechanism module JSAM is embedded.
Detailed Description
As shown in fig. 1, the spatial-spectral combined attention mechanism module is mainly composed of three sub-modules: a spatial attention score extraction sub-module, a spectral attention score extraction sub-module, and an attention score assignment sub-module. The spatial attention score extraction submodule mainly extracts similarity characteristics between any two pixels in a space to obtain a spatial attention score map; the spectral attention fraction extraction submodule mainly extracts correlation dependencies in different spectral bands to obtain an attention fraction graph of the spectral bands; and the attention score distribution branch distributes the spatial attention scores and the spectral attention scores which are respectively extracted into the original feature space to obtain an attention score cube containing attention features of different spatial domains and different wave bands.
(1) Spatial attention score extraction submodule
The hyperspectral cube of the input network is denoted by X as follows:
Figure BDA0002369032810000031
wherein H is the length of the input hyperspectral cube;
w is the width of the input hyperspectral cube;
c is the spectral dimension of the input hyperspectral cube;
and N ═ hxw;
the method comprises the following steps: respectively mapping an input image X according to a formula (1) into an embedded spectral feature space to obtain two new feature maps theta (X) and phi (X);
Figure BDA0002369032810000032
wherein i and j are the numbers of pixels in the feature map;
Figure BDA0002369032810000033
and
Figure BDA0002369032810000034
linear mapping matrixes are adopted, and the linear mapping matrixes are parameters which can be learned in the neural network;
d is the spectral dimension mapped to the new feature maps θ (X) and φ (X) in the embedded spectral space;
step two: calculating the similarity s of any two pixels by using a Gaussian function embedded in spaceijObtaining a spatial attention point map S, and specifically calculating a process map formula (2) and shown in FIG. 1:
Figure BDA0002369032810000041
Figure BDA0002369032810000042
wherein s isijRepresenting the similarity between the ith and jth pixels;
in the procedure, WθAnd WφIs capable of learningThe network parameters of (1) are realized by adopting a 1-by-1 convolution layer; first in formula (2)θ(xi)Transposing to obtain theta (x)i)TThen, theta (x) is addedi)TPhi (x)j) And performing matrix multiplication operation, and finally performing normalization operation by using a neural network softmax layer.
(2) Spectral attention score extraction submodule
The hyperspectral cube of the input network is denoted by X below
Figure BDA0002369032810000043
Wherein H is the length of the input hyperspectral cube;
w is the width of the input hyperspectral cube;
c is the spectral dimension of the input hyperspectral cube;
the method comprises the following steps: respectively mapping an input image X according to a formula (4) into an embedding space feature space to obtain two new feature maps upsilon (X) and omega (X);
Figure BDA0002369032810000044
wherein i and j are numbers of spectral bands corresponding to the characteristic diagram;
Wυand WωLinear mapping matrixes are adopted, and the linear mapping matrixes are parameters which can be learned in the neural network;
step two: calculating the similarity q of the corresponding characteristic graphs of any two spectral bands by using a Gaussian function embedded in spaceijObtaining a spatial attention point map Q, and specifically calculating a process map formula (5) and shown in FIG. 1:
Figure BDA0002369032810000051
Figure BDA0002369032810000052
wherein q isijRepresenting the similarity between corresponding signatures in the ith and jth spectral bands;
in the procedure, WυAnd WωThe method is a learnable network parameter and is realized by adopting a 3X 3Depth-wise convolution layer; in the formula (5), v (x) is first measuredi) Transposing the resulting product to give v (x)i)TThen v (x)i)TAnd ω (x)i) And performing matrix multiplication operation, and finally performing normalization operation by using a neural network softmax layer.
(3) Attention score assignment submodule
The attention score distribution submodule has the main function of distributing the spatial attention scores and the spectral attention scores extracted respectively to the original feature space by the attention score distribution branch to obtain the attention score cube containing the attention features of different spatial domains and different wave bands.
The input image X is represented as follows:
Figure BDA0002369032810000053
the method comprises the following steps: in order to ensure that the attention mechanism module can adaptively focus on the local space and the local spectral band of the feature map according to task requirements, firstly mapping is carried out in the feature space to obtain a brand new feature map
Figure BDA0002369032810000054
As in equation (7); in the program, formula (7) is implemented by using 3 × 3 convolutional layers, where W isζIs the 3 x 3 convolution kernel parameter.
Figure BDA0002369032810000055
A＝S·ζ(X)·Q (8)
Step two: the attention mechanism score cube a is obtained by assigning the spatial attention score S and the spectral attention score Q to the original feature space by formula (8).
In addition, the algorithm also designs a set of spatial-spectral combined attention mechanism module and convolution neural network embedding modes, which mainly comprise the following three embedding modes:
(1) series embedded mode
(2) Parallel embedded mode
(3) Series-parallel connection embedding mode
Detailed diagrams of three structures of the convolutional neural network embedded with the spatial-spectral combined attention mechanism module are shown in FIG. 2.

Claims (1)

1. The invention provides a hyperspectral image classification algorithm based on a space-spectrum combined attention mechanism, which mainly comprises a space-spectrum combined attention mechanism module and a convolution neural network embedded mode:
1) the space-spectrum combined attention mechanism module consists of three sub-modules, namely a space attention score extraction sub-module, a spectrum attention score extraction sub-module and an attention score distribution sub-module; the spatial attention score extraction branch mainly extracts similarity characteristics between any two pixels in a space to obtain a spatial attention score map; the spectral attention fraction extraction branch is mainly used for extracting relevant dependencies in different spectral bands to obtain an attention fraction graph of the spectral bands; distributing the spatial attention score map and the spectral attention score map respectively extracted by the attention score distribution submodule into the original feature space pixel by pixel and spectrum by spectrum to obtain an attention score cube containing different pixel points and attention features of different wave bands; the method comprises the following specific steps:
(a) spatial attention score extraction submodule
The method comprises the following steps: mapping the input image X into an embedded spectral feature space respectively to obtain two new feature maps theta (X) and phi (X);
step two: calculating the similarity s of any two pixels by using a Gaussian function embedded in spaceijObtaining a spatial attention point chart S, and finally performing normalization operation by using a neural network softmax layer;
(b) spectral attention score extraction submodule
Step three: mapping an input image X into an embedding space feature space respectively to obtain two new feature maps upsilon (X) and omega (X);
step four: calculating the similarity q of the corresponding characteristic graphs of any two spectral bands by using a Gaussian function embedded in spaceijObtaining a spatial attention point diagram Q, wherein in the experiment, the spatial attention point diagram Q is realized by adopting a 3 x 3 layering (Depth-wise) convolution layer; finally, carrying out normalization operation by utilizing a neural network Softmax layer;
(c) attention score assignment submodule
The attention score distribution submodule has the main function that the attention score distribution branch distributes the space attention score and the spectrum attention score which are respectively extracted into the original characteristic space to obtain an attention score cube containing attention characteristics of different space domains and different wave bands;
step five: in order to ensure that the attention mechanism module can adaptively focus on the local space and the local spectral band of the feature map according to task requirements, firstly mapping is carried out in the feature space to obtain a brand new feature map
Figure RE-FDA0002577938500000021
Realized by 3 x 3 convolution layers, in which WzIs the 3 x 3 convolution kernel parameter;
step six: the spatial attention score S and the spectral attention score Q are assigned to the original feature space, and an attention mechanism score cube a is obtained.
2) The spatial-spectral combined attention mechanism module is embedded into the convolutional neural network in three ways:
(a) a serial embedding mode;
(b) a parallel embedding mode;
(c) series-parallel embedded mode.
CN202010044989.1A 2020-01-16 2020-01-16 Hyperspectral image classification method based on space-spectrum combined attention mechanism Active CN111563520B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010044989.1A CN111563520B (en) 2020-01-16 2020-01-16 Hyperspectral image classification method based on space-spectrum combined attention mechanism

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010044989.1A CN111563520B (en) 2020-01-16 2020-01-16 Hyperspectral image classification method based on space-spectrum combined attention mechanism

Publications (2)

Publication Number Publication Date
CN111563520A true CN111563520A (en) 2020-08-21
CN111563520B CN111563520B (en) 2023-01-13

Family

ID=72071383

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010044989.1A Active CN111563520B (en) 2020-01-16 2020-01-16 Hyperspectral image classification method based on space-spectrum combined attention mechanism

Country Status (1)

Country Link
CN (1) CN111563520B (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112052755A (en) * 2020-08-24 2020-12-08 西安电子科技大学 Semantic convolution hyperspectral image classification method based on multi-path attention mechanism
CN112232343A (en) * 2020-09-03 2021-01-15 国家粮食和物资储备局科学研究院 Neural network and method for recognizing grain mildewed grains
CN112287989A (en) * 2020-10-20 2021-01-29 武汉大学 Aerial image ground object classification method based on self-attention mechanism
CN113537239A (en) * 2021-07-08 2021-10-22 宁波大学 Hyperspectral image band selection method based on global relationship perception attention
CN114462596A (en) * 2022-02-10 2022-05-10 黑龙江省农业科学院 Disease and insect pest monitoring method and system for industrial hemp growth period

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120108997A1 (en) * 2008-12-19 2012-05-03 Cuntai Guan Device and method for generating a representation of a subject's attention level
CN109376804A (en) * 2018-12-19 2019-02-22 中国地质大学（武汉） Based on attention mechanism and convolutional neural networks Classification of hyperspectral remote sensing image method
CN109993220A (en) * 2019-03-23 2019-07-09 西安电子科技大学 Multi-source Remote Sensing Images Classification method based on two-way attention fused neural network
CN110458192A (en) * 2019-07-05 2019-11-15 中国地质大学（武汉） The Classification of hyperspectral remote sensing image method and system of view-based access control model conspicuousness
CN110516596A (en) * 2019-08-27 2019-11-29 西安电子科技大学 Empty spectrum attention hyperspectral image classification method based on Octave convolution

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120108997A1 (en) * 2008-12-19 2012-05-03 Cuntai Guan Device and method for generating a representation of a subject's attention level
CN109376804A (en) * 2018-12-19 2019-02-22 中国地质大学（武汉） Based on attention mechanism and convolutional neural networks Classification of hyperspectral remote sensing image method
CN109993220A (en) * 2019-03-23 2019-07-09 西安电子科技大学 Multi-source Remote Sensing Images Classification method based on two-way attention fused neural network
CN110458192A (en) * 2019-07-05 2019-11-15 中国地质大学（武汉） The Classification of hyperspectral remote sensing image method and system of view-based access control model conspicuousness
CN110516596A (en) * 2019-08-27 2019-11-29 西安电子科技大学 Empty spectrum attention hyperspectral image classification method based on Octave convolution

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
XIAOGUANG MEI 等: "Spectral-Spatial Attention Networks for Hyperspectral Image Classification", 《REMOTE SENSING》 *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112052755A (en) * 2020-08-24 2020-12-08 西安电子科技大学 Semantic convolution hyperspectral image classification method based on multi-path attention mechanism
CN112232343A (en) * 2020-09-03 2021-01-15 国家粮食和物资储备局科学研究院 Neural network and method for recognizing grain mildewed grains
CN112232343B (en) * 2020-09-03 2023-11-21 国家粮食和物资储备局科学研究院 Grain mildew grain identification neural network and identification method
CN112287989A (en) * 2020-10-20 2021-01-29 武汉大学 Aerial image ground object classification method based on self-attention mechanism
CN112287989B (en) * 2020-10-20 2022-06-07 武汉大学 Aerial image ground object classification method based on self-attention mechanism
CN113537239A (en) * 2021-07-08 2021-10-22 宁波大学 Hyperspectral image band selection method based on global relationship perception attention
CN114462596A (en) * 2022-02-10 2022-05-10 黑龙江省农业科学院 Disease and insect pest monitoring method and system for industrial hemp growth period

Also Published As

Publication number Publication date
CN111563520B (en) 2023-01-13

Similar Documents

Publication Publication Date Title
CN111563520B (en) Hyperspectral image classification method based on space-spectrum combined attention mechanism
Chen et al. Land-use scene classification using multi-scale completed local binary patterns
CN108052966B (en) Remote sensing image scene automatic extraction and classification method based on convolutional neural network
Lin et al. Multispectral change detection with bilinear convolutional neural networks
CN111160273B (en) Hyperspectral image spatial spectrum joint classification method and device
Zhu et al. Plant identification based on very deep convolutional neural networks
US8295604B2 (en) Image search method and device using affine-invariant regions
CN111310773A (en) Efficient license plate positioning method of convolutional neural network
CN109961096B (en) Multimode hyperspectral image migration classification method
Ortiz et al. Local context normalization: Revisiting local normalization
CN113705641B (en) Hyperspectral image classification method based on rich context network
Chen et al. Object-based multi-modal convolution neural networks for building extraction using panchromatic and multispectral imagery
Singhal et al. Directional local ternary co-occurrence pattern for natural image retrieval
Shi et al. F 3 Net: Fast Fourier filter network for hyperspectral image classification
Agilandeeswari et al. SWIN transformer based contrastive self-supervised learning for animal detection and classification
Rahiche et al. Blind decomposition of multispectral document images using orthogonal nonnegative matrix factorization
Mittal et al. On the performance evaluation of object classification models in low altitude aerial data
Barroso-Laguna et al. Scalenet: A shallow architecture for scale estimation
CN116486238B (en) Target fine granularity identification method combining point set representation and graph classification
CN106971402B (en) SAR image change detection method based on optical assistance
Hamam et al. Single-band infrared texture-based image colorization
CN114037922B (en) Aerial image segmentation method based on hierarchical context network
Sheoran et al. Analysis on application of swarm-based techniques in processing remote sensed data
Jodas et al. Deep Learning Semantic Segmentation Models for Detecting the Tree Crown Foliage.
CN113850316A (en) Hyperspectral image classification method and device of combined spectrum space multilayer perceptron

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant