CN115272776B - Hyperspectral image classification method based on double-path convolution and double attention and storage medium - Google Patents
Hyperspectral image classification method based on double-path convolution and double attention and storage medium Download PDFInfo
- Publication number
- CN115272776B CN115272776B CN202211169177.5A CN202211169177A CN115272776B CN 115272776 B CN115272776 B CN 115272776B CN 202211169177 A CN202211169177 A CN 202211169177A CN 115272776 B CN115272776 B CN 115272776B
- Authority
- CN
- China
- Prior art keywords
- attention
- convolution
- channel
- space
- way
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 33
- 238000003860 storage Methods 0.000 title claims abstract description 8
- 238000000605 extraction Methods 0.000 claims abstract description 40
- 238000012549 training Methods 0.000 claims abstract description 40
- 238000013507 mapping Methods 0.000 claims abstract description 37
- 230000004927 fusion Effects 0.000 claims abstract description 9
- 238000005070 sampling Methods 0.000 claims abstract description 8
- 238000005520 cutting process Methods 0.000 claims abstract description 4
- 238000007781 pre-processing Methods 0.000 claims abstract description 4
- 230000006870 function Effects 0.000 claims description 54
- 230000004913 activation Effects 0.000 claims description 38
- 238000010606 normalization Methods 0.000 claims description 27
- 238000011176 pooling Methods 0.000 claims description 11
- 238000012360 testing method Methods 0.000 claims description 11
- 238000012545 processing Methods 0.000 claims description 9
- 238000012795 verification Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 3
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 3
- 230000003247 decreasing effect Effects 0.000 claims description 2
- 238000005457 optimization Methods 0.000 claims description 2
- 238000013145 classification model Methods 0.000 abstract description 6
- 230000009286 beneficial effect Effects 0.000 description 7
- 230000003044 adaptive effect Effects 0.000 description 6
- 238000001514 detection method Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 230000003595 spectral effect Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000005304 joining Methods 0.000 description 2
- 238000005065 mining Methods 0.000 description 2
- 238000010200 validation analysis Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 235000008331 Pinus X rigitaeda Nutrition 0.000 description 1
- 235000011613 Pinus brutia Nutrition 0.000 description 1
- 241000018646 Pinus brutia Species 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000000630 rising effect Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 239000004575 stone Substances 0.000 description 1
- 230000001629 suppression Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
- G06V10/765—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects using rules for classification or partitioning the feature space
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/194—Terrestrial scenes using hyperspectral data, i.e. more or other wavelengths than RGB
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Multimedia (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Data Mining & Analysis (AREA)
- Remote Sensing (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a hyperspectral image classification method and a storage medium based on double-path convolution and double attention, and belongs to the technical field of remote sensing images. The method comprises the following steps of S1, preprocessing an image; s2, cutting a sampling image block, and dividing a data set; s3, respectively sending the image blocks into the constructed space attention double-path convolution module and the constructed channel attention double-path convolution module, and respectively extracting the surface spectrum-space characteristics; s4, performing double-branch feature fusion on the spectrum-space features extracted by the two modules, and inputting the spectrum-space features into a basic double-path convolution network block to further extract the spectrum-space features; s5, sending the extracted spectrum-space characteristic mapping into a classifier for pixel classification, and calculating a loss value; and S6, iteratively training and optimizing the model, and obtaining the final hyperspectral image classification mapping by using the final model. The method can realize the extraction of discriminant and fine characteristics, and improve the classification performance and the generalization capability of the classification model.
Description
Technical Field
The invention relates to a hyperspectral image classification method, in particular to a hyperspectral image classification method and a storage medium based on two-way convolution and two-way attention, and belongs to the technical field of convolution neural networks, attention mechanisms and remote sensing images.
Background
The hyperspectral image is usually captured by an image spectrometer carried on an aviation platform, and abundant spectral information and spatial ground object information are recorded. Therefore, the hyperspectral image has very wide application in a plurality of fields such as mining exploration, ecological engineering, precision agriculture and urban planning. As a basic task in the field of hyperspectral image application, hyperspectral image classification is a base stone for a plurality of hyperspectral downstream applications, such as hyperspectral image target detection, hyperspectral image anomaly detection and hyperspectral image change detection, and the like, and needs to be based on the hyperspectral image target detection, the hyperspectral image anomaly detection and the hyperspectral image change detection. The hyperspectral image classification task aims at assigning a unique ground object semantic label to each pixel of the hyperspectral image.
According to the traditional hyperspectral image classification method based on the classical machine learning method, due to the fact that the prior knowledge and the manually set hyper-parameters are compared, ideal ground object assignment results are difficult to obtain in the aspects of discriminative classification and generalization classification of hyperspectral ground object scenes. In recent years, due to the strong spectrum-space feature extraction capability, a more ideal result is obtained in the hyperspectral image classification task, and the development and application of the hyperspectral image classification task are greatly promoted.
Currently, classification methods based on convolutional neural networks are receiving extensive attention in hyperspectral image classification tasks and demonstrate their excellent performance with their local perception and parameter sharing characteristics. For the hyperspectral image classification task, learning of more discriminative spectrum-space characteristics is the key for realizing more superior hyperspectral image classification results. In a hyperspectral image classification model based on image blocks widely used at present, the image blocks represent central pixels of the image blocks to complete extraction and final classification of spectral-spatial features, and ground object class labels of the image blocks and the ground object class labels are kept consistent. Based on the assumption that the adjacent pixels have a high probability of belonging to the unified ground object class, the neighborhood pixels of the central pixel play an assisting role in the classification process. However, it should be noted that the feature class labels of the neighborhood pixels relative to the center pixel may be different, and the inconsistency is more pronounced at feature class boundaries. Therefore, the contribution of the self-adaptive determination neighborhood pixels in the classification process can play a positive role in the ground feature discrimination. On the other hand, in the process of feature learning of the convolution model, different feature channels generate different contributions to the final discriminative power of the model, so that the adaptive enhancement and suppression of channel features is also the key for extracting more discriminative feature representation and realizing superior classification performance.
Attention is drawn to many artificial intelligence tasks because adaptive relationship mining can be implemented to facilitate discriminant feature learning. Attention mechanism is often embedded into a model and is trained with the model in a coordinated mode, feature units beneficial to a model task are emphasized through soft weight calculation, and feature units interfering with the model task are restrained to extract more robust features in a refined mode. At present, aiming at a hyperspectral image classification task, how to reasonably and efficiently utilize an attention mechanism to fully mine the characteristics of a spectrum domain and a space domain to enhance the ground object assignment performance of a model still remains to be solved.
Disclosure of Invention
The invention aims to overcome the defects and provide a hyperspectral image classification method based on two-way convolution and two-way attention.
The technical scheme adopted by the invention is as follows:
the hyperspectral image classification method based on two-way convolution and two-way attention comprises the following steps:
s1, carrying out standardization preprocessing on a loaded original image;
s2, cutting a sampling image block of the preprocessed image, and dividing a sampling data set into a training set, a verification set and a test set;
s3, respectively sending the same image block into a constructed space attention double-path convolution module and a constructed channel attention double-path convolution module, and respectively extracting the spectrum-space characteristics facing the self-adaptive space information and the self-adaptive channel information; the step of extracting the spectrum-space characteristics by the spatial attention double-path convolution module comprises the steps of performing primary characteristic extraction on an image block by using two paths of convolution networks arranged in parallel, merging the two paths of primary characteristics, performing channel-by-channel batch normalization and nonlinear activation function processing on the merged spectrum-space characteristics, performing spatial attention mapping extraction after the final batch normalization processing of the module, and then performing nonlinear activation function processing; the step of spectrum-space feature extraction of the channel attention double-path convolution module comprises the steps of performing primary feature extraction on an image block by using two paths of convolution networks which are arranged in parallel, performing channel attention mapping extraction after one path of primary feature extraction, then combining the extracted mapping feature with the other path of primary feature, and performing channel-by-channel batch normalization and nonlinear activation function processing on the combined spectrum-space feature;
s4, performing double-branch feature fusion on the spectrum-space features extracted by the two paths of modules, and inputting the spectrum-space features into a basic two-path convolution network block to further refine the spectrum-space features;
s5, the spectrum-space characteristic mapping obtained by thinning is sent to a classifier for pixel classification, and a loss value is calculated according to the generated label classification probability value;
and S6, iteratively training and optimizing the model, and obtaining the final hyperspectral image classification mapping by using the final model.
In the method, the two parallel convolution networks in the spatial attention two-way convolution module in the step S3 are realized by a 1 × 1 convolution layer, the output channel of the 1 × 1 convolution layer is configured with batch normalization operation and a ReLU nonlinear activation function, the output channel of the 3 × 3 convolution layer is configured with batch normalization operation and a ReLU nonlinear activation function; and S3, combining the two primary characteristics in the spatial attention two-way convolution module by using element-by-element addition. The spatial attention mapping extraction is realized through a spatial attention module, firstly, input spectrum-spatial feature maps are subjected to feature abstraction along channel dimensions by respectively using maximum pooling and mean pooling to respectively obtain a 2D spatial feature descriptor, the obtained two spatial feature descriptors are subjected to channel dimension splicing, then are sent to a 7 x 7 convolution layer for attention mapping learning, and are activated by joining a Sigmoid nonlinear function to obtain spatial attention mapping (soft weight spatial attention mapping capable of reflecting the importance degree of pixel information in an image block). And S3, the step of extracting the spectrum-space characteristics by the space attention double-path convolution module further comprises the step of multiplying the space attention mapping processed by the nonlinear activation function and the spectrum-space characteristic diagram originally input by the space attention double-path convolution module element by element in the space dimension so as to emphasize the neighborhood characteristics beneficial to characteristic extraction and inhibit the neighborhood characteristics with interference on the characteristic extraction.
S3, one path of the two paths of convolution networks arranged in parallel in the channel attention two-path convolution module is realized by a 1 x 1 convolution layer, the output channel of the 1 x 1 convolution layer is configured with batch normalization operation and a ReLU nonlinear activation function, the other path is realized by a 3 x 3 convolution layer, and the output channel of the 3 x 3 convolution layer is configured with batch normalization operation and a ReLU nonlinear activation function and then connected with channel attention mapping extraction; and S3, combining the mapping characteristics and the preliminary characteristics in the channel attention double-path convolution module by using element-by-element addition. Step S3, channel attention mapping extraction is achieved through a channel attention module, firstly, mean pooling is used for carrying out feature abstraction on an input spectrum-space feature graph along the space dimension to obtain a 1D channel feature descriptor, then a small double-layer fully-connected group is connected to achieve dimension increasing and dimension decreasing extraction of channel features, a nonlinear activation function embedded in the double-layer fully-connected group is a ReLU function, and then a Sigmoid nonlinear function is used for further nonlinear activation to obtain channel attention mapping (soft weight channel attention mapping capable of reflecting different channel importance degrees in the feature mapping). And S3, the step of extracting the spectrum-space characteristics by the channel attention two-way convolution module further comprises the step of multiplying the combined spectrum-space characteristics processed by the channel-by-channel batch normalization and the nonlinear activation function with the spectrum-space characteristic diagram originally input by the channel attention two-way convolution module element by element in the channel dimension so as to reinforce the self-adaptive channel characteristics.
And step 4, the dual-branch feature fusion is carried out by element-by-element addition, and the batch normalization processing is carried out channel-by-channel after the dual-branch feature fusion, so that the offset in the batch is eliminated, and the stability of feature extraction is improved. The basic two-way convolutional network block described in step S4 is composed of 1 × 1 and 3 × 3 convolutional layer branches, element-by-element addition operation, batch normalization operation, and ReLU activation function.
And the classifier in the step S5 uses a classic three-layer classifier to carry out final hyperspectral image pixel classification and comprises a mean pooling layer, a flattening layer and a full-link layer. The final classifier uses softmax activation function prediction to generate label classification probability, and uses cross entropy loss function to calculate loss value, wherein the cross entropy loss function is expressed as:
wherein,
the calculated loss value is represented by the value,Nrepresenting the number of samples of the single batch training set in which the model employs the small batch training mode, here a value of 32,Krepresenting the number of categories in the data scene,nandkseparately indexing the first of the current batch of training setsnSample and class labelsetkThe number of the categories is one,y n represents the first in the training set of the current batchnA true value of a number of samples of the hyperspectral image block,
represents an indication function wheny n Is composed ofkWhen the temperature of the water is higher than the set temperature,
is 1; if not, then the mobile terminal can be switched to the normal mode,
is 0, and in addition to this,
represents the considered secondnA hyperspectral image block sample belongs tokThe softmax function of the category outputs a probability value.
The iterative training and optimization model in the step S6 uses an Adam optimizer, the learning rate is set to be 0.001, the size of a single-batch training set in a small-batch training mode is 32, in model training, loss values corresponding to the training set and the verification set are calculated after each iteration is completed, and the model after 100 iterations is used as a final model. (iterative training is based on a training set, one iteration of the model is the process of inputting all training set samples and performing one-pass on the proposed model; and the Adam optimizer overall parameter setting is used for the stages of training, verifying and testing).
It is a further object of the present invention to provide a computer readable storage medium, on which a computer program is stored, which program, when being executed by a processor, carries out the steps of the hyperspectral image classification method based on two-way convolution and two-way attention as described above.
The beneficial effects of the invention are as follows:
(1) Convolution units with two scales of 1 × 1 and 3 × 3 convolution are used as basic branches to form a parallel feature extraction unit and are properly embedded into an attention module, and spatial attention and channel attention are embedded into a basic double-path convolution block to obtain more robust spectrum-space features in a spatial domain and a channel domain, so that more precise and more discriminant spectrum-space feature extraction in the spatial dimension and the channel dimension is realized;
(2) The two attention double-path convolution modules are integrally arranged in parallel, namely, neighborhood features beneficial to feature extraction can be emphasized, neighborhood features with interference on feature extraction can be restrained, and adaptive channel features can be strengthened, so that beneficial spectrum-space features in a space domain and a channel domain can be adaptively mined; the two adaptive attention modules can be seamlessly embedded into the proposed hyperspectral image classification model and trained along with a forward-backward propagation algorithm, and finally the feature distinguishing capability of the proposed classification model is enhanced;
(3) According to the method, two attention double-path convolution modules which are arranged in parallel, feature fusion and a further double-path convolution module are arranged to realize a superior hyperspectral image classification result in a cooperative mode, so that the model performance and the generalization capability on a hyperspectral image classification task are improved, and finally a foundation is laid for the superior ground feature distinguishing performance of the proposed model in multiple scenes.
Drawings
FIG. 1 is a schematic diagram of a model of the process of the present invention;
FIG. 2 is a flow chart of the method of the present invention;
FIG. 3 is a block diagram of a spatial attention module according to the present invention;
FIG. 4 is a block diagram of a channel attention module according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to specific embodiments and the accompanying drawings.
Example 1
A hyperspectral image classification method based on two-way convolution and two-way attention comprises the following steps:
s1, image preprocessing
And respectively carrying out mean-variance standardization treatment on each spectral dimension of all pixel samples in the loaded original image so as to accelerate the convergence speed of the proposed classification model in the training process.
S2, image cutting blocking and data set splitting
Firstly, the image block of the preprocessed image is cut, and finally the image block is used as an input unit of the proposed classification model to finish the judgment of the ground feature type. Specifically, the boundary of the image is first filled with 0, and then random sampling is performed on each category in each data scene according to a set proportion to obtain a training sample, a verification sample and a test sample. For example, for the Indian Pines dataset and the Kennedy Space Center dataset, the training sample and validation sample account for 10% and 1%, respectively, with the remainder used as the sample test. For the Pavia University dataset, the training and validation samples account for 5% and 0.5%, respectively, with the remainder used as sample tests. When the number of certain sample classes is too small to meet the sampling requirement of the verification set, the lowest sampling number is set to ensure that each class is uniformly sampled approximately according to the proportion of the number of class samples. Specifically, when each pixel sample is sampled, an image block with the size of 9 × 9 × b is cut by taking the pixel as a center, wherein 9 × 9 represents the size of a spatial window of the image block, and b is the original spectral dimension of the image. And finally, respectively aggregating the training samples, the verification samples and the test samples of each category into a training set, a verification set and a test set.
S3, feature extraction is carried out on the space attention double-path convolution module and the channel attention double-path convolution module at the same time
As shown in fig. 1, in the present invention, an input spectrum-space image block is first sent to a space attention two-way convolution module and a channel attention two-way convolution module respectively to perform spectrum-space feature extraction oriented to adaptive space information and adaptive channel information. The proposed space attention two-way convolution module and channel attention two-way convolution module are based on the proposed two-way convolution block, the two-way convolution block is realized by two parallel convolution networks, namely a 1 x 1 convolution layer, a batch normalization operation and a ReLU nonlinear activation function are configured for an output channel of the 1 x 1 convolution layer, a 3 x 3 convolution layer is realized for one output channel of the 3 x 3 convolution layer, and the batch normalization operation and the ReLU nonlinear activation function are configured for the output channel of the 3 x 3 convolution layer. The merging of the two-way features is then done using element-by-element addition. The combined spectral-spatial features are further processed by batch normalization and ReLU nonlinear activation functions channel by channel. For the spatial attention two-way convolution module, spatial attention is used to be embedded between the last batch normalization of the module and the ReLU nonlinear activation function, wherein the spatial attention module used is the spatial attention module in the classical CBAM attention module, as shown in fig. 3. For the channel attention two-way convolution module, the channel attention is embedded between the 3 × 3 convolution layer branch and the element-by-element addition, wherein the adopted channel attention module is the classic SE attention module as shown in fig. 4.
For the spatial attention module, firstly, performing feature abstraction on a spectrum-spatial feature map input by the module by respectively using maximum pooling and mean pooling along channel dimensions to respectively obtain a 2D spatial feature descriptor, splicing the two obtained spatial descriptors in the channel dimensions, sending the two spatial descriptors into a 7 × 7 convolution layer for attention mapping learning, and joining a Sigmoid nonlinear function for activation to obtain spatial attention mapping (soft weight spatial attention mapping capable of reflecting the importance degree of pixel information in an image block). Finally, the spatial attention mapping processed by the nonlinear activation function and the spectrum-spatial feature graph originally input by the spatial attention module are subjected to element-by-element multiplication in spatial dimension to emphasize neighborhood features beneficial to feature extraction and inhibit the neighborhood features having interference on feature extraction. For the channel attention module, firstly, the spectral-spatial feature map input by the module is subjected to feature abstraction along the spatial dimension by using mean pooling to obtain a 1D channel feature descriptor. And then, a small double-layer fully-connected group is connected to realize the rising dimension and the falling dimension extraction of the channel characteristics, wherein the embedded nonlinear activation function is still the ReLU function used before. The channel attention map (a soft weighted channel attention map that may reflect different channel importance levels in the feature map) is then derived using a Sigmoid nonlinear function for further nonlinear activation. Finally, the spectrum-space characteristics after being combined and processed by the channel-by-channel batch normalization and the ReLU nonlinear activation function are multiplied element by element in the channel dimension by the spectrum-space characteristic diagram originally input by the channel attention module, so that the self-adaptive channel characteristics are strengthened.
Spatial attention and channel attention are embedded into the basic two-way convolution module, so that spectrum-spatial feature extraction with more fineness and more discriminativity in spatial dimension and channel dimension is realized, and finally a foundation is laid for the superior ground feature discrimination performance of the proposed model in multiple scenes. In addition to this, the two attention two-way convolution modules are arranged in parallel as a whole, so that the beneficial spectral-spatial features in the spatial domain and the channel domain can be adaptively mined.
S4, fusing double-branch characteristics, and further extracting characteristics by using basic two-way convolution network blocks
As shown in fig. 1, spectral-spatial feature mapping after refinement and extraction is performed by the spatial attention two-way convolution module and the channel attention two-way convolution module, dual-branch feature fusion is performed by element-by-element addition, and then batch normalization is performed channel-by-channel to eliminate offset in batches, thereby increasing the stability of feature extraction. And then, sending a basic two-way convolution network block to further extract the spectral-spatial characteristics, wherein the basic two-way convolution network block consists of 1 x 1 and 3 x 3 convolution layer branches, element-by-element addition operation, batch normalization operation and a ReLU activation function. The basic two-way convolutional network block further enhances the feature extraction capability of the proposed model.
S5, the obtained spectrum-space characteristic mapping is sent to a classifier for classification, and a loss value is calculated
As shown in figure 1, the invention uses a classic three-layer classifier to perform final hyperspectral image pixel classification, and comprises a mean pooling layer, a flattening layer and a full-link layer. The final classifier uses a softmax activation function to predict and generate label classification probability, and uses a cross entropy loss function to calculate a loss value, wherein the cross entropy loss function is expressed as:
wherein,
the calculated loss value is represented by the value,Nrepresenting the number of samples of the single batch training set in which the model employs the small batch training mode, here a value of 32,Krepresenting the number of categories in the scene of the data,nandkseparately index the first of the training set of the current batchnSample and class labelsetkEach class，y n Represents the first in the current batch training setnA true value of a number of samples of the hyperspectral image block,
represents an indication function wheny n Is composed ofkWhen the temperature of the water is higher than the set temperature,
is 1; if not, then,
is 0, and in addition to this,
represents the first considerednA hyperspectral image block sample belongs tokThe softmax function of the category outputs a probability value.
And S6, iteratively training and optimizing the model, and obtaining the final hyperspectral image classification mapping (namely the classified ground feature scene visual map) by using the final model. And iteratively training the proposed model in a back propagation mode according to the loss value, and updating parameters of the model. The iterative training is based on a training set. The network model provided by the invention uses an Adam optimizer, the learning rate is set to be 0.001, and the size of a single batch training set in a small batch training mode is 32. In model training, calculating loss values corresponding to a training set and a verification set after each iteration is completed, and using a model after 100 iterations as a final model. In the model testing stage, the cut image block testing set is used for performing model testing in different hyperspectral scenes, the classification performance of the model in each hyperspectral scene can be quantitatively measured according to the corresponding truth value label, and the model can obtain the visual image of the whole scene by assigning class labels to each pixel in the scene.
Example 2
The present embodiment provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps in the hyperspectral image classification method based on two-way convolution and two-way attention as described in embodiment 1 above.
The above description is only exemplary of the present invention and should not be taken as limiting the invention, as any modification, equivalent replacement, or improvement made within the spirit and scope of the present invention should be included in the present invention.
Claims (10)
1. The hyperspectral image classification method based on two-way convolution and two-way attention is characterized by comprising the following steps of:
s1, carrying out standardized preprocessing on a loaded original image;
s2, cutting a sampling image block of the preprocessed image, and dividing a sampling data set into a training set, a verification set and a test set;
s3, respectively sending the same image block into a constructed space attention double-path convolution module and a channel attention double-path convolution module, and respectively extracting the spectrum-space characteristics facing the self-adaptive space information and the self-adaptive channel information; the step of extracting the spectrum-space characteristics by the spatial attention double-path convolution module comprises the steps of performing primary characteristic extraction on an image block by using two paths of convolution networks arranged in parallel, merging the two paths of primary characteristics, performing channel-by-channel batch normalization and nonlinear activation function processing on the merged spectrum-space characteristics, performing spatial attention mapping extraction after the final batch normalization processing of the module, and then performing nonlinear activation function processing; the step of spectrum-space feature extraction of the channel attention double-path convolution module comprises the steps of performing primary feature extraction on an image block by using two paths of convolution networks which are arranged in parallel, performing channel attention mapping extraction after one path of primary feature extraction, then combining the extracted mapping feature with the other path of primary feature, and performing channel-by-channel batch normalization and nonlinear activation function processing on the combined spectrum-space feature;
s4, performing double-branch feature fusion on the spectrum-space features extracted by the two paths of modules, and inputting the spectrum-space features into a basic two-path convolution network block to further refine the spectrum-space features;
s5, the spectrum-space characteristic mapping obtained by thinning is sent to a classifier for pixel classification, and a loss value is calculated according to the generated label classification probability value;
and S6, performing iterative training and model optimization, and obtaining the final hyperspectral image classification mapping by using the final model.
2. The hyperspectral image classification method based on two-way convolution and two-way attention according to claim 1 is characterized in that the two-way parallel convolution networks in the spatial attention two-way convolution module in the step S3 are realized by a 1 × 1 convolution layer, an output channel of the 1 × 1 convolution layer is configured with batch normalization operation and a ReLU nonlinear activation function, and one channel is realized by a 3 × 3 convolution layer, and an output channel of the 3 × 3 convolution layer is configured with batch normalization operation and a ReLU nonlinear activation function; and S3, combining the two primary characteristics in the spatial attention two-way convolution module by using element-by-element addition.
3. The hyperspectral image classification method based on two-way convolution and two-way attention according to claim 1 is characterized in that the spatial attention mapping extraction is realized through a spatial attention module, firstly, feature abstraction is respectively performed on an input spectrum-spatial feature map by using maximum pooling and mean pooling along channel dimensions, respectively, a 2D spatial feature descriptor is obtained, the obtained two spatial feature descriptors are spliced through the channel dimensions, and then are sent to a 7 x 7 convolutional layer for attention mapping learning, and a Sigmoid nonlinear function is linked for activation, so that spatial attention mapping is obtained.
4. The hyperspectral image classification method based on two-way convolution and two-way attention according to claim 1 is characterized in that the step of performing spectrum-space feature extraction by the space attention two-way convolution module in the step S3 further comprises the step of performing element-by-element multiplication on a space dimension by using a space attention map processed by a nonlinear activation function and a spectrum-space feature map originally input by the space attention two-way convolution module.
5. The hyperspectral image classification method based on two-way convolution and two-way attention according to claim 1 is characterized in that the two-way convolution network in parallel arrangement in the channel attention two-way convolution module in the step S3 is realized by a 1 x 1 convolution layer, an output channel of the 1 x 1 convolution layer is configured with batch normalization operation and a ReLU nonlinear activation function, one is realized by a 3 x 3 convolution layer, and an output channel of the 3 x 3 convolution layer is configured with batch normalization operation and a ReLU nonlinear activation function and is followed by channel attention mapping extraction; and S3, combining the extracted mapping characteristics and the other primary characteristics in the channel attention two-way convolution module by using element-by-element addition.
6. The hyperspectral image classification method based on two-way convolution and two-way attention according to claim 1 is characterized in that the channel attention mapping extraction in the step S3 is realized through a channel attention module, firstly, the input spectrum-space feature map is subjected to feature abstraction along the space dimension by using mean pooling to obtain a 1D channel feature descriptor, then, a small double-layer fully-connected group is connected to realize the dimension increasing and dimension decreasing extraction of the channel feature, wherein a nonlinear activation function embedded in the double-layer fully-connected group is a ReLU function, and then, a Sigmoid nonlinear function is used for further nonlinear activation to obtain the channel attention mapping.
7. The hyperspectral image classification method based on two-way convolution and two-way attention according to claim 1 is characterized in that the step of extracting the spectrum-space features by the channel attention two-way convolution module in the step S3 further comprises the step of multiplying the combined spectrum-space features processed by batch normalization and nonlinear activation function with the spectrum-space feature map originally input by the channel attention two-way convolution module element by element in the channel dimension.
8. The method for classifying hyperspectral images based on two-way convolution and two-note according to claim 1, wherein the two-branch feature fusion in step S4 is performed by element-by-element addition, and the two-branch feature fusion is followed by channel-by-channel batch normalization, and the basic two-way convolution network block consists of two convolution layer branches of 1 x 1 and 3 x 3, an element-by-element addition operation, a batch normalization operation, and a ReLU activation function.
9. The hyperspectral image classification method based on two-way convolution and two-way attention according to claim 1 is characterized in that the classifier in the step S5 uses a classic three-layer classifier to perform final hyperspectral image pixel classification, the final classifier uses a softmax activation function to predict and generate label classification probability, a cross entropy loss function is used to calculate a loss value, and the cross entropy loss function is expressed as:
wherein,
the calculated loss value is represented by the value,Nthe number of samples representing the single batch training set in which the model used the small batch training mode, here the value of 32,Krepresenting the number of categories in the data scene,nandkseparately index the first of the training set of the current batchnSample and class labelsetkThe number of the categories is one,y n represents the first in the training set of the current batchnA true value of a number of samples of the hyperspectral image block,
represents an indicator function wheny n Is composed ofkWhen the utility model is used, the water is discharged,
is 1; if not, then,
is 0, and in addition to this,
represents the considered secondnSample of hyperspectral image blockkThe softmax function of the category outputs a probability value.
10. A storage medium being a computer readable storage medium having stored thereon a computer program, characterized in that the program, when being executed by a processor, is adapted to carry out the steps of the method for hyperspectral image classification based on two-way convolution and two-note according to any of the claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211169177.5A CN115272776B (en) | 2022-09-26 | 2022-09-26 | Hyperspectral image classification method based on double-path convolution and double attention and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211169177.5A CN115272776B (en) | 2022-09-26 | 2022-09-26 | Hyperspectral image classification method based on double-path convolution and double attention and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115272776A CN115272776A (en) | 2022-11-01 |
| CN115272776B true CN115272776B (en) | 2023-01-20 |
Family
ID=83756415
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211169177.5A Active CN115272776B (en) | 2022-09-26 | 2022-09-26 | Hyperspectral image classification method based on double-path convolution and double attention and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115272776B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116229178B (en) * | 2023-03-14 | 2023-11-24 | 安徽大学 | Image classification method for small quantity of training samples based on Transformer |
| CN117152616B (en) * | 2023-09-12 | 2024-08-20 | 电子科技大学 | Remote sensing image typical object extraction method based on spectrum enhancement and double-path coding |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111832620A (en) * | 2020-06-11 | 2020-10-27 | 桂林电子科技大学 | Image emotion classification method based on double-attention multilayer feature fusion |
| CN113642634A (en) * | 2021-08-12 | 2021-11-12 | 南京邮电大学 | Shadow detection method based on mixed attention |
| WO2021248687A1 (en) * | 2020-06-10 | 2021-12-16 | 南京理工大学 | Driving fatigue detection method and system combining pseudo 3d convolutional neural network and attention mechanism |
| CN114067107A (en) * | 2022-01-13 | 2022-02-18 | 中国海洋大学 | Multi-scale fine-grained image recognition method and system based on multi-grained attention |
| CN114187261A (en) * | 2021-12-07 | 2022-03-15 | 天津大学 | Non-reference stereo image quality evaluation method based on multi-dimensional attention mechanism |
| CN114418003A (en) * | 2022-01-20 | 2022-04-29 | 北京科技大学 | Double-image identification and classification method based on attention mechanism and multi-size information extraction |
| WO2022127227A1 (en) * | 2020-12-15 | 2022-06-23 | 西安交通大学 | Multi-view semi-supervised lymph node classification method and system, and device |
| CN114663309A (en) * | 2022-03-23 | 2022-06-24 | 山东大学 | Image defogging method and system based on multi-scale information selection attention mechanism |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111667489B (en) * | 2020-04-30 | 2022-04-05 | 华东师范大学 | Cancer hyperspectral image segmentation method and system based on double-branch attention deep learning |
| CN114821164B (en) * | 2022-04-13 | 2024-06-14 | 北京工业大学 | Hyperspectral image classification method based on twin network |
-
2022
- 2022-09-26 CN CN202211169177.5A patent/CN115272776B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021248687A1 (en) * | 2020-06-10 | 2021-12-16 | 南京理工大学 | Driving fatigue detection method and system combining pseudo 3d convolutional neural network and attention mechanism |
| CN111832620A (en) * | 2020-06-11 | 2020-10-27 | 桂林电子科技大学 | Image emotion classification method based on double-attention multilayer feature fusion |
| WO2022127227A1 (en) * | 2020-12-15 | 2022-06-23 | 西安交通大学 | Multi-view semi-supervised lymph node classification method and system, and device |
| CN113642634A (en) * | 2021-08-12 | 2021-11-12 | 南京邮电大学 | Shadow detection method based on mixed attention |
| CN114187261A (en) * | 2021-12-07 | 2022-03-15 | 天津大学 | Non-reference stereo image quality evaluation method based on multi-dimensional attention mechanism |
| CN114067107A (en) * | 2022-01-13 | 2022-02-18 | 中国海洋大学 | Multi-scale fine-grained image recognition method and system based on multi-grained attention |
| CN114418003A (en) * | 2022-01-20 | 2022-04-29 | 北京科技大学 | Double-image identification and classification method based on attention mechanism and multi-size information extraction |
| CN114663309A (en) * | 2022-03-23 | 2022-06-24 | 山东大学 | Image defogging method and system based on multi-scale information selection attention mechanism |
Non-Patent Citations (4)
| Title |
|---|
| A Spectral Spatial Attention Fusion with Deformable Convolutional Residual Network for Hyperspectral Image Classification;Tianyu Zhang et al;《Remote Sensing》;20210909;全文 * |
| Residual Group Channel and Space Attention Network for Hyperspectral Image Classification;Peida Wu et al;《Remote sensing》;20200624;全文 * |
| 土地利用/覆被深度学习遥感分类研究综述;冯权泷;《农业机械学报》;20220331;全文 * |
| 融合空间和通道注意力机制的害虫图像识别方法研究;罗勇琛;《中国优秀硕士学位论文数据库》;20220415;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115272776A (en) | 2022-11-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115249332B (en) | Hyperspectral image classification method and device based on space spectrum double-branch convolution network | |
| CN115272776B (en) | Hyperspectral image classification method based on double-path convolution and double attention and storage medium | |
| CN113159051B (en) | Remote sensing image lightweight semantic segmentation method based on edge decoupling | |
| CN114092832B (en) | High-resolution remote sensing image classification method based on parallel hybrid convolutional network | |
| CN108229550B (en) | Cloud picture classification method based on multi-granularity cascade forest network | |
| CN112347970B (en) | Remote sensing image ground object identification method based on graph convolution neural network | |
| CN106295613A (en) | A kind of unmanned plane target localization method and system | |
| CN112232328A (en) | Remote sensing image building area extraction method and device based on convolutional neural network | |
| CN113705641A (en) | Hyperspectral image classification method based on rich context network | |
| CN117475236B (en) | Data processing system and method for mineral resource exploration | |
| CN112884758A (en) | Defective insulator sample generation method and system based on style migration method | |
| CN112801185B (en) | Network security situation understanding and evaluating method based on improved neural network | |
| CN117557775B (en) | Substation power equipment detection method and system based on infrared and visible light fusion | |
| CN114255403A (en) | Optical remote sensing image data processing method and system based on deep learning | |
| CN111815526B (en) | Rain image rainstrip removing method and system based on image filtering and CNN | |
| CN104156943A (en) | Multi-target fuzzy cluster image variance detecting method based on non-control-neighborhood immune algorithm | |
| Zhou et al. | MSAR‐DefogNet: Lightweight cloud removal network for high resolution remote sensing images based on multi scale convolution | |
| CN113837191A (en) | Cross-satellite remote sensing image semantic segmentation method based on bidirectional unsupervised domain adaptive fusion | |
| CN117516937A (en) | Rolling bearing unknown fault detection method based on multi-mode feature fusion enhancement | |
| CN117315381A (en) | Hyperspectral image classification method based on second-order biased random walk | |
| CN113657472B (en) | Multi-source remote sensing data fusion method based on subspace learning | |
| CN110263638A (en) | A kind of video classification methods based on significant information | |
| CN111783688B (en) | Remote sensing image scene classification method based on convolutional neural network | |
| CN113962332B (en) | Salient target identification method based on self-optimizing fusion feedback | |
| CN115987730A (en) | Signal modulation identification method based on tree-shaped perception fusion convolutional network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |