CN116503636A - Multi-mode remote sensing image classification method based on self-supervision pre-training - Google Patents
Multi-mode remote sensing image classification method based on self-supervision pre-training Download PDFInfo
- Publication number
- CN116503636A CN116503636A CN202211551345.7A CN202211551345A CN116503636A CN 116503636 A CN116503636 A CN 116503636A CN 202211551345 A CN202211551345 A CN 202211551345A CN 116503636 A CN116503636 A CN 116503636A
- Authority
- CN
- China
- Prior art keywords
- remote sensing
- mode
- training
- self
- classification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000012549 training Methods 0.000 title claims abstract description 47
- 238000013145 classification model Methods 0.000 claims abstract description 23
- 230000000873 masking effect Effects 0.000 claims description 12
- 238000012706 support-vector machine Methods 0.000 claims description 8
- 238000010606 normalization Methods 0.000 claims description 6
- 230000007246 mechanism Effects 0.000 claims description 5
- 230000000903 blocking effect Effects 0.000 claims description 2
- 230000006870 function Effects 0.000 claims description 2
- 238000012545 processing Methods 0.000 abstract description 6
- 230000003993 interaction Effects 0.000 abstract description 2
- 238000013527 convolutional neural network Methods 0.000 description 6
- 238000002474 experimental method Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000002372 labelling Methods 0.000 description 5
- 230000000295 complement effect Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000000605 extraction Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000013136 deep learning model Methods 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000003595 spectral effect Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000003703 image analysis method Methods 0.000 description 1
- 238000013140 knowledge distillation Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a multi-mode remote sensing image classification method based on self-supervision pre-training, and belongs to the technical field of remote sensing image classification processing. The invention performs pre-training on a self-supervision learning model by using unlabeled multi-mode remote sensing images to obtain a trained encoder and a cross attention module, uses the trained encoder and the cross attention module as a multi-mode feature extractor to establish a multi-mode remote sensing image classification model, performs fine tuning training on the classification model by using a small number of multi-mode remote sensing images with labels, and inputs the multi-mode remote sensing images to be classified into the classification model after fine tuning training to realize remote sensing image classification. According to the method, a large number of images with labels are not needed, the encoder can learn key features of the remote sensing images through pre-training, meanwhile, information interaction among multi-mode features is achieved through the cross attention module, data of all modes are fully utilized, and the classification accuracy of the images is improved.
Description
Technical Field
The invention relates to a multi-mode remote sensing image classification method based on self-supervision pre-training, and belongs to the technical field of remote sensing image classification processing.
Background
With the continuous development of remote sensing technology, a large number of multi-mode remote sensing images can be used for cognizing the earth environment. Passive remote sensing technology can obtain high spatial and spectral resolution information of features, so that the obtained hyperspectral image (HSI), multispectral image (MSI) and ultrahigh resolution (VHR) image contain rich information and can be used for interpretation of observed scenes. Active remote sensing technology monitors targets by transmitting and receiving return information, and Synthetic Aperture Radar (SAR) and airborne laser data (LiDAR) record specific electromagnetic characteristics of the object under test. These multi-modal remote sensing images contain general and specific information of the observed scene, and analyzing these multi-modal images for land cover classification is very challenging.
The supervised classification method is the most common mode of land cover classification. Early studies focused mainly on image analysis methods such as band selection, feature extraction, and classifier design. Due to the strong feature extraction capability of deep learning models, typical deep networks are used for image classification, especially in the field of hyperspectral image classification. Convolutional Neural Networks (CNNs) may extract local depth features for recognition tasks, 1D CNN, 2D CNN, and 3D CNN for land cover classification. The cyclic neural network (RNN) classifies sequence information included in hyperspectral images, and recently, in order to explore long-range dependencies, a transducer has been studied to further improve classification performance. In addition to the above model-driven approach, hyperspectral image classification also employs several machine learning strategies such as attention mechanisms, knowledge distillation, multi-scale learning, sparse representation, and the like. For the collaborative classification of the multi-mode remote sensing images, hyperspectral and LiDAR data classification, hyperspectral and multispectral image classification, hyperspectral image and SAR image collaborative classification exist. These supervised methods are data driven methods, the accuracy of which depends to a large extent on the number of training samples. However, labeling high quality samples is laborious, thus solving the problem of the scarcity of labeled samples in the field of remote sensing image classification.
Semi-supervised learning paradigms provide a viable solution to the small sample classification problem by using labeled and unlabeled samples. Graph-based learning is a typical semi-supervised classification model of remote sensing data, where graph-rolling neural networks and generation countermeasure networks can extract depth features from both labeled and unlabeled data for land cover classification, which have shortcomings in solving sample generation and large image classification problems. Another important direction of research to address the scarcity of annotation samples is the small sample learning paradigm, but this pre-training approach is supervised and this process also requires a large number of annotation samples for feature learning.
Although both current supervised and semi-supervised models achieve significant performance, these classification schemes still fail to effectively address the most prominent problem, namely the large amount of unlabeled multimodal data and the limited number of labeled samples. As a novel learning paradigm, self-supervised learning utilizes inherent features of unlabeled data to learn salient features and uses the learned feature representations for downstream recognition tasks. Self-supervised learning can be generalized to methods based on contrast and generative formulas, depending on the learning objectives in the designed interface task. Contrast learning aims at learning a potential space in which similar pairs of samples are clustered together and different pairs of samples are separated, thereby learning invariant and distinguishing feature representations. Contrast self-supervised learning typically employs CNN as a basic feature extractor to extract high-level abstract features, which have limitations in remote dependency extraction and multi-modal data processing.
The goal of the generative self-supervision to achieve feature learning by recovering artificially corrupted data is that once the model can recover the original signal from the corrupted data, this means that the model has learned key features that characterize the original signal. The generated self-supervision pre-training scheme is very successful in the field of natural language processing, but is lagged in the field of vision, and the main reason is that images are natural signals with a large amount of spatial redundancy, so that advanced key information is difficult to learn from the images, particularly in the field of remote sensing image processing, multi-mode remote sensing images exist in the same observation scene, and the information of the heterogeneous images is rich and complementary, however, the current generated self-supervision model cannot fully utilize the heterogeneous images, and has limited multi-mode characteristic learning capability, so that the final precision is lower.
Disclosure of Invention
The invention aims to provide a multi-mode remote sensing image classification method based on self-supervision pre-training, which aims to solve the problem that the existing self-supervision model cannot fully utilize heterogeneous images to cause low classification precision.
The invention provides a multi-mode remote sensing image classification method based on self-supervision pre-training for solving the technical problems, which comprises the following steps:
1) Acquiring remote sensing images of at least two modes, respectively performing blocking treatment on the remote sensing images of each mode, dividing each mode of remote sensing image into regular non-overlapping blocks, performing random masking treatment on the obtained image blocks to obtain embedded characteristic information of the unmasked image blocks, and recording the characteristics of the masked image blocks;
2) The method comprises the steps of inputting the obtained embedded characteristic information of an unmasked image block into an encoder, learning the input embedded characteristic information by the encoder, and exchanging results among mode characteristics through a cross attention module;
3) Inputting the learning result of the cross attention module and the characteristics of the masking image blocks into a decoder, reconstructing the masking image blocks in each mode of remote sensing image by the decoder, taking the difference between the reconstructed blocks and the corresponding masking image blocks as loss, and training the encoder and the cross attention module by utilizing the loss;
4) Taking the trained encoder and the cross attention module as a multi-modal feature extractor, constructing a lightweight classifier, taking the multi-modal feature extractor and the lightweight classifier as a multi-modal remote sensing image classification model, and performing fine tuning training on the classification model;
5) And inputting each mode of remote sensing image to be classified into the classification model after fine adjustment training, namely realizing the classification of the multi-mode remote sensing image.
The invention performs pre-training on a self-supervision learning model by using unlabeled multi-mode remote sensing images to obtain a trained encoder and a cross attention module, uses the trained encoder and the cross attention module as a multi-mode feature extractor to establish a multi-mode remote sensing image classification model, performs fine tuning training on the classification model by using a small number of multi-mode remote sensing images with labels, and inputs the multi-mode remote sensing images to be classified into the classification model after fine tuning training to realize remote sensing image classification. According to the method, a large number of images with labels are not needed, the encoder can learn key features of the remote sensing images through pre-training, meanwhile, information interaction among multi-mode features is achieved through the cross attention module, data of all modes are fully utilized, and the classification accuracy of the images is improved.
Further, the fine tuning in the step 4) uses the mean square error between the masking image block and the reconstruction block as a loss function.
The invention only calculates the reconstruction loss on the masking image block and the corresponding reconstruction block during the pre-training, thereby not only ensuring the training precision, but also avoiding the problem of large calculation amount caused by calculation on all the image blocks.
Further, the encoder and decoder adopt a transducer structure; the encoder comprises a position coding module and a plurality of transducer structures; the decoder comprises N decoding units, wherein N is the number of modes of the remote sensing image, and each decoding unit comprises two transducer structures and a multi-layer sensor.
Further, the transducer structure comprises a multi-layer sensor, a layer normalization module and a multi-head attention.
The invention adopts a multi-layer perceptron, a layer normalization module and a multi-head attention to construct a transducer structure.
Further, the cross-attention module adopts a multi-head cross-attention mechanism.
The invention uses the cross attention module to exchange information among heterogeneous features after the feature learning of the encoder to enhance the representation capability of the learned features, and realizes the cross fusion among different modal features through a multi-head cross attention (MCA) mechanism.
Further, the lightweight classifier is a support vector machine classifier.
The invention adopts the support vector machine classifier as the lightweight classifier, and can quickly and efficiently realize classification.
Drawings
FIG. 1 is a flow chart of a multi-modal remote sensing image classification method based on self-supervision pre-training in accordance with the present invention;
FIG. 2 is a network structure diagram of a self-supervised learning model constructed in accordance with the present invention;
FIG. 3 is a schematic diagram of a multi-modal self-supervising pre-training and fine tuning scheme employed by the present invention;
FIG. 4 is a schematic diagram of experimental data used in the experimental verification process;
FIG. 5 is a graph comparing the classification of the present invention with the prior classification method on a reference dataset.
Detailed Description
The following describes the embodiments of the present invention further with reference to the drawings.
The invention adopts a self-supervision learning model formed by an asymmetric encoder-decoder to pretrain, the learning model firstly utilizes the encoder to learn the advanced generalized information contained in the multi-mode data, exchanges among the mode characteristics through the cross attention module to learn more complementary information from the multi-mode remote sensing data, and finally utilizes the decoder to reconstruct so as to realize the training of the encoder and the cross attention module; then taking the trained encoder and the cross attention module as a multi-modal feature extractor, constructing a lightweight classifier, taking the multi-modal feature extractor and the lightweight classifier as a multi-modal remote sensing image classification model, and performing fine tuning training on the classification model; and finally, inputting each mode of remote sensing image to be classified into a classification model after fine adjustment training, namely realizing the classification of the multi-mode remote sensing image. The implementation principle of the method is shown in fig. 1, and is described in detail below with reference to a specific example.
1. And constructing a self-supervision learning model.
As shown in fig. 2, the self-supervised learning model constructed by the present invention employs an asymmetric encoder-decoder structure, and both the encoder and decoder employ a transducer structure. The transform structure comprises a Layer normalization module (Layer Norm), a residual block, a multi-Layer perceptron (MLP) and a multi-head attention (MHA) structure, wherein the multi-head attention is used for constructing a plurality of subspaces from multi-mode marks to learn complex dependency relations, the residual connection structure can comprehensively utilize characteristics of different stages and promote training of a model, and the Layer normalization performs normalization operation on the extracted characteristics. The formula of the multi-head attention is expressed as:
MHA(Q,K,V)＝Concat(h 1 ,h 2 ,…,h h )W O
wherein Q is i ，K i And V i Queries, indexes, and inliers respectively representing the ith headerThe capacity, the number of heads is h.
The encoder comprises a position encoding module and a plurality of transducer structures, wherein the input of the transducer encoder is a multi-mode visible mark, and the class mark represents information of corresponding mode data and is used for learning universality information from the multi-mode data. Relative position encoding is used to represent the position information of the feature, and class labels are also added to each modality data for feature T v Representation, T v And representing the characteristics of the mask-free image sub-blocks in the remote sensing images after the embedded characteristics are stacked. Visible embedded feature T after the embedding operation v Is input to the encoder for learning, and can be expressed as:
T e ＝Encoder(T v )
in order to enrich the learned multi-modal features, the cross attention module is used for exchanging information after the encoder, the cross attention module is shown in fig. 2, a multi-head cross attention (MCA) structure is adopted, and when the features of the three modes are fused, the features of any two modes are fused based on the cross attention. Tagging classes from a modalityAs proxy, with patch flag from another modality +.>The connection is represented as follows:
as shown in FIG. 2, a multi-headed cross-attention (MCA) mechanism pair is employedAnd->The processing is performed, which can be expressed as follows,
where Q, K and V represent query, index, and content values in a cross-attention operation, and C represents an embedded feature dimension. Pairs of heterogeneous features are integrated into the content aware mode through the cross-attention layer, thereby enhancing presentation capabilities.
The decoder is used for reconstructing specific modes according to the multi-mode features and mask marks, and comprises N decoding units, wherein N is the number of modes of the remote sensing image, each decoding unit comprises two transducer structures and a multi-layer sensor, and the input of each decoding unit is an unmasked embedded featureMask feature->Location embedding is used to provide location information. Can be expressed as:
wherein, the liquid crystal display device comprises a liquid crystal display device,and->The visual and mask feature embedding representing the i-th modality data, respectively. At the last layer of each decoding module, the prediction head reconstructs the image block in pixel space, using the fully connected layer as the prediction head. Prediction header of decoding moduleIt projects into a vector that is the same dimension as the input samples. Then obtaining a prediction block P through matrix transformation operation i Can be used as a watchThe method comprises the following steps:
for the present embodiment, the acquired multi-modal data refers to images of three modes, i.e., HIS (hyperspectral image), DSM (digital earth model image) and VHR (high resolution image), and as another embodiment, images of other modes may be selected. The image data of the three modes contains rich and complementary information, can be subjected to self-supervision feature learning, and needs to be preprocessed before self-learning.
The pretreatment process is as follows: taking each pixel in the image as a center, cutting out a local image with the size of H multiplied by W as a self-supervision pre-training processing unit, and enablingRepresenting a sample representing ith modality data, wherein H, W, B i Representing the height, width and depth of the sample, respectively; dividing the samples of each modality into regular non-overlapping blocks, randomly masking a subset of these blocks at a ratio of m, embedding a trainable linear projection into the unmasked patch, and concatenating the embedded markers of all modalities as input to a unified encoder, denoted T v The method comprises the steps of carrying out a first treatment on the surface of the Each mask mark is represented by a trainable vector which is used as a placeholder for the decoder, denoted T m 。
2. And pre-training the constructed self-supervision learning model.
Similar to a Masked Automatic Encoder (MAE), the present invention normalizes the original block and the reconstructed block, calculates the Mean Square Error (MSE) between the original block and the reconstructed block, and calculates the reconstruction loss, which is the sum of each modal error, only on the masked block and the corresponding reconstructed block. Based on the total reconstruction loss, the built self-supervision learning model is pre-trained, and a training sample with marks is not needed in the pre-training process.
3. And constructing a multi-mode remote sensing image classification model by using the pre-trained encoder and the cross attention module.
After the pre-training process, the trained encoder and the cross attention module are used as a multi-modal feature extractor, and a multi-modal remote sensing image classification model is constructed by using the multi-modal feature extractor and a lightweight classifier, wherein the lightweight classifier in the embodiment adopts a Support Vector Machine (SVM). Because the self-monitoring pre-training process can convert the local remote sensing image block into the feature vector, other classifiers, such as a nearest neighbor classification model, can also be adopted. As shown in FIG. 3, the method utilizes the trained encoder and the cross attention layer to extract the characteristics of the input multi-mode remote sensing image to obtain corresponding characteristics, normalizes class marks of the multi-mode characteristics, connects the class marks with corresponding spectrum information, inputs the class marks into the SVM for classification, and utilizes a small number of samples with labels to finely tune the classification model before the classifier, so that the classification of the multi-mode remote sensing image can be realized through the finely tuned classification model.
Because the features learned by the pre-training stage encoder and the cross attention module have high discriminant, when the classification is performed, only a small amount of labeling samples are needed to train the lightweight classifier in the fine adjustment stage, so that the urgent requirement on the labeling samples is relieved, and the labeling samples do not need to be relied on.
Experiment verification
To further verify the effect of the present invention, the model proposed by the present invention was implemented in the PyTorch framework using Python programming language, using the main classification evaluation coefficients, namely Overall Accuracy (OA), average Accuracy (AA) and kappa coefficient (κ), to quantitatively evaluate classification performance, while also qualitatively evaluating experimental results using classification graphs.
The experiment selects a Berlin data set as experimental data, wherein the data set comprises hyperspectral data and PolSAR data in Berlin areas, the hyperspectral data consists of 244 wave bands, and the spectral range is 400nm to 2500nm; the spatial size is 380×2384 pixels, and the spatial resolution is 30m; the image coverage contains 8 distinguishable features, as shown in fig. 4.
To evaluate the classification performance of the proposed method, the experiment was compared using common classification patterns, including supervised deep learning models (i.e., SVM, CDCNN, SSUN and SSRN), semi-supervised learning models (i.e., TSVM and CEGCN), and small sample learning methods (i.e., DFSL and DMVL). In the experiment, 20 samples are selected for classification of each type of ground feature in order to verify the classification precision of different models under the condition of small samples.
From the experimental results, the average OA, AA, kappa coefficient and the corresponding variance of the present invention (MultiSSL) and each method and the classification accuracy of each feature are shown in Table 1.
TABLE 1
Table 1 shows detailed comparative experimental results obtained for the different classification methods with respect to average OA, AA, kappa and accuracy for each class, and also reports the root mean square error for OA, AA and kappa. By the above quantitative comparison, it can be determined that:
semi-supervised and self-supervised methods have better performance than supervised methods when classifying with a small number of labelled samples. In the supervised classification method, a small number of marked samples can cause a fitting phenomenon, so that classification performance is seriously affected; the semi-supervised learning method utilizes marked and unmarked samples simultaneously, and the classification precision is higher than that of a supervised model; the self-supervision learning method learns key feature representation in a designed interface task, effectively improves classification performance, particularly DMVL, can learn meaningful features from unlabeled data based on a comparison learning paradigm, and has a higher classification accuracy than other comparison methods.
According to the invention, a multi-mode self-supervision pre-training and fine tuning scheme is adopted, and when the same number of labeling samples are used, the obtained main evaluation coefficient and the individual classification accuracy achieve the best classification performance, and the overall classification accuracy on a reference data set reaches 70.34%. For the Berlin dataset selected in the experiment, the existing learning paradigm realizes low classification accuracy through a small number of marked samples, and the method provided by the invention realizes higher accuracy on a complex scene because the model provided by the invention utilizes multi-mode remote sensing data to perform self-supervision pre-training, thereby learning key features of subsequent classification tasks and effectively relieving the high dependence on annotation samples.
As shown in fig. 5, classification charts obtained by different classification methods on the reference data set are GT, SVM, CDCNN, SSUN, SSRN, TSVM, CEGCN, 3DCAE, DFSL, DMVL and then MultiSSL (the present invention) in this order from left to right, and in these charts, different land objects are represented by different colors. It can be observed that the semi-supervised and self-supervised learning methods obtain more uniform classification graphs, the classification graphs obtained by the model provided by the invention have fewer noise pixels, and the details in the enlarged view are more accurate, which indicates that the multi-SSL method can learn more distinguishing features from the multi-mode remote sensing image.
The invention constructs a self-supervision learning model by an asymmetric encoder-decoder, pretrains the self-supervision learning model by utilizing unlabeled multi-mode remote sensing images to obtain a trained encoder and a cross attention module, takes the trained encoder and the cross attention module as a multi-mode feature extractor, takes the multi-mode feature extractor and a lightweight classifier as a multi-mode remote sensing image classification model, and carries out fine tuning training on the classification model by utilizing a small quantity of multi-mode remote sensing images with labels. The effectiveness and superiority of the classification method provided by the invention are further proved by carrying out comprehensive experiments on the multi-mode reference data set, and a feasible solution is provided for solving the problem of classifying small samples.
Claims (6)
1. A multi-mode remote sensing image classification method based on self-supervision pre-training is characterized by comprising the following steps:
1) Acquiring remote sensing images of at least two modes, respectively performing blocking treatment on the remote sensing images of each mode, dividing each mode of remote sensing image into regular non-overlapping blocks, performing random masking treatment on the obtained image blocks to obtain embedded characteristic information of the unmasked image blocks, and recording the characteristics of the masked image blocks;
2) The method comprises the steps of inputting the obtained embedded characteristic information of an unmasked image block into an encoder, learning the input embedded characteristic information by the encoder, and exchanging results among mode characteristics through a cross attention module;
3) Inputting the learning result of the cross attention module and the characteristics of the masking image blocks into a decoder, reconstructing the masking image blocks in each mode of remote sensing image by the decoder, taking the difference between the reconstructed blocks and the corresponding masking image blocks as loss, and training the encoder and the cross attention module by utilizing the loss;
4) Taking the trained encoder and the cross attention module as a multi-modal feature extractor, constructing a lightweight classifier, taking the multi-modal feature extractor and the lightweight classifier as a multi-modal remote sensing image classification model, and performing fine tuning training on the classification model;
5) And inputting each mode of remote sensing image to be classified into the classification model after fine adjustment training, namely realizing the classification of the multi-mode remote sensing image.
2. The method for classifying multi-modal remote sensing images based on self-supervised pre-training according to claim 1, wherein the fine tuning in step 4) uses the mean square error between the masking image block and the reconstruction block as a loss function.
3. The method for classifying the multi-mode remote sensing images based on the self-supervision pre-training according to claim 1 or 2, wherein the encoder and the decoder adopt a transducer structure, and the encoder comprises a position encoding module and a plurality of transducer structures; the decoder comprises N decoding units, wherein N is the number of modes of the remote sensing image, and each decoding unit comprises two transducer structures and a multi-layer sensor.
4. The method for classifying multi-modal remote sensing images based on self-supervision pre-training according to claim 3, wherein the transducer structure comprises a multi-layer sensor, a layer normalization module and a multi-head attention.
5. The multi-modal remote sensing image classification method based on self-supervision pre-training according to claim 1, wherein the cross-attention module adopts a multi-head cross-attention mechanism.
6. The method for classifying multi-modal remote sensing images based on self-supervision pre-training according to claim 1, wherein the lightweight classifier is a support vector machine classifier.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211551345.7A CN116503636A (en) | 2022-12-05 | 2022-12-05 | Multi-mode remote sensing image classification method based on self-supervision pre-training |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211551345.7A CN116503636A (en) | 2022-12-05 | 2022-12-05 | Multi-mode remote sensing image classification method based on self-supervision pre-training |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116503636A true CN116503636A (en) | 2023-07-28 |
Family
ID=87319016
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211551345.7A Pending CN116503636A (en) | 2022-12-05 | 2022-12-05 | Multi-mode remote sensing image classification method based on self-supervision pre-training |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116503636A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117095309A (en) * | 2023-10-20 | 2023-11-21 | 武汉工程大学 | Polarized SAR image rotation domain feature expression extraction and classification method |
-
2022
- 2022-12-05 CN CN202211551345.7A patent/CN116503636A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117095309A (en) * | 2023-10-20 | 2023-11-21 | 武汉工程大学 | Polarized SAR image rotation domain feature expression extraction and classification method |
| CN117095309B (en) * | 2023-10-20 | 2024-01-16 | 武汉工程大学 | Polarized SAR image rotation domain feature expression extraction and classification method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zhong et al. | Generalizing a person retrieval model hetero-and homogeneously | |
| Yang et al. | A survey of DNN methods for blind image quality assessment | |
| Nandhini Abirami et al. | Deep CNN and deep GAN in computational visual perception-driven image analysis | |
| Dong et al. | Local information enhanced graph-transformer for hyperspectral image change detection with limited training samples | |
| Shi et al. | Image manipulation detection and localization based on the dual-domain convolutional neural networks | |
| CN113936339A (en) | Fighting identification method and device based on double-channel cross attention mechanism | |
| CN110046579B (en) | Deep Hash pedestrian re-identification method | |
| CN110188827A (en) | A kind of scene recognition method based on convolutional neural networks and recurrence autocoder model | |
| Xu et al. | Dktnet: dual-key transformer network for small object detection | |
| CN115222994A (en) | Hyperspectral image classification method based on hybrid spectrum network and multi-head self-attention mechanism | |
| Xue et al. | Grafting transformer on automatically designed convolutional neural network for hyperspectral image classification | |
| Anwer et al. | Compact deep color features for remote sensing scene classification | |
| CN116503636A (en) | Multi-mode remote sensing image classification method based on self-supervision pre-training | |
| Khurshid et al. | A residual-dyad encoder discriminator network for remote sensing image matching | |
| CN114708455A (en) | Hyperspectral image and LiDAR data collaborative classification method | |
| Li et al. | Anomaly detection methods based on GAN: a survey | |
| Xu et al. | Spatial-Spectral 1DSwin Transformer with Group-wise Feature Tokenization for Hyperspectral Image Classification | |
| Zhang et al. | Temporal transformer networks with self-supervision for action recognition | |
| Kumari et al. | Deep learning techniques for remote sensing image scene classification: A comprehensive review, current challenges, and future directions | |
| CN117036904A (en) | Attention-guided semi-supervised corn hyperspectral image data expansion method | |
| Salmi et al. | Low complexity image enhancement GAN-based algorithm for improving low-resolution image crop disease recognition and diagnosis | |
| CN116612004A (en) | Double-path fusion-based hyperspectral image reconstruction method | |
| Aziz et al. | Multi-level refinement feature pyramid network for scale imbalance object detection | |
| Juefei-Xu et al. | DeepGender2: A generative approach toward occlusion and low-resolution robust facial gender classification via progressively trained attention shift convolutional neural networks (PTAS-CNN) and deep convolutional generative adversarial networks (DCGAN) | |
| Fondje et al. | Learning domain and pose invariance for thermal-to-visible face recognition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |