CN117611896B - Multi-mode remote sensing data classification basic model self-adaption method - Google Patents
Multi-mode remote sensing data classification basic model self-adaption method Download PDFInfo
- Publication number
- CN117611896B CN117611896B CN202311580794.9A CN202311580794A CN117611896B CN 117611896 B CN117611896 B CN 117611896B CN 202311580794 A CN202311580794 A CN 202311580794A CN 117611896 B CN117611896 B CN 117611896B
- Authority
- CN
- China
- Prior art keywords
- remote sensing
- mode
- sensing data
- data
- spatial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 239000013598 vector Substances 0.000 claims abstract description 33
- 230000003595 spectral effect Effects 0.000 claims abstract description 27
- 230000003993 interaction Effects 0.000 claims abstract description 20
- 230000007246 mechanism Effects 0.000 claims abstract description 17
- 238000001228 spectrum Methods 0.000 claims abstract description 8
- 238000013507 mapping Methods 0.000 claims abstract description 7
- 238000007781 pre-processing Methods 0.000 claims abstract description 4
- 239000011159 matrix material Substances 0.000 claims description 27
- 238000013527 convolutional neural network Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 12
- 230000006870 function Effects 0.000 claims description 11
- 230000008569 process Effects 0.000 claims description 9
- 230000004913 activation Effects 0.000 claims description 7
- 238000010606 normalization Methods 0.000 claims description 7
- 230000009467 reduction Effects 0.000 claims description 6
- 230000006978 adaptation Effects 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000012545 processing Methods 0.000 abstract description 6
- 238000013135 deep learning Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000008033 biological extinction Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
- G06V10/765—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects using rules for classification or partitioning the feature space
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A40/00—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production
- Y02A40/10—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production in agriculture
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
A multi-mode remote sensing data classification basic model self-adaption method belongs to the technical field of image processing. The method solves the problem that the classification of the multi-mode remote sensing data is difficult to realize by directly using the pre-trained basic model in the prior art. The key points are as follows: acquiring multi-mode remote sensing data; preprocessing multi-mode remote sensing data; establishing a mapping layer and selecting a basic model; constructing a cross-space interaction module to realize interaction of the general features and the space features in the multi-mode data, generating a learnable space coding vector set along the space dimension of the multi-mode remote sensing data, adding the learnable space coding vector set into the feature coding vector set, and outputting the space features; constructing a cross-channel interaction module, adding the characteristic interaction between the general characteristic and the channel dimension in the multi-mode data in a multi-head attention mechanism MSA of an encoder module of a basic model, and outputting a spectrum characteristic; and inputting the spatial features and the spectral features into the full-connection layer to obtain a classification result. The method and the device improve the classification accuracy of the multi-mode remote sensing data.
Description
Technical Field
The invention relates to a multi-mode remote sensing data classification method, in particular to a basic model self-adaption method for multi-mode remote sensing data classification, and belongs to the technical field of image processing.
Background
Multimodal in the field of remote sensing generally refers to the imaging results of a scene and a target under different sensors (multispectral, hyperspectral, synthetic aperture radar, lidar, etc.). The multi-mode remote sensing data can be reasonably utilized to provide more comprehensive description information for ground objects from spectrum, time and space, so that the interpretation capability of remote sensing is improved, and the requirements of practical applications such as military reconnaissance and intelligent agriculture are met.
In order to promote the deep and wide application of the multi-mode remote sensing data in the application, an effective information processing means is needed. Classification is one of important links of remote sensing data processing technology, and is a hot spot problem of current research. In recent years, deep learning has become a mainstream of multi-modal remote sensing data classification due to its strong feature extraction capability. Among the many deep learning methods, the transform-based multi-modal remote sensing data method has attracted a great deal of attention. The self-attention mechanism in the transducer enables it to focus on different modalities simultaneously and capture long-range dependencies, but the classification accuracy of these methods still needs to be improved.
With the development of the deep learning method, a neural network, namely a basic model, which is trained by unsupervised learning on the basis of a large amount of original data emerges. On the downstream task, better classification performance can be obtained by only a small amount of marking data for the basic model. However, the data set used for training the basic model and the multi-modal data have large differences, so that it is difficult to directly use the pre-trained basic model to realize the classification of the multi-modal remote sensing data.
Disclosure of Invention
In order to solve the problem that the pre-trained basic model is difficult to directly use to realize the classification of the multi-mode remote sensing data in the prior art, the invention explores the potential and the effectiveness of the basic model in the multi-mode remote sensing data classification task and further provides a multi-mode remote sensing data classification basic model self-adaption method, and the classification effect can be improved without fine adjustment of basic model parameters. The following presents a simplified summary of the invention in order to provide a basic understanding of some aspects of the invention. It should be understood that this summary is not an exhaustive overview of the invention. It is not intended to identify key or critical elements of the invention or to delineate the scope of the invention.
The technical scheme of the invention is as follows:
a multi-mode remote sensing data classification basic model self-adaption method comprises the following steps:
s1, acquiring multi-mode remote sensing data;
s2, preprocessing multi-mode remote sensing data, taking a pixel with a label in each mode as a center, selecting a rectangular window with the size of 27 multiplied by 27 as a sample of each single-mode data, amplifying all samples to 224 multiplied by 224 by a bilinear interpolation method for the sample of each mode data, and matching the spatial input dimension of a basic model;
S3, establishing a mapping layer, respectively sending samples of each mode data into the mapping layer to perform dimension reduction matching on channel dimensions, and fusing different single-mode data samples subjected to dimension reduction to obtain fused multi-mode data input samples;
S4, selecting a basic model based on a transducer, inputting the multi-mode remote sensing data into the basic model, and extracting general characteristics of the multi-mode data;
S5, constructing a cross-space interaction module to realize interaction of the general features and the spatial features in the multi-mode data, generating a learnable spatial coding vector set S 0 along the spatial dimension of the multi-mode remote sensing data, adding the learnable spatial coding vector set S 0 into a feature coding vector set Z 0, and outputting the spatial features;
S6, constructing a cross-channel interaction module, and adding the characteristic interaction between the general characteristic and the channel dimension in the multi-mode data in a multi-head attention mechanism MSA of an encoder module of a basic model to output a spectrum characteristic;
s7, inputting the output spatial features and the spectral features into the full-connection layer to obtain a classification result.
Further: the specific implementation process of the step S5 is as follows:
S51, selecting hyperspectral data in multimode remote sensing data, selecting a hyperspectral data sample x HSI, and extracting spatial characteristics of the hyperspectral data sample x HSI by a convolutional neural network; CNN consists of a1×1 convolutional layer, a batch normalization layer, a ReLU activation function, and a3×3 convolutional layer, and its output spatial feature S is formulated as:
S＝CNN(xHSI) (9)
S52, dividing the spatial feature S into N non-overlapping feature blocks to form a spatial feature coding set S 0, and splicing the S 0 and a feature coding vector set Z 0, wherein the method is expressed as follows:
further: the specific implementation process of the step S6 is as follows:
S61, taking the average spectrum characteristic of each hyperspectral data sample x HSI as an input;
s62, inputting the spectral features into CNNs of a 1X 1 convolution layer, a batch normalization layer, a ReLU activation function and an MLP layer to output a spectral feature coding set D';
S63.D' is used to generate D Q、DK and D V that match the three matrices of query matrix Q, key value matrix K, value matrix V, and add them to Q i,Ki and V i in the multi-headed self-attention mechanism MSA in the base model;
Wherein, D Q and D K are represented by the intermediate vector of D 'for extracting the spectral feature of the feature represented by the center pixel, and D V is the same as D' and represents the global spectral feature;
s64, updating the calculation mode in the MSA:
Where D Q represents the spectral vector added to Q i, D K represents the spectral vector added to K i, and D V represents the spectral vector added to V i.
The beneficial effects of the invention are as follows:
Compared with the prior art, the basic model self-adaptive framework for multi-mode remote sensing data classification provided by the invention fuses the general features, the spatial features and the relations among the spectral features extracted by all multi-mode basic models only by adjusting the cross-space interaction module and the cross-channel interaction module, simultaneously freezes the parameters of the basic models, retains strong general knowledge extracted by the basic models, finally reasonably utilizes the pre-trained parameters of the basic models, and fuses the unique features of the multi-mode remote sensing data such as space, spectrum and the like to improve the classification precision of the multi-mode remote sensing data.
Drawings
FIG. 1 is a schematic diagram of a multi-modal remote sensing data classification method based on basic model fine tuning as described in example 1;
FIG. 2 is a schematic diagram of a cross-space interaction module in a basic model adaptive method for multi-modal remote sensing data classification according to embodiment 1;
fig. 3 is a schematic diagram of a cross-channel interaction module in a basic model adaptive method for multi-modal remote sensing data classification according to embodiment 1.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the present invention is described below by means of specific embodiments shown in the accompanying drawings. It should be understood that the description is only illustrative and is not intended to limit the scope of the invention. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the present invention. It should be further understood that the terms "first," "second," "third," and the like in this specification are used merely for distinguishing between various components, elements, steps, etc. in the specification and not for indicating a logical or sequential relationship between the various components, elements, steps, etc., unless otherwise indicated.
Embodiment 1, which is described with reference to fig. 1-3, provides a method for adapting a base model of multi-modal remote sensing data classification, comprising the following steps:
s1, acquiring multi-mode remote sensing data;
s2, preprocessing multi-mode remote sensing data, taking a pixel with a label in each mode as a center, selecting a rectangular window with the size of 27 multiplied by 27 as a sample of each single-mode data, amplifying all samples to 224 multiplied by 224 by a bilinear interpolation method for the sample of each mode data, and matching the spatial input dimension of a basic model;
S3, establishing a mapping layer, respectively sending samples of each mode data into the mapping layer to perform dimension reduction matching on channel dimensions, and fusing different single-mode data samples subjected to dimension reduction to obtain fused multi-mode data input samples;
S4, selecting a basic model based on a transducer, for example DINOv, inputting the multi-mode remote sensing data into the basic model to extract the general characteristics of the multi-mode data, such as the characteristics of edges, angles and the like; the specific implementation process of the steps is as follows:
S41, dividing the fused multi-mode input sample into N sample blocks in a non-overlapping mode to form an input sequence of X= [ X 1,x2,…,xN ];
S42, each sample block is sent into a linear projection layer to perform linear transformation, N feature coding vectors Z n are output, and in order to obtain global information of an input sequence, a learnable feature coding vector Z cls is added to Z n to form a feature coding vector set Z 0;
S43, adding the feature coding vector set and the position coding vectors, and inputting the feature coding vector set and the position coding vectors into L encoders; in the encoder, the core module is a multi-head (MS) attention mechanism, defining a query weight matrix W Q, a key weight matrix W K, and a value weight matrix W V; x obtains three matrixes of a query matrix Q, a key matrix K and a value matrix V according to three different weight matrixes, and a calculation formula is shown in a formula (1):
Q＝XWQ,K＝XWK,V＝XWV (1)
S44, inquiring the matrix Q and the value matrix V, and calculating the similarity between each element in the X through matrix multiplication QK T to obtain a similarity matrix;
The similarity matrix is processed through a Softmax activation function and is calculated as follows:
Wherein z i represents input elements, C represents the number of the input elements, and the similarity matrix is processed through a Softmax function to obtain a normalized result, wherein a larger value represents a higher similarity between the elements;
s45, performing matrix multiplication on the similarity matrix and V to obtain an output result of the self-attention mechanism, wherein the calculation formula is as follows:
Where d k represents the dimension of input X.
In order to enrich the extracted features of the self-attention mechanism and avoid sinking into certain local features, a transducer introduces a multi-head attention mechanism on the basis of a single-head self-attention mechanism. The multi-headed self-attention mechanism is an attention mechanism that parallelizes the scaled dot product attention mechanism, the number of parallelism being called the "head count", so that the model focuses on multiple subspace information simultaneously. The process of the multi-head self-attention mechanism MSA is expressed as:
Zi＝Attention(Qi,Ki,Vi),i＝1...h (5)
MS(Q,K,V)＝Concat(Z1,Z2,...,Zh)Wo (6)
wherein i represents the number of the header, Representing the output projection matrix, Z i representing the output matrix for each head,/>Key matrix representing the ith head,/>Representing the value matrix of the i-th header.
In each encoder, the post-MSA connection layer normalizes (Layer Normalization, LN) the layers and multi-layer perceptrons (Multilayer perceptron, MLP), and a residual connection is added after each module to prevent gradient extinction, the process being expressed as follows:
Hl-1＝MSA(LN(Zl-1))+Zl-1,l＝0,...,L-1 (7)
Zl＝MLP(LN(Hl-1))+Hl-1,l＝0,...,L-1 (8)。
S5, constructing a cross-space interaction module to realize interaction of the general features and the spatial features in the multi-mode data, generating a learnable spatial coding vector set S 0 along the spatial dimension of the multi-mode remote sensing data, adding the learnable spatial coding vector set S 0 into the feature coding vector set Z 0, and outputting the spatial features. The specific implementation process of the steps is as follows:
S51, selecting hyperspectral data in the multimode remote sensing data, selecting a hyperspectral data sample x HSI, and extracting spatial characteristics of the hyperspectral data sample x HSI by a convolutional neural network (Convolutional Neural Network, CNN). CNN consists of a1×1 convolutional layer, a batch normalization layer, a ReLU activation function, and a 3×3 convolutional layer, and its output spatial feature S is formulated as:
S＝CNN(xHSI) (9)
S52, dividing the spatial feature S into N non-overlapping feature blocks to form a spatial feature coding set S 0, and splicing the S 0 with the Z 0 on the basis of a formula (7) and a formula (8), wherein the method is expressed as follows:
s6, constructing a cross-channel interaction module, and adding the characteristic interaction between the general characteristic and the channel dimension in the multi-mode data in a multi-head attention mechanism MSA of an encoder module of the basic model to output the spectral characteristic. The specific implementation process of the steps is as follows:
S61, taking the average spectrum characteristic of each hyperspectral data sample x HSI as an input;
s62, inputting the spectral features into CNNs of a 1X 1 convolution layer, a batch normalization layer, a ReLU activation function and an MLP layer to output a spectral feature coding set D';
S63.D' is used to generate D Q、DK and D V that match the three matrices of query matrix Q, key value matrix K, value matrix V, and add them to Q i,Ki and V i in the multi-headed self-attention mechanism MSA in the base model;
Wherein, D Q and D K are represented by the intermediate vector of D 'for extracting the spectral feature of the feature represented by the center pixel, and D V is the same as D' and represents the global spectral feature;
s64, updating the calculation mode in the MSA:
Wherein, D Q represents the spectral vector added to Q i, D K represents the spectral vector added to K i, and D V represents the spectral vector added to V i.
S7, inputting the output spatial features and the spectral features into the full-connection layer to obtain a classification result.
Example 2:
an electronic device comprising a memory and a processor, the memory storing a computer program, the processor implementing the steps of the base model adaptation method of multimodal remote sensing data classification of embodiment 1 when executing the computer program.
The computer device of the present invention may be a device including a processor and a memory, such as a single chip microcomputer including a central processing unit. And the processor is used for realizing the steps of the basic model self-adaption method for classifying the multi-mode remote sensing data when executing the computer program stored in the memory.
The Processor may be a central processing unit (Central Processing Unit, CPU), other general purpose Processor, digital signal Processor (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), off-the-shelf Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like; the storage data area may store data (such as audio data, phonebook, etc.) created according to the use of the handset, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart memory card (SMART MEDIA CARD, SMC), secure Digital (SD) card, flash memory card (FLASH CARD), at least one disk storage device, flash memory device, or other volatile solid-state storage device.
Example 3:
A computer readable storage medium having stored thereon a computer program which when executed by a processor implements a base model adaptation method of multimodal remote sensing data classification as described in embodiment 1.
The computer readable storage medium of the present invention may be any form of storage medium readable by a processor of a computer device, including but not limited to non-volatile memory, ferroelectric memory, etc., having a computer program stored thereon, which when read and executed by the processor of the computer device, implements the steps of a multi-modal remote sensing data classification based model adaptation method described above.
The computer program comprises computer program code which may be in source code form, object code form, executable file or in some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.
The foregoing embodiments have further described the objects, technical solutions and advantageous effects of the present application in detail, and it should be understood that the foregoing embodiments are merely examples of the present application and are not intended to limit the scope of the present application, and any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the scope of the present application.
Claims (3)
1. The basic model self-adaption method for multi-mode remote sensing data classification is characterized by comprising the following steps of:
s1, acquiring multi-mode remote sensing data;
s2, preprocessing multi-mode remote sensing data, taking a pixel with a label in each mode as a center, selecting a rectangular window with the size of 27 multiplied by 27 as a sample of each single-mode data, amplifying all samples to 224 multiplied by 224 by a bilinear interpolation method for the sample of each mode data, and matching the spatial input dimension of a basic model;
S3, establishing a mapping layer, respectively sending samples of each mode data into the mapping layer to perform dimension reduction matching on channel dimensions, and fusing different single-mode data samples subjected to dimension reduction to obtain fused multi-mode data input samples;
S4, selecting a basic model based on a transducer, inputting the multi-mode remote sensing data into the basic model, and extracting general characteristics of the multi-mode data;
S5, constructing a cross-space interaction module to realize interaction of the general features and the spatial features in the multi-mode data, generating a learnable spatial coding vector set S 0 along the spatial dimension of the multi-mode remote sensing data, adding the learnable spatial coding vector set S 0 into a feature coding vector set Z 0, and outputting the spatial features;
S6, constructing a cross-channel interaction module, and adding the characteristic interaction between the general characteristic and the channel dimension in the multi-mode data in a multi-head attention mechanism MSA of an encoder module of a basic model to output a spectrum characteristic;
s7, inputting the output spatial features and the spectral features into the full-connection layer to obtain a classification result;
the specific implementation process of the step S5 is as follows:
S51, selecting hyperspectral data in multimode remote sensing data, selecting a hyperspectral data sample x HSI, and extracting spatial characteristics of the hyperspectral data sample x HSI by a convolutional neural network; CNN consists of a1×1 convolutional layer, a batch normalization layer, a ReLU activation function, and a3×3 convolutional layer, and its output spatial feature S is formulated as:
S＝CNN(xHSI) (9)
S52, dividing the spatial feature S into N non-overlapping feature blocks to form a spatial feature coding set S 0, and splicing the S 0 and a feature coding vector set Z 0, wherein the method is expressed as follows:
the specific implementation process of the step S6 is as follows:
S61, taking the average spectrum characteristic of each hyperspectral data sample x HSI as an input;
s62, inputting the spectral features into CNNs of a 1X 1 convolution layer, a batch normalization layer, a ReLU activation function and an MLP layer to output a spectral feature coding set D';
S63.D' is used to generate D Q、DK and D V that match the three matrices of query matrix Q, key value matrix K, value matrix V, and add them to Q i,Ki and V i in the multi-headed self-attention mechanism MSA in the base model;
Wherein, D Q and D K are represented by the intermediate vector of D 'for extracting the spectral feature of the feature represented by the center pixel, and D V is the same as D' and represents the global spectral feature;
s64, updating the calculation mode in the MSA:
Where D Q represents the spectral vector added to Q i, D K represents the spectral vector added to K i, and D V represents the spectral vector added to V i.
2. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor implementing the steps of a base model adaptation method of multi-modal remote sensing data classification as claimed in claim 1 when executing the computer program.
3. A computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements a method of base model adaptation for multi-modal remote sensing data classification as claimed in claim 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311580794.9A CN117611896B (en) | 2023-11-24 | 2023-11-24 | Multi-mode remote sensing data classification basic model self-adaption method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311580794.9A CN117611896B (en) | 2023-11-24 | 2023-11-24 | Multi-mode remote sensing data classification basic model self-adaption method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117611896A CN117611896A (en) | 2024-02-27 |
| CN117611896B true CN117611896B (en) | 2024-05-07 |
Family
ID=89950954
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311580794.9A Active CN117611896B (en) | 2023-11-24 | 2023-11-24 | Multi-mode remote sensing data classification basic model self-adaption method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117611896B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104021396A (en) * | 2014-06-23 | 2014-09-03 | 哈尔滨工业大学 | Hyperspectral remote sensing data classification method based on ensemble learning |
| CN114708455A (en) * | 2022-03-24 | 2022-07-05 | 中国人民解放军战略支援部队信息工程大学 | Hyperspectral image and LiDAR data collaborative classification method |
| CN116109925A (en) * | 2023-01-12 | 2023-05-12 | 中国人民解放军战略支援部队信息工程大学 | Multi-mode remote sensing image classification method based on heterogeneous feature learning network |
| CN116740474A (en) * | 2023-08-15 | 2023-09-12 | 南京信息工程大学 | Remote sensing image classification method based on anchoring stripe attention mechanism |
| CN116912642A (en) * | 2023-05-19 | 2023-10-20 | 电子科技大学 | Multimode emotion analysis method, device and medium based on dual-mode and multi-granularity interaction |
| CN116935224A (en) * | 2023-07-25 | 2023-10-24 | 哈尔滨工业大学 | Discriminative distribution self-adaptive multi-mode remote sensing image collaborative classification method based on structure maintenance |
-
2023
- 2023-11-24 CN CN202311580794.9A patent/CN117611896B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104021396A (en) * | 2014-06-23 | 2014-09-03 | 哈尔滨工业大学 | Hyperspectral remote sensing data classification method based on ensemble learning |
| CN114708455A (en) * | 2022-03-24 | 2022-07-05 | 中国人民解放军战略支援部队信息工程大学 | Hyperspectral image and LiDAR data collaborative classification method |
| CN116109925A (en) * | 2023-01-12 | 2023-05-12 | 中国人民解放军战略支援部队信息工程大学 | Multi-mode remote sensing image classification method based on heterogeneous feature learning network |
| CN116912642A (en) * | 2023-05-19 | 2023-10-20 | 电子科技大学 | Multimode emotion analysis method, device and medium based on dual-mode and multi-granularity interaction |
| CN116935224A (en) * | 2023-07-25 | 2023-10-24 | 哈尔滨工业大学 | Discriminative distribution self-adaptive multi-mode remote sensing image collaborative classification method based on structure maintenance |
| CN116740474A (en) * | 2023-08-15 | 2023-09-12 | 南京信息工程大学 | Remote sensing image classification method based on anchoring stripe attention mechanism |
Non-Patent Citations (3)
| Title |
|---|
| Peng Li等.Bi-Modal Learning With Channel-Wise Attention for Multi-Label Image Classification.IEEE Access .2020,全文. * |
| 张良培 ; 李家艺 ; .高光谱图像稀疏信息处理综述与展望.遥感学报.2016,(05),全文. * |
| 张辉 ; 刘万军 ; 吕欢欢 ; .引导滤波联合局部判别嵌入的高光谱影像分类.吉林大学学报(地球科学版).2020,(04),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117611896A (en) | 2024-02-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110378381B (en) | Object detection method, device and computer storage medium | |
| CN112446270B (en) | Training method of pedestrian re-recognition network, pedestrian re-recognition method and device | |
| Ahmad | Deep image retrieval using artificial neural network interpolation and indexing based on similarity measurement | |
| WO2020228446A1 (en) | Model training method and apparatus, and terminal and storage medium | |
| US20170011280A1 (en) | Extracting gradient features from neural networks | |
| Komorowski et al. | Minkloc++: lidar and monocular image fusion for place recognition | |
| CN112639828A (en) | Data processing method, method and equipment for training neural network model | |
| Bazi et al. | Bi-modal transformer-based approach for visual question answering in remote sensing imagery | |
| Jiang et al. | Cascaded subpatch networks for effective CNNs | |
| CN114418030B (en) | Image classification method, training method and device for image classification model | |
| CN113298096B (en) | Method, system, electronic device and storage medium for training zero sample classification model | |
| CN111695673B (en) | Method for training neural network predictor, image processing method and device | |
| KR20200144398A (en) | Apparatus for performing class incremental learning and operation method thereof | |
| Liu et al. | Iterative relaxed collaborative representation with adaptive weights learning for noise robust face hallucination | |
| Pad et al. | Efficient neural vision systems based on convolutional image acquisition | |
| CN112534445A (en) | Neural network with reduced number of parameters | |
| US20230401838A1 (en) | Image processing method and related apparatus | |
| CN111242228B (en) | Hyperspectral image classification method, hyperspectral image classification device, hyperspectral image classification equipment and storage medium | |
| CN116189265A (en) | Sketch face recognition method, device and equipment based on lightweight semantic transducer model | |
| US20200151518A1 (en) | Regularized multi-metric active learning system for image classification | |
| CN117392539B (en) | River water body identification method based on deep learning, electronic equipment and storage medium | |
| CN117611896B (en) | Multi-mode remote sensing data classification basic model self-adaption method | |
| Panda et al. | Feedback through emotion extraction using logistic regression and CNN | |
| Yu et al. | Target recognition in SAR image based on robust locality discriminant projection | |
| CN114155388B (en) | Image recognition method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |