CN117611896A - Multi-mode remote sensing data classification basic model self-adaption method - Google Patents
Multi-mode remote sensing data classification basic model self-adaption method Download PDFInfo
- Publication number
- CN117611896A CN117611896A CN202311580794.9A CN202311580794A CN117611896A CN 117611896 A CN117611896 A CN 117611896A CN 202311580794 A CN202311580794 A CN 202311580794A CN 117611896 A CN117611896 A CN 117611896A
- Authority
- CN
- China
- Prior art keywords
- remote sensing
- mode
- sensing data
- data
- basic model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 239000013598 vector Substances 0.000 claims abstract description 30
- 230000003595 spectral effect Effects 0.000 claims abstract description 27
- 230000003993 interaction Effects 0.000 claims abstract description 20
- 230000007246 mechanism Effects 0.000 claims abstract description 17
- 238000001228 spectrum Methods 0.000 claims abstract description 8
- 238000013507 mapping Methods 0.000 claims abstract description 7
- 238000007781 pre-processing Methods 0.000 claims abstract description 4
- 239000011159 matrix material Substances 0.000 claims description 27
- 238000013527 convolutional neural network Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 11
- 230000006870 function Effects 0.000 claims description 11
- 230000008569 process Effects 0.000 claims description 9
- 238000010606 normalization Methods 0.000 claims description 8
- 230000004913 activation Effects 0.000 claims description 7
- 230000009467 reduction Effects 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 230000006978 adaptation Effects 0.000 claims description 4
- 238000012545 processing Methods 0.000 abstract description 6
- 238000013135 deep learning Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000008033 biological extinction Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
- G06V10/765—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects using rules for classification or partitioning the feature space
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A40/00—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production
- Y02A40/10—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production in agriculture
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
A multi-mode remote sensing data classification basic model self-adaption method belongs to the technical field of image processing. The method solves the problem that the classification of the multi-mode remote sensing data is difficult to realize by directly using the pre-trained basic model in the prior art. The key points are as follows: acquiring multi-mode remote sensing data; preprocessing multi-mode remote sensing data; establishing a mapping layer and selecting a basic model; constructing a cross-space interaction module to realize interaction of the general features and the space features in the multi-mode data, generating a learnable space coding vector set along the space dimension of the multi-mode remote sensing data, adding the learnable space coding vector set into the feature coding vector set, and outputting the space features; constructing a cross-channel interaction module, adding the characteristic interaction between the general characteristic and the channel dimension in the multi-mode data in a multi-head attention mechanism MSA of an encoder module of a basic model, and outputting a spectrum characteristic; and inputting the spatial features and the spectral features into the full-connection layer to obtain a classification result. The method and the device improve the classification accuracy of the multi-mode remote sensing data.
Description
Technical Field
The invention relates to a multi-mode remote sensing data classification method, in particular to a basic model self-adaption method for multi-mode remote sensing data classification, and belongs to the technical field of image processing.
Background
Multimodal in the field of remote sensing generally refers to the imaging results of a scene and a target under different sensors (multispectral, hyperspectral, synthetic aperture radar, lidar, etc.). The multi-mode remote sensing data can be reasonably utilized to provide more comprehensive description information for ground objects from spectrum, time and space, so that the interpretation capability of remote sensing is improved, and the requirements of practical applications such as military reconnaissance and intelligent agriculture are met.
In order to promote the deep and wide application of the multi-mode remote sensing data in the application, an effective information processing means is needed. Classification is one of important links of remote sensing data processing technology, and is a hot spot problem of current research. In recent years, deep learning has become a mainstream of multi-modal remote sensing data classification due to its strong feature extraction capability. Among the many deep learning methods, the transform-based multi-modal remote sensing data method has attracted a great deal of attention. The self-attention mechanism in the transducer enables it to focus on different modalities simultaneously and capture long-range dependencies, but the classification accuracy of these methods still needs to be improved.
With the development of the deep learning method, a neural network, namely a basic model, which is trained by unsupervised learning on the basis of a large amount of original data emerges. On the downstream task, better classification performance can be obtained by only a small amount of marking data for the basic model. However, the data set used for training the basic model and the multi-modal data have large differences, so that it is difficult to directly use the pre-trained basic model to realize the classification of the multi-modal remote sensing data.
Disclosure of Invention
In order to solve the problem that the pre-trained basic model is difficult to directly use to realize the classification of the multi-mode remote sensing data in the prior art, the invention explores the potential and the effectiveness of the basic model in the multi-mode remote sensing data classification task and further provides a multi-mode remote sensing data classification basic model self-adaption method, and the classification effect can be improved without fine adjustment of basic model parameters. The following presents a simplified summary of the invention in order to provide a basic understanding of some aspects of the invention. It should be understood that this summary is not an exhaustive overview of the invention. It is not intended to identify key or critical elements of the invention or to delineate the scope of the invention.
The technical scheme of the invention is as follows:
a multi-mode remote sensing data classification basic model self-adaption method comprises the following steps:
s1, acquiring multi-mode remote sensing data;
s2, preprocessing multi-mode remote sensing data, taking a pixel with a label in each mode as a center, selecting a rectangular window with the size of 27 multiplied by 27 as a sample of each single-mode data, amplifying all samples to 224 multiplied by 224 by a bilinear interpolation method for the sample of each mode data, and matching the spatial input dimension of a basic model;
s3, establishing a mapping layer, respectively sending samples of each mode data into the mapping layer to perform dimension reduction matching on channel dimensions, and fusing different single-mode data samples subjected to dimension reduction to obtain fused multi-mode data input samples;
s4, selecting a basic model based on a transducer, inputting the multi-mode remote sensing data into the basic model, and extracting general characteristics of the multi-mode data;
s5, constructing a cross-space interaction module to realize interaction of the general features and the space features in the multi-mode data, and generating a learnable space coding vector set S along the space dimension of the multi-mode remote sensing data 0 And adds it to the set of feature encoding vectors Z 0 Outputting the spatial characteristics;
s6, constructing a cross-channel interaction module, and adding the characteristic interaction between the general characteristic and the channel dimension in the multi-mode data in a multi-head attention mechanism MSA of an encoder module of a basic model to output a spectrum characteristic;
s7, inputting the output spatial features and the spectral features into the full-connection layer to obtain a classification result.
Further: the specific implementation process of the step S5 is as follows:
s51, selecting hyperspectral data in multimode remote sensing data, and selecting hyperspectral data samples x HSI Extracting the spatial characteristics of the convolutional neural network; CNN consists of a 1×1 convolutional layer, a batch normalization layer, a ReLU activation function, and a 3×3 convolutional layer, and its output spatial feature S is formulated as:
S=CNN(x HSI ) (9)
s52, dividing the spatial feature S into N non-overlapping feature blocks to form a spatial feature coding set S 0 Will S 0 And feature encoding vector set Z 0 Splicing is performed as follows:
further: the specific implementation process of the step S6 is as follows:
s61, each hyperspectral data sample x HSI As input the average spectral features of (a);
s62, inputting the spectral features into CNNs of a 1X 1 convolution layer, a batch normalization layer, a ReLU activation function and an MLP layer to output a spectral feature coding set D';
s63.D' is used to generate D matching three matrices, namely, query matrix Q, key value matrix K, and value matrix V Q 、D K And D V And combine it with the base modelQ in multi-head self-attention mechanism MSA in (B) i ,K i And V i Adding;
wherein D is Q And D K Represented by the intermediate vector of D' for extracting the spectral features of the feature represented by the center pixel, D V As with D', represents global spectral features;
s64, updating the calculation mode in the MSA:
wherein D is Q The representation is added to Q i Spectral vector, D K The representation is added to K i Spectral vector, D V The representation is added to V i Is a spectrum vector of (a).
The beneficial effects of the invention are as follows:
compared with the prior art, the basic model self-adaptive framework for multi-mode remote sensing data classification provided by the invention fuses the general features, the spatial features and the relations among the spectral features extracted by all multi-mode basic models only by adjusting the cross-space interaction module and the cross-channel interaction module, simultaneously freezes the parameters of the basic models, retains strong general knowledge extracted by the basic models, finally reasonably utilizes the pre-trained parameters of the basic models, and fuses the unique features of the multi-mode remote sensing data such as space, spectrum and the like to improve the classification precision of the multi-mode remote sensing data.
Drawings
FIG. 1 is a schematic diagram of a multi-modal remote sensing data classification method based on basic model fine tuning as described in example 1;
FIG. 2 is a schematic diagram of a cross-space interaction module in a basic model adaptive method for multi-modal remote sensing data classification according to embodiment 1;
fig. 3 is a schematic diagram of a cross-channel interaction module in a basic model adaptive method for multi-modal remote sensing data classification according to embodiment 1.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the present invention is described below by means of specific embodiments shown in the accompanying drawings. It should be understood that the description is only illustrative and is not intended to limit the scope of the invention. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the present invention. It should be further understood that the terms "first," "second," "third," and the like in this specification are used merely for distinguishing between various components, elements, steps, etc. in the specification and not for indicating a logical or sequential relationship between the various components, elements, steps, etc., unless otherwise indicated.
Embodiment 1, which is described with reference to fig. 1-3, provides a method for adapting a base model of multi-modal remote sensing data classification, comprising the following steps:
s1, acquiring multi-mode remote sensing data;
s2, preprocessing multi-mode remote sensing data, taking a pixel with a label in each mode as a center, selecting a rectangular window with the size of 27 multiplied by 27 as a sample of each single-mode data, amplifying all samples to 224 multiplied by 224 by a bilinear interpolation method for the sample of each mode data, and matching the spatial input dimension of a basic model;
s3, establishing a mapping layer, respectively sending samples of each mode data into the mapping layer to perform dimension reduction matching on channel dimensions, and fusing different single-mode data samples subjected to dimension reduction to obtain fused multi-mode data input samples;
s4, selecting a basic model based on a transducer, such as DINov2, inputting the multi-mode remote sensing data into the basic model to extract the general characteristics of the multi-mode data, such as the characteristics of edges, angles and the like; the specific implementation process of the steps is as follows:
s41, dividing the fused multi-mode input sample into N sample blocks in a non-overlapping mode to form an input sequence of X= [ X ] 1 ,x 2 ,…,x N ];
S42, each sample block is sent into a linear projection layer to perform linear transformation, and N characteristics are outputCoding vector Z n To obtain global information of an input sequence, a learnable feature code vector Z cls Added to Z n Forming a feature code vector set Z 0 ;
S43, adding the feature coding vector set and the position coding vectors, and inputting the feature coding vector set and the position coding vectors into L encoders; in the encoder, the core module is a multi-head (MS) attention mechanism, defining a query weight matrix W Q Key weight matrix W K Value weight matrix W V The method comprises the steps of carrying out a first treatment on the surface of the X obtains three matrixes of a query matrix Q, a key matrix K and a value matrix V according to three different weight matrixes, and a calculation formula is shown in a formula (1):
Q=XW Q ,K=XW K ,V=XW V (1)
s44, inquiring the matrix Q and the value matrix V through matrix multiplication QK T Calculating the similarity among all elements in X to obtain a similarity matrix;
the similarity matrix is processed through a Softmax activation function and is calculated as follows:
wherein z is i Representing input elements, wherein C represents the number of the input elements, and the similarity matrix is processed through a Softmax function to obtain a normalized result, wherein the larger a certain value is, the higher the similarity among the elements is;
s45, performing matrix multiplication on the similarity matrix and V to obtain an output result of the self-attention mechanism, wherein the calculation formula is as follows:
wherein d k Representing the dimension of the input X.
In order to enrich the extracted features of the self-attention mechanism and avoid sinking into certain local features, a transducer introduces a multi-head attention mechanism on the basis of a single-head self-attention mechanism. The multi-headed self-attention mechanism is an attention mechanism that parallelizes the scaled dot product attention mechanism, the number of parallelism being called the "head count", so that the model focuses on multiple subspace information simultaneously. The process of the multi-head self-attention mechanism MSA is expressed as:
Z i =Attention(Q i ,K i ,V i ),i=1...h (5)
MS(Q,K,V)=Concat(Z 1 ,Z 2 ,...,Z h )W o (6)
wherein i represents the number of the header,representing the output projection matrix, Z i Representing the output matrix of each head, +.>A key matrix representing the ith head, +.>Representing the value matrix of the i-th header.
In each encoder, the MSA later connects the layer normalization (Layer Normalization, LN) layer and the multi-layer perceptron (Multilayer perceptron, MLP), and adds a residual connection after each module, preventing gradient extinction, the process being expressed as follows:
H l-1 =MSA(LN(Z l-1 ))+Z l-1 ,l=0,...,L-1 (7)
Z l =MLP(LN(H l-1 ))+H l-1 ,l=0,...,L-1 (8)。
s5, constructing a cross-space interaction module to realize interaction of the general features and the space features in the multi-mode data, and generating a learnable space coding vector set S along the space dimension of the multi-mode remote sensing data 0 And adds it to the set of feature encoding vectors Z 0 In which the spatial signature is output. The specific implementation process of the steps is as follows:
s51, selecting hyperspectral data in multimode remote sensing data, and selecting hyperspectral data samples x HSI And its spatial features are extracted by convolutional neural networks (Convolutional Neural Network, CNN). CNN consists of a 1×1 convolutional layer, a batch normalization layer, a ReLU activation function, and a 3×3 convolutional layer, and its output spatial feature S is formulated as:
S=CNN(x HSI ) (9)
s52, dividing the spatial feature S into N non-overlapping feature blocks to form a spatial feature coding set S 0 S based on the formula (7) and the formula (8) 0 And Z is 0 Splicing is performed as follows:
s6, constructing a cross-channel interaction module, and adding the characteristic interaction between the general characteristic and the channel dimension in the multi-mode data in a multi-head attention mechanism MSA of an encoder module of the basic model to output the spectral characteristic. The specific implementation process of the steps is as follows:
s61, each hyperspectral data sample x HSI As input the average spectral features of (a);
s62, inputting the spectral features into CNNs of a 1X 1 convolution layer, a batch normalization layer, a ReLU activation function and an MLP layer to output a spectral feature coding set D';
s63.D' is used to generate D matching three matrices, namely, query matrix Q, key value matrix K, and value matrix V Q 、D K And D V And combine it with Q in the multi-headed self-attention mechanism MSA in the base model i ,K i And V i Adding;
wherein D is Q And D K Represented by the intermediate vector of D' for extracting the spectral features of the feature represented by the center pixel, D V As with D', represents global spectral features;
s64, updating the calculation mode in the MSA:
wherein, D is Q The representation is added to Q i Spectral vector, D K The representation is added to K i Spectral vector, D V The representation is added to V i Is a spectrum vector of (a).
S7, inputting the output spatial features and the spectral features into the full-connection layer to obtain a classification result.
Example 2:
an electronic device comprising a memory and a processor, the memory storing a computer program, the processor implementing the steps of the base model adaptation method of multimodal remote sensing data classification of embodiment 1 when executing the computer program.
The computer device of the present invention may be a device including a processor and a memory, such as a single chip microcomputer including a central processing unit. And the processor is used for realizing the steps of the basic model self-adaption method for classifying the multi-mode remote sensing data when executing the computer program stored in the memory.
The processor may be a central processing unit (Central Processing Unit, CPU), other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like; the storage data area may store data (such as audio data, phonebook, etc.) created according to the use of the handset, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart Media Card (SMC), secure Digital (SD) Card, flash Card (Flash Card), at least one disk storage device, flash memory device, or other volatile solid-state storage device.
Example 3:
a computer readable storage medium having stored thereon a computer program which when executed by a processor implements a base model adaptation method of multimodal remote sensing data classification as described in embodiment 1.
The computer readable storage medium of the present invention may be any form of storage medium readable by a processor of a computer device, including but not limited to non-volatile memory, ferroelectric memory, etc., having a computer program stored thereon, which when read and executed by the processor of the computer device, implements the steps of a multi-modal remote sensing data classification based model adaptation method described above.
The computer program comprises computer program code which may be in source code form, object code form, executable file or in some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.
The foregoing embodiments have further been described in detail for the purpose, technical solutions and advantageous effects of the present application, and it should be understood that the foregoing embodiments are merely examples of the present application, and are not intended to limit the scope of the present application, and any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the scope of protection of the present application.
Claims (5)
1. The basic model self-adaption method for multi-mode remote sensing data classification is characterized by comprising the following steps of:
s1, acquiring multi-mode remote sensing data;
s2, preprocessing multi-mode remote sensing data, taking a pixel with a label in each mode as a center, selecting a rectangular window with the size of 27 multiplied by 27 as a sample of each single-mode data, amplifying all samples to 224 multiplied by 224 by a bilinear interpolation method for the sample of each mode data, and matching the spatial input dimension of a basic model;
s3, establishing a mapping layer, respectively sending samples of each mode data into the mapping layer to perform dimension reduction matching on channel dimensions, and fusing different single-mode data samples subjected to dimension reduction to obtain fused multi-mode data input samples;
s4, selecting a basic model based on a transducer, inputting the multi-mode remote sensing data into the basic model, and extracting general characteristics of the multi-mode data;
s5, constructing a cross-space interaction module to realize interaction of the general features and the space features in the multi-mode data, and generating a learnable space coding vector set S along the space dimension of the multi-mode remote sensing data 0 And adds it to the set of feature encoding vectors Z 0 Outputting the spatial characteristics;
s6, constructing a cross-channel interaction module, and adding the characteristic interaction between the general characteristic and the channel dimension in the multi-mode data in a multi-head attention mechanism MSA of an encoder module of a basic model to output a spectrum characteristic;
s7, inputting the output spatial features and the spectral features into the full-connection layer to obtain a classification result.
2. The method for adapting a base model for classifying multimodal remote sensing data as recited in claim 1, wherein: the specific implementation process of the step S5 is as follows:
s51, selecting hyperspectral data in multimode remote sensing data, and selecting hyperspectral data samples x HSI Extracting the spatial characteristics of the convolutional neural network; CNN consists of a 1×1 convolutional layer, a batch normalization layer, a ReLU activation function, and a 3×3 convolutional layer, and its output spatial feature S is formulated as:
S=CNN(x HSI ) (9)
s52, dividing the spatial feature S into N non-overlapping feature blocks to form a spatial feature coding set S 0 Will S 0 And feature encoding vector set Z 0 Splicing is performed as follows:
3. the method for adapting a base model for classifying multimodal remote sensing data as claimed in claim 2, wherein: the specific implementation process of the step S6 is as follows:
s61, each hyperspectral data sample x HSI As input the average spectral features of (a);
s62, inputting the spectral features into CNNs of a 1X 1 convolution layer, a batch normalization layer, a ReLU activation function and an MLP layer to output a spectral feature coding set D';
s63.D' is used to generate D matching three matrices, namely, query matrix Q, key value matrix K, and value matrix V Q 、D K And D V And combine it with Q in the multi-headed self-attention mechanism MSA in the base model i ,K i And V i Adding;
wherein D is Q And D K Represented by the intermediate vector of D' for extracting the spectral features of the feature represented by the center pixel, D V As with D', represents global spectral features;
s64, updating the calculation mode in the MSA:
wherein D is Q The representation is added to Q i Spectral vector, D K The representation is added to K i Spectral vector, D V The representation is added to V i Is a spectrum vector of (a).
4. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor implementing the steps of a method of adapting a base model of multi-modal remote sensing data classification as claimed in any one of claims 1 to 3 when the computer program is executed.
5. A computer readable storage medium having stored thereon a computer program, which when executed by a processor implements a base model adaptation method of multimodal remote sensing data classification as claimed in any one of claims 1-3.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311580794.9A CN117611896B (en) | 2023-11-24 | 2023-11-24 | Multi-mode remote sensing data classification basic model self-adaption method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311580794.9A CN117611896B (en) | 2023-11-24 | 2023-11-24 | Multi-mode remote sensing data classification basic model self-adaption method |
Publications (2)
Publication Number | Publication Date |
---|---|
CN117611896A true CN117611896A (en) | 2024-02-27 |
CN117611896B CN117611896B (en) | 2024-05-07 |
Family
ID=89950954
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202311580794.9A Active CN117611896B (en) | 2023-11-24 | 2023-11-24 | Multi-mode remote sensing data classification basic model self-adaption method |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN117611896B (en) |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104021396A (en) * | 2014-06-23 | 2014-09-03 | 哈尔滨工业大学 | Hyperspectral remote sensing data classification method based on ensemble learning |
CN114708455A (en) * | 2022-03-24 | 2022-07-05 | 中国人民解放军战略支援部队信息工程大学 | Hyperspectral image and LiDAR data collaborative classification method |
CN116109925A (en) * | 2023-01-12 | 2023-05-12 | 中国人民解放军战略支援部队信息工程大学 | Multi-mode remote sensing image classification method based on heterogeneous feature learning network |
CN116740474A (en) * | 2023-08-15 | 2023-09-12 | 南京信息工程大学 | Remote sensing image classification method based on anchoring stripe attention mechanism |
CN116912642A (en) * | 2023-05-19 | 2023-10-20 | 电子科技大学 | Multimode emotion analysis method, device and medium based on dual-mode and multi-granularity interaction |
CN116935224A (en) * | 2023-07-25 | 2023-10-24 | 哈尔滨工业大学 | Discriminative distribution self-adaptive multi-mode remote sensing image collaborative classification method based on structure maintenance |
-
2023
- 2023-11-24 CN CN202311580794.9A patent/CN117611896B/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104021396A (en) * | 2014-06-23 | 2014-09-03 | 哈尔滨工业大学 | Hyperspectral remote sensing data classification method based on ensemble learning |
CN114708455A (en) * | 2022-03-24 | 2022-07-05 | 中国人民解放军战略支援部队信息工程大学 | Hyperspectral image and LiDAR data collaborative classification method |
CN116109925A (en) * | 2023-01-12 | 2023-05-12 | 中国人民解放军战略支援部队信息工程大学 | Multi-mode remote sensing image classification method based on heterogeneous feature learning network |
CN116912642A (en) * | 2023-05-19 | 2023-10-20 | 电子科技大学 | Multimode emotion analysis method, device and medium based on dual-mode and multi-granularity interaction |
CN116935224A (en) * | 2023-07-25 | 2023-10-24 | 哈尔滨工业大学 | Discriminative distribution self-adaptive multi-mode remote sensing image collaborative classification method based on structure maintenance |
CN116740474A (en) * | 2023-08-15 | 2023-09-12 | 南京信息工程大学 | Remote sensing image classification method based on anchoring stripe attention mechanism |
Non-Patent Citations (3)
Title |
---|
PENG LI等: "Bi-Modal Learning With Channel-Wise Attention for Multi-Label Image Classification", IEEE ACCESS, 7 January 2020 (2020-01-07) * |
张良培;李家艺;: "高光谱图像稀疏信息处理综述与展望", 遥感学报, no. 05, 25 September 2016 (2016-09-25) * |
张辉;刘万军;吕欢欢;: "引导滤波联合局部判别嵌入的高光谱影像分类", 吉林大学学报(地球科学版), no. 04, 26 July 2020 (2020-07-26) * |
Also Published As
Publication number | Publication date |
---|---|
CN117611896B (en) | 2024-05-07 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110378381B (en) | Object detection method, device and computer storage medium | |
US20210326656A1 (en) | Panoptic segmentation | |
CN112446270B (en) | Training method of pedestrian re-recognition network, pedestrian re-recognition method and device | |
Ahmad | Deep image retrieval using artificial neural network interpolation and indexing based on similarity measurement | |
US9792492B2 (en) | Extracting gradient features from neural networks | |
CN108681746B (en) | Image identification method and device, electronic equipment and computer readable medium | |
KR20210036244A (en) | System and method for boundary aware semantic segmentation | |
Komorowski et al. | Minkloc++: lidar and monocular image fusion for place recognition | |
Bazi et al. | Bi-modal transformer-based approach for visual question answering in remote sensing imagery | |
CN112639828A (en) | Data processing method, method and equipment for training neural network model | |
Du et al. | Learning signal-agnostic manifolds of neural fields | |
Jiang et al. | Cascaded subpatch networks for effective CNNs | |
CN111695673B (en) | Method for training neural network predictor, image processing method and device | |
CN114418030B (en) | Image classification method, training method and device for image classification model | |
CN117079139B (en) | Remote sensing image target detection method and system based on multi-scale semantic features | |
Liu et al. | Iterative relaxed collaborative representation with adaptive weights learning for noise robust face hallucination | |
CN114549913A (en) | Semantic segmentation method and device, computer equipment and storage medium | |
CN113298096A (en) | Method, system, electronic device and storage medium for training zero sample classification model | |
Pad et al. | Efficient neural vision systems based on convolutional image acquisition | |
US20230401838A1 (en) | Image processing method and related apparatus | |
CN117131376A (en) | Hyperspectral cross-domain robust anomaly detection method, system, equipment and medium for continuous learning based on visual transformation combined generation countermeasure network | |
CN116189265A (en) | Sketch face recognition method, device and equipment based on lightweight semantic transducer model | |
CN117392539B (en) | River water body identification method based on deep learning, electronic equipment and storage medium | |
CN111242228A (en) | Hyperspectral image classification method, device, equipment and storage medium | |
CN117611896B (en) | Multi-mode remote sensing data classification basic model self-adaption method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |