CN113240017A - Multispectral and panchromatic image classification method based on attention mechanism - Google Patents
Multispectral and panchromatic image classification method based on attention mechanism Download PDFInfo
- Publication number
- CN113240017A CN113240017A CN202110542410.9A CN202110542410A CN113240017A CN 113240017 A CN113240017 A CN 113240017A CN 202110542410 A CN202110542410 A CN 202110542410A CN 113240017 A CN113240017 A CN 113240017A
- Authority
- CN
- China
- Prior art keywords
- multispectral
- panchromatic
- sample
- samples
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A40/00—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production
- Y02A40/10—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production in agriculture
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Evolutionary Biology (AREA)
- Probability & Statistics with Applications (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a multispectral and panchromatic image classification method based on an attention mechanism, and belongs to the technical field of image processing. Performing initial feature extraction on the multispectral samples and the panchromatic samples in the training sample set by constructing the training sample set and the testing sample set to obtain initial features of the multispectral samples and initial features of the panchromatic samples with the same feature size; continuously and respectively inputting the deep features into a lightweight sharing-separating network to extract the deep features, and respectively obtaining the deep features of the multispectral sample and the deep features of the full-color sample; continuing to respectively perform self-adaptive feature fusion and outputting fusion features; classifying the fusion characteristics, and training the network according to the classification result; and obtaining classification results of the multispectral samples and the panchromatic samples in the training sample set based on the trained network. The problem of categorised time cost, multisensor fuse the effect not good among the prior art is solved.
Description
Technical Field
The invention belongs to the technical field of image processing, and relates to a multispectral and panchromatic image classification method based on an attention mechanism.
Background
The rapid development of aerospace technology generates a large number of remote sensing images of different sensors, so that the effective classification of ground objects by combining a plurality of sensor images becomes a hot point of research, particularly the classification research of multispectral and full-color images. The remote sensing satellite generally has a panchromatic sensor which can perform spectral response on a large range of spectrums to form a full-color image. The full-color image is a gray-scale image and has high spatial resolution, but because of only one spectral band, the spectral resolution is low, the type of the ground object cannot be determined, and the identification of the type of the ground object is extremely disadvantageous. In order to compensate for the deficiency of full-color images, a satellite is generally equipped with a multispectral sensor (red, green, blue, near infrared, etc. are common). Due to physical device limitations, multispectral sensors have high spectral resolution, but low spatial resolution. Therefore, how to fully utilize the complementary features of the multispectral and panchromatic images to improve the classification accuracy is an urgent problem to be solved.
Methods of classification combining multispectral and panchromatic images generally fall into two categories: the first type is that a full-color image is utilized to improve the spatial resolution of a multispectral image to obtain a fused multispectral image, and then the fused multispectral image is classified; the second category is to extract the features of the multispectral and panchromatic images respectively and then perform fusion classification on the extracted features. The first category focuses more on how to obtain an effective fused image, while the second category focuses more on how to effectively extract features. In recent years, in order to improve the classification accuracy, a deep neural network has achieved a desirable effect in the classification of remote sensing images. Several typical deep learning networks, such as a stack type automatic encoder, a convolutional neural network, a generative countermeasure network, and a recursive neural network, have been widely used in remote sensing image classification. At present, multispectral and panchromatic image classification methods based on a deep neural network mostly belong to a second class of methods, namely: and respectively extracting deep features of the multispectral image and the full-color image by adopting a deep neural network, and fusing and classifying the deep features of the multispectral image and the full-color image in a stacked connection mode. This type of method is simple and effective, but also has certain limitations: (1) extracting the features of the multispectral and panchromatic images requires two separate networks to complete, but training the two separate networks incurs a large time penalty. (2) The simple stacked feature fusion method considers that the contribution degree of the multispectral image and the panchromatic image to the classification is the same, and neglects the problem that the contribution of different images to the classification is unequal.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention aims to provide a multispectral and panchromatic image classification method based on an attention mechanism, which solves the problems of poor classification time cost and poor multi-sensor fusion effect in the prior art, and balances the classification precision and time cost of multispectral and panchromatic images.
In order to achieve the purpose, the invention adopts the following technical scheme to realize the purpose:
the invention discloses a multispectral and panchromatic image classification method based on an attention mechanism, which comprises the following steps of:
respectively constructing a training sample set and a test sample set which are composed of multispectral samples and panchromatic samples, and carrying out initial feature extraction on the multispectral samples and the panchromatic samples in the training sample set to respectively obtain initial features of the multispectral samples and initial features of the panchromatic samples with the same feature size;
constructing a lightweight sharing-separating network; respectively inputting the initial characteristics of the multispectral samples and the initial characteristics of the panchromatic samples in the training sample set obtained in the step one into a lightweight sharing-separating network, and carrying out deep characteristic extraction to respectively obtain the deep characteristics of the multispectral samples and the deep characteristics of the panchromatic samples;
step three, respectively carrying out self-adaptive feature fusion on the deep features of the multispectral sample and the deep features of the full-color sample obtained in the step two, and outputting fusion features;
step four, classifying the fusion characteristics obtained by the output of the step three, and training the network according to the classification result; based on the trained network, obtaining classification results of multispectral samples and panchromatic samples in a training sample set; and completing the classification method of the multispectral image and the panchromatic image based on the attention mechanism.
Preferably, in the first step, constructing a training sample set and a testing sample set composed of multispectral samples and panchromatic samples, and performing initial feature extraction on the multispectral samples and the panchromatic samples in the training sample set, the method includes the following steps:
according to each pixel with a class mark on the multispectral image (part of pixels are not marked), respectively extracting multispectral samples and panchromatic samples in pairs on the multispectral image and the panchromatic image to form a sample pair of the pixels with the class mark; forming a sample set according to the obtained sample pairs; selecting part of sample pairs in the sample set to form a training sample set, and selecting the rest sample pairs in the sample set to form a test sample set;
constructing a multispectral image initial feature extraction network consisting of 2 convolutional layers and 2 downsampling layers, and initializing parameters of each layer; taking multispectral samples in the training sample set as input of a multispectral image initial feature extraction network, and obtaining initial features of the multispectral samples through forward propagation;
constructing a full-color image initial feature extraction network consisting of 3 convolutional layers and 3 downsampling layers, and initializing parameters of each layer; and taking the panchromatic sample in the training sample set as the input of the panchromatic image initial characteristic extraction network, and obtaining the initial characteristic of the panchromatic sample through forward propagation.
Further preferably, the specific network design of the multispectral sample initial feature extraction network is as follows: the convolution kernel size of the first convolution layer is 3 x 3, the output characteristic number is 32, the convolution kernel size of the second convolution layer is 3 x 3, the output characteristic number is 64, the maximum down-sampling method is adopted by all down-sampling layers, and the sampling kernel is 2 x 2.
Further preferably, the specific network design of the panchromatic sample initial feature extraction network is as follows: the convolution kernel size of the first convolution layer is 3 x 3, the output characteristic number is 16, the convolution kernel size of the second convolution layer is 3 x 3, the output characteristic number is 32, the convolution kernel size of the second convolution layer is 3 x 3, the output characteristic number is 64, the maximum down-sampling method is adopted by the down-sampling layers, and the sampling kernel is 2 x 2.
Preferably, in step two, constructing a lightweight shared-split network includes the following steps:
building a sharing sub-network consisting of 1 convolutional layer and 1 down-sampling layer, and initializing the parameters of each layer;
respectively building a spectrum separation sub-network and a panchromatic separation sub-network; the spectrum separation sub-network and the panchromatic separation sub-network have the same network structure and respectively comprise a compression part and an excitation part;
the compression is to perform global average operation on input features to obtain global compressed feature vectors of the input features, the excitation is to input the global compressed feature vectors into 2 full-connection layers, output the global compressed feature vectors to obtain a weight of each channel in initial features, finally perform linear weighted summation on each channel of the input features by using the obtained weights, and use the features of the linear weighted summation as the input of a next layer of network.
Further preferably, the compressing is to perform global average operation on the input features to obtain global compressed feature vector z of the input featurescWherein the global averaging operation is according to the following formula:
wherein f isc(i, j) represents the feature value of the input feature in the ith row and jth column, and H and W represent the height and width of the input feature, respectively;
the excitation is to compress the global feature vector zcInputting into 2 full-connection layers, wherein the characteristic number of the first full-connection layer is 4And finally, performing linear weighted summation on each channel of the input features by using the obtained weights, and taking the features subjected to the linear weighted summation as the input of the next layer of network.
Further preferably, in the second step, the deep layer feature extraction is performed by the following specific steps:
firstly, inputting the initial characteristics of the multispectral sample and the initial characteristics of the full-color sample into a constructed sharing sub-network, and respectively outputting the sharing characteristics of the multispectral sample and the sharing characteristics of the full-color sample after forward propagation;
then, inputting the shared characteristics of the obtained multispectral samples into a built spectral separation sub-network, and outputting the separation characteristics of the multispectral samples through forward propagation; inputting the shared characteristics of the obtained panchromatic sample into a constructed panchromatic separation sub-network, and outputting the separation characteristics of the panchromatic sample through forward propagation;
and finally, taking the separation characteristics of the output multispectral sample and the separation characteristics of the full-color sample as the input of the constructed sharing sub-network, and circulating the steps for three times to respectively obtain the deep characteristics of the multispectral sample and the deep characteristics of the full-color sample.
Preferably, in step three, the specific steps of performing adaptive feature fusion are as follows:
adding the deep features of the obtained multispectral sample and the deep features of the full-color sample to obtain an initial fusion feature;
performing global compression and excitation on the obtained initial fusion features to respectively obtain fusion weights of the multispectral samples and fusion weights of the panchromatic samples; carrying out linear weighted summation on the fusion weight of the obtained multispectral sample and the deep features of the obtained multispectral sample according to the channel to obtain the linear weighted summation features of the multispectral sample; carrying out linear weighted summation on the fusion weight of the obtained panchromatic sample and the deep layer characteristics of the panchromatic sample according to the channel to obtain the linear weighted summation characteristics of the panchromatic sample; and adding the linear weighted summation characteristics of the multispectral samples and the linear weighted summation characteristics of the panchromatic samples, and outputting the self-adaptive fusion characteristics.
Preferably, in the fourth step, the network is trained according to the classification result; based on the trained network, obtaining classification results of multispectral samples and panchromatic samples in a training sample set, comprising the following steps:
inputting the obtained fusion characteristics into a softmax classifier for classification, and outputting the probability of each sample pair in the training sample set belonging to each class;
constructing a cross entropy loss function according to the output probability and the class mark of the sample pair; a gradient descent method is adopted, the constructed cross entropy loss function is propagated reversely, and all networks involved in the steps 1 to 4 are trained; repeating the network training in the steps 1 to 4 until a preset iteration step is reached, and stopping iteration;
and (3) taking the multispectral samples and the panchromatic samples in the test sample set as the input of the step (1), executing the step (1) to the step (4), outputting the probability of each sample pair in the test sample set belonging to each class, and setting the position corresponding to the maximum value of the probability value as the class mark of the sample pair in the test sample set to finish the classification.
Compared with the prior art, the invention has the following beneficial effects:
the invention discloses a multispectral and panchromatic image classification method based on an attention mechanism, which can effectively extract the characteristic difference of different sensors by capturing the correlation of characteristic channels; by the design of the lightweight sharing-separating network, the problem of high calculation cost in the traditional multispectral and panchromatic image classification method is solved, and the training efficiency of the network is improved; by considering the differences of the contributions of different sensors to the classification results and introducing an attention mechanism, the self-adaptive feature fusion network is realized, the problem that the traditional feature fusion method does not consider the unbalanced contribution of input images to classification is solved, the fusion effect is improved, and the classification precision is further improved. Therefore, the multispectral and panchromatic image classification method based on the attention mechanism improves the effect of feature fusion by designing a lightweight network structure so as to better balance time cost and classification effect.
Drawings
FIG. 1 is a block flow diagram of a method for multi-spectral and panchromatic image classification based on an attention mechanism in accordance with the present invention;
FIG. 2 is a data set for experimental use of the present invention wherein (a) is a Western suburban multispectral and panchromatic image data set and (b) is a Western urban multispectral and panchromatic image data set;
FIG. 3 is a classification diagram of different approaches on a rural area data set; wherein, (a) is EMAP method, (b) is CAE method, (c) is RNN method, (d) is SCPF-ResNet method, (e) is CNN-MS method, (f) is CNN-PAN method, (g) is SFNet method, (h) is the method of the invention;
FIG. 4 is a classification diagram of different approaches on the city of Western Ann data set; wherein, (a) is EMAP method, (b) is CAE method, (c) is RNN method, (d) is SCPF-ResNet method, (e) is CNN-MS method, (f) is CNN-PAN method, (g) is SFNet method, (h) is the method of the invention;
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The technical solution and effects of the present invention will be described in further detail with reference to the accompanying drawings.
Referring to fig. 1, the implementation steps of the invention are as follows:
step 1, inputting a multispectral and full-color image.
A multispectral image and a panchromatic image are input, respectively, as shown in fig. 2.
And 2, acquiring a training sample set and a testing sample set.
In step 2, multispectral samples and panchromatic samples are respectively extracted in pairs on the multispectral image and the panchromatic image to form a sample pair, and the steps are carried out as follows:
for each labeled pixel on the multispectral image:
2a) defining a space window with the size of 32 multiplied by 32 on the multispectral image by taking the pixel as the center to obtain a multispectral sample;
2b) multiplying the position of the pixel on the multispectral image by 4, mapping the pixel to the corresponding position on the panchromatic image, and defining a space window with the size of 128 multiplied by 128 on the panchromatic image by taking the pixel on the mapped position as the center to obtain a panchromatic sample corresponding to the multispectral sample, wherein the panchromatic sample is the same as the multispectral sample in class mark;
2c) forming the multispectral samples and panchromatic samples into sample pairs;
all the sample pairs form a sample set, and 100 sample pairs are randomly selected for each class in the sample set to form a training sample set; forming the rest samples into a test sample set;
and 3, extracting initial features.
3a) Multispectral sample initial feature extraction:
3a1) constructing a multispectral image initial feature extraction network consisting of 2 convolutional layers and 2 down-sampling layers, wherein the size of convolution kernels of the first convolutional layer is 3 multiplied by 3, the number of output features is 32, the size of convolution kernels of the second convolutional layer is 3 multiplied by 3, the number of output features is 64, the down-sampling layers adopt a maximum down-sampling method, and the sampling kernels are 2 multiplied by 2;
3a2) taking multispectral samples in the training sample pair as input of a multispectral image initial feature extraction network, and obtaining initial features of the multispectral samples through forward propagation;
3b) extracting initial features of a panchromatic sample:
3b1) constructing a full-color image initial feature extraction network consisting of 3 convolutional layers and 3 down-sampling layers, wherein the size of a convolution kernel of the first convolutional layer is 3 multiplied by 3, the number of output features is 16, the size of a convolution kernel of the second convolutional layer is 3 multiplied by 3, the number of output features is 32, the size of a convolution kernel of the second convolutional layer is 3 multiplied by 3, the number of output features is 64, the down-sampling layers adopt a maximum down-sampling method, and the sampling kernel is 2 multiplied by 2;
3b2) taking a panchromatic sample corresponding to the training sample pair as the input of a panchromatic image initial characteristic extraction network, and carrying out forward propagation to obtain the initial characteristic of the panchromatic sample;
and 4, building a sharing-separating network.
4a) Building a sharing sub-network: the sharing sub-network consists of 1 convolution layer and a down-sampling layer, wherein the convolution kernel size of the convolution layer is 3 multiplied by 3, the output characteristic number is 64, the maximum down-sampling method is adopted, and the sampling kernel is 2 multiplied by 2;
4b) respectively building a spectrum separation sub-network and a panchromatic separation sub-network: each separation sub-network has the same network structure, the separation sub-network is divided into a compression part and an excitation part, and the network structures are respectively as follows:
4b1) compression is on the input features fcGlobal compressed feature vector z of input feature is obtained by global average operationcWherein the global averaging operation is according to the following formula:
wherein f isc(i, j) represents the feature value of the input feature in the ith row and jth column, and H and W represent the height and width of the input feature, respectively.
4b2) Excitation is to input global compressed feature vectors into 2 full-connection layers, wherein the feature number of the first full-connection layer is 4, the feature number of the second full-connection layer is 64, a 64-dimensional weight vector is output, the vector dimension is the weight of each channel in the initial features obtained in the step 3, finally, the obtained weight is used for carrying out linear weighted summation on each channel of the input features, and the features subjected to the linear weighted summation are used as the input of the next layer of network;
step 5, respectively extracting deep layer characteristics f of the multispectral sample and the panchromatic sample according to the established sharing-separating networkMSAnd fPAN。
5a) Inputting the initial characteristic of the multispectral sample obtained in the step 3 into the sharing sub-network built in the step 4a), and outputting the sharing characteristic of the multispectral sample through forward propagation;
5b) inputting the initial characteristics of the panchromatic sample obtained in the step 3 into the sharing sub-network built in the step 4a), and outputting the sharing characteristics of the panchromatic sample through forward propagation;
note that: the sharing sub-networks in the step 5a) and the step 5b) are the same network and share parameters;
5c) inputting the shared characteristics of the multispectral samples as input characteristics into the spectral separation sub-network built in the step 4b), and outputting the separation characteristics of the multispectral samples through forward propagation;
5d) inputting the shared characteristic of the panchromatic sample as an input characteristic into the panchromatic separation sub-network built in the step 4b), and outputting the separation characteristic of the panchromatic sample through forward propagation;
note that: the spectrum separating sub-network and the panchromatic separating sub-network in the step 5c) and the step 5d) have the same network structure, but are not the same network, and the parameters are not shared;
5e) the separation characteristic of the multispectral sample output in the step 5c) and the separation characteristic of the panchromatic sample output in the step 5d) are respectively taken as the output characteristics of the step 5a) and the step 5b), and the steps 5a) to 5d) are circulated three times to respectively obtain the deep layer characteristic f of the multispectral sampleMSAnd depth features f of panchromatic samplesPAN；
And 6, fusing the self-adaptive features.
6a) Deep features f on multispectral samplesMSAnd depth features f of panchromatic samplesPANAdding to obtain an initial fusion characteristic faddWherein the adding operation is:
fadd＝fMS+fPAN； (B)
6b) for the initial fusion feature faddGlobal compression and excitation are carried out to respectively obtain fusion weights w of the multispectral sample and the panchromatic sample(1)And w(2)The method comprises the following steps:
6b1) for the initial fusion feature faddCarrying out global average operation to obtain a global compressed feature vector of the initial fusion feature, wherein the global average operation is shown as a formula (A);
6b2) inputting the global compressed feature vector of the initial fusion feature into 1 full-connection layer with 6 features, and outputting a fusion compressed feature after forward propagation;
6b3) building a multispectral full-connection layer with 64 characteristics, taking the fusion compression characteristics obtained in the step 6c) as the input of the multispectral full-connection layer, calculating in the forward direction, and outputting multispectral fusion excitation characteristics;
6b4) building a full-color full-connection layer with 64 characteristics, taking the fusion compression characteristics obtained in the step 6c) as the input of the full-color full-connection layer, calculating in the forward direction, and outputting full-color fusion excitation characteristics;
note that: the multispectral full-connection layer and the panchromatic full-connection layer are two networks;
6b5) the multispectral fusion excitation feature and the panchromatic fusion excitation feature are connected in a stacked mode, the connected features are used as the input of a softmax classifier, the features output by the classifier are equally divided into two parts, and the two parts are respectively used for dividing the features into two partsGenerating multispectral sample fusion weights w(1)And panchromatic sample fusion weight w(2)；
6c) Weighting w of multispectral samples(1)And the deep features f of the multispectral samplesMSCarrying out linear weighted summation according to the channels to obtain the linear weighted summation characteristic F of the multispectral samples(1)And the weight of the panchromatic sample and the deep layer characteristic f of the panchromatic sample are comparedPANCarrying out linear weighted summation according to channels to obtain the linear weighted summation characteristic F of the panchromatic sample(2)；
6d) F is to be(1)And F(2)Adding and outputting a self-adaptive fusion characteristic F;
step 7, taking the self-adaptive fusion characteristic F as the input of the softmax classifier, and outputting the probability that each multispectral and panchromatic image sample pair belongs to each class;
and 8, network training.
Constructing a cross entropy loss function according to the probability output in the step 7) and class marks of multispectral and panchromatic sample pairs in a training sample set, reversely transmitting the cross entropy loss function by adopting a gradient descent method, training all networks related in the steps 3) to 7), repeating the steps 3 to 8 until a preset iteration step is reached, and stopping iteration;
and 9, taking the multispectral and panchromatic sample pairs in the test sample set as the input of the step 3, executing the step 3 to the step 7, outputting the probability that each sample pair in the test sample set belongs to each class, and setting the position corresponding to the maximum value of the probability value as the class mark of the test sample pair to finish classification.
Examples
The effect of the present invention can be further illustrated by the following simulation experiments:
(1) simulation conditions
The hardware conditions of the simulation of the invention are as follows: windows XP, SPI, CPU pentium (r)4, with a fundamental frequency of 2.4 GHZ; the software platform is as follows: MatlabR2016a, pyrrch;
the image source selected by simulation is a multispectral and panchromatic image data set of the rural areas and the urban areas, wherein the multispectral and panchromatic image data set of the rural areas contains 8 types of ground objects, as shown in fig. 2(a), and the multispectral and panchromatic image data set of the urban areas contains 7 types of ground objects, as shown in fig. 2 (b); in the invention, 100 pixel points are randomly selected for each type as initial training pixels.
Simulation content and results
Simulation 1, the present invention and the seven existing technologies are used to perform classification simulation on the two data sets shown in fig. 2, and the results are shown in the figure, where:
FIGS. 3 (a) to (h) are graphs of the effect of classification of EMAP, CAE, RNN, SCPF-ResNet, CNN-MS, CNN-PAN, SFNet and the present technology on the data set in the rural area of Western Ann, respectively;
FIGS. 4 (a) to (h) are graphs of the effect of classification of EMAP, CAE, RNN, SCPF-ResNet, CNN-MS, CNN-PAN, SFNet and the inventive technique on the data set in the city of Western Ann, respectively;
as can be seen from the classification result graphs of FIGS. 3-4, the classification method of the present invention has better precision and classification effect. Tables 1 and 2 show the index values of the classification method of the present invention and other seven classification methods in terms of numerical values, and also show that the classification accuracy obtained by the present invention is better. And table 3 also shows the magnitude of the training parameters of various methods in the training process, and obviously the training parameters of the method are reduced, so that the time cost and the classification effect are better balanced.
TABLE 1 index values for the multi-spectral and panchromatic image classification method based on attention mechanism (inventive method) and other seven classification methods described in the present invention
TABLE 2 index values for the multi-spectral and panchromatic image classification method based on attention mechanism (inventive method) and other seven classification methods described in this invention
TABLE 3 training parameters for the attention-based multi-spectral and panchromatic image classification method of the present invention
The above experimental results show that: compared with the prior art, the method has the advantages of improving the classification precision, reducing network training parameters, balancing time cost and classification effect, and the like.
In conclusion, the invention discloses a multispectral and panchromatic image classification method based on an attention mechanism, which mainly solves the problems of high classification time cost and poor fusion effect of multiple sensors in the prior art. The method comprises the following implementation steps: 1) selecting a training sample set and a testing sample set on the multispectral image and the panchromatic image respectively; 2) respectively extracting initial features; 3) building a sharing-separating network, and respectively extracting deep features of the multispectral image and the panchromatic image; 4) performing self-adaptive fusion on the extracted deep features; 5) inputting the fusion characteristics into a classifier, and constructing a loss function to train the network; 6) and inputting the test sample into the trained network, outputting a class mark and finishing classification. The invention can utilize the light-weight network extraction and the characteristics of the multispectral and panchromatic images, and can effectively fuse, and can better balance the time cost and the classification effect.
The above-mentioned contents are only for illustrating the technical idea of the present invention, and the protection scope of the present invention is not limited thereby, and any modification made on the basis of the technical idea of the present invention falls within the protection scope of the claims of the present invention.
Claims (9)
1. A method for multi-spectral and panchromatic image classification based on an attention mechanism, comprising the steps of:
respectively constructing a training sample set and a test sample set which are composed of multispectral samples and panchromatic samples, and carrying out initial feature extraction on the multispectral samples and the panchromatic samples in the training sample set to respectively obtain initial features of the multispectral samples and initial features of the panchromatic samples with the same feature size;
constructing a lightweight sharing-separating network; respectively inputting the initial characteristics of the multispectral samples and the initial characteristics of the panchromatic samples in the training sample set obtained in the step one into a lightweight sharing-separating network, and carrying out deep characteristic extraction to respectively obtain the deep characteristics of the multispectral samples and the deep characteristics of the panchromatic samples;
step three, respectively carrying out self-adaptive feature fusion on the deep features of the multispectral sample and the deep features of the full-color sample obtained in the step two, and outputting fusion features;
step four, classifying the fusion characteristics obtained by the output of the step three, and training the network according to the classification result; based on the trained network, obtaining classification results of multispectral samples and panchromatic samples in a training sample set; and completing the classification method of the multispectral image and the panchromatic image based on the attention mechanism.
2. The method according to claim 1, wherein the step one of constructing a training sample set and a testing sample set composed of multispectral samples and panchromatic samples, and performing initial feature extraction on the multispectral samples and panchromatic samples in the training sample set comprises the following steps:
according to each pixel with a class mark on the multispectral image (part of pixels are not marked), respectively extracting multispectral samples and panchromatic samples in pairs on the multispectral image and the panchromatic image to form a sample pair of the pixels with the class mark; forming a sample set according to the obtained sample pairs; selecting part of sample pairs in the sample set to form a training sample set, and selecting the rest sample pairs in the sample set to form a test sample set;
constructing a multispectral image initial feature extraction network consisting of 2 convolutional layers and 2 downsampling layers, and initializing parameters of each layer; taking multispectral samples in the training sample set as input of a multispectral image initial feature extraction network, and obtaining initial features of the multispectral samples through forward propagation;
constructing a full-color image initial feature extraction network consisting of 3 convolutional layers and 3 downsampling layers, and initializing parameters of each layer; and taking the panchromatic sample in the training sample set as the input of the panchromatic image initial characteristic extraction network, and obtaining the initial characteristic of the panchromatic sample through forward propagation.
3. The method according to claim 2, wherein the specific network design of the multispectral and panchromatic image classification based on the attention mechanism is as follows: the convolution kernel size of the first convolution layer is 3 x 3, the output characteristic number is 32, the convolution kernel size of the second convolution layer is 3 x 3, the output characteristic number is 64, the maximum down-sampling method is adopted by all down-sampling layers, and the sampling kernel is 2 x 2.
4. The method for multispectral and panchromatic image classification based on an attention mechanism as claimed in claim 2, wherein the specific network design of the panchromatic sample initial feature extraction network is as follows: the convolution kernel size of the first convolution layer is 3 x 3, the output characteristic number is 16, the convolution kernel size of the second convolution layer is 3 x 3, the output characteristic number is 32, the convolution kernel size of the second convolution layer is 3 x 3, the output characteristic number is 64, the maximum down-sampling method is adopted by the down-sampling layers, and the sampling kernel is 2 x 2.
5. The method for multispectral and panchromatic image classification based on attention mechanism as claimed in claim 1, wherein in the second step, constructing a lightweight shared-separated network comprises the following steps:
building a sharing sub-network consisting of 1 convolutional layer and 1 down-sampling layer, and initializing the parameters of each layer;
respectively building a spectrum separation sub-network and a panchromatic separation sub-network; the spectrum separation sub-network and the panchromatic separation sub-network have the same network structure and respectively comprise a compression part and an excitation part;
the compression is to perform global average operation on input features to obtain global compressed feature vectors of the input features, the excitation is to input the global compressed feature vectors into 2 full-connection layers, output the global compressed feature vectors to obtain a weight of each channel in initial features, finally perform linear weighted summation on each channel of the input features by using the obtained weights, and use the features of the linear weighted summation as the input of a next layer of network.
6. The method of claim 5 wherein the compression is a global compressed feature vector z of the input features obtained by global averaging of the input featurescWherein the global averaging operation is according to the following formula:
wherein f isc(i, j) represents the feature value of the input feature in the ith row and jth column, and H and W represent the height and width of the input feature, respectively;
the excitation is to compress the global feature vector zcInputting the data into 2 full-connection layers, wherein the number of the features of the first full-connection layer is 4, the number of the features of the second full-connection layer is 64, outputting a 64-dimensional weight vector, the vector dimension is the weight of each channel in the initial features of the multispectral sample and the initial features of the full-color sample, and finally performing linear weighted summation on each channel of the input features by using the obtained weight, and taking the features subjected to the linear weighted summation as the input of the next layer of network.
7. The method according to claim 5, wherein the deep feature extraction in step two comprises the following specific steps:
firstly, inputting the initial characteristics of the multispectral sample and the initial characteristics of the full-color sample into a constructed sharing sub-network, and respectively outputting the sharing characteristics of the multispectral sample and the sharing characteristics of the full-color sample after forward propagation;
then, inputting the shared characteristics of the obtained multispectral samples into a built spectral separation sub-network, and outputting the separation characteristics of the multispectral samples through forward propagation; inputting the shared characteristics of the obtained panchromatic sample into a constructed panchromatic separation sub-network, and outputting the separation characteristics of the panchromatic sample through forward propagation;
and finally, taking the separation characteristics of the output multispectral sample and the separation characteristics of the full-color sample as the input of the constructed sharing sub-network, and circulating the steps for three times to respectively obtain the deep characteristics of the multispectral sample and the deep characteristics of the full-color sample.
8. The method for multi-spectral and panchromatic image classification based on attention mechanism according to claim 1 is characterized in that in the third step, the specific steps of adaptive feature fusion are as follows:
adding the deep features of the obtained multispectral sample and the deep features of the full-color sample to obtain an initial fusion feature;
performing global compression and excitation on the obtained initial fusion features to respectively obtain fusion weights of the multispectral samples and fusion weights of the panchromatic samples; carrying out linear weighted summation on the fusion weight of the obtained multispectral sample and the deep features of the obtained multispectral sample according to the channel to obtain the linear weighted summation features of the multispectral sample; carrying out linear weighted summation on the fusion weight of the obtained panchromatic sample and the deep layer characteristics of the panchromatic sample according to the channel to obtain the linear weighted summation characteristics of the panchromatic sample; and adding the linear weighted summation characteristics of the multispectral samples and the linear weighted summation characteristics of the panchromatic samples, and outputting the self-adaptive fusion characteristics.
9. The method according to claim 1, wherein in step four, the network is trained according to the classification result; based on the trained network, obtaining classification results of multispectral samples and panchromatic samples in a training sample set, comprising the following steps:
inputting the obtained fusion characteristics into a softmax classifier for classification, and outputting the probability of each sample pair in the training sample set belonging to each class;
constructing a cross entropy loss function according to the output probability and the class mark of the sample pair; adopting a gradient descent method to reversely propagate the constructed cross entropy loss function, and training all networks involved in the step one to the step four; repeating the network training from the first step to the fourth step until reaching a preset iteration step, and stopping iteration;
and (3) taking the multispectral samples and the panchromatic samples in the test sample set as the input of the step one, executing the step one to the step four, outputting the probability of each sample pair in the test sample set belonging to each class, and setting the position corresponding to the maximum value of the probability value as the class mark of the sample pair in the test sample set to finish the classification.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110542410.9A CN113240017B (en) | 2021-05-18 | 2021-05-18 | Multispectral and panchromatic image classification method based on attention mechanism |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110542410.9A CN113240017B (en) | 2021-05-18 | 2021-05-18 | Multispectral and panchromatic image classification method based on attention mechanism |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113240017A true CN113240017A (en) | 2021-08-10 |
| CN113240017B CN113240017B (en) | 2023-09-12 |
Family
ID=77135131
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110542410.9A Active CN113240017B (en) | 2021-05-18 | 2021-05-18 | Multispectral and panchromatic image classification method based on attention mechanism |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113240017B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116758361A (en) * | 2023-08-22 | 2023-09-15 | 中国铁路设计集团有限公司 | Engineering rock group remote sensing classification method and system based on spatial and spectral joint characteristics |
| CN118429726A (en) * | 2024-07-02 | 2024-08-02 | 中国科学院空天信息创新研究院 | Remote sensing image classification method, device, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108830330A (en) * | 2018-06-22 | 2018-11-16 | 西安电子科技大学 | Classification of Multispectral Images method based on self-adaptive features fusion residual error net |
| CN111199214A (en) * | 2020-01-04 | 2020-05-26 | 西安电子科技大学 | Residual error network multispectral image ground feature classification method |
| WO2020181685A1 (en) * | 2019-03-12 | 2020-09-17 | 南京邮电大学 | Vehicle-mounted video target detection method based on deep learning |
-
2021
- 2021-05-18 CN CN202110542410.9A patent/CN113240017B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108830330A (en) * | 2018-06-22 | 2018-11-16 | 西安电子科技大学 | Classification of Multispectral Images method based on self-adaptive features fusion residual error net |
| WO2020181685A1 (en) * | 2019-03-12 | 2020-09-17 | 南京邮电大学 | Vehicle-mounted video target detection method based on deep learning |
| CN111199214A (en) * | 2020-01-04 | 2020-05-26 | 西安电子科技大学 | Residual error network multispectral image ground feature classification method |
Non-Patent Citations (2)
| Title |
|---|
| 刘启超;肖亮;刘芳;徐金环;: "SSCDenseNet:一种空-谱卷积稠密网络的高光谱图像分类算法", 电子学报, no. 04 * |
| 魏祥坡;余旭初;张鹏强;职露;杨帆;: "联合局部二值模式的CNN高光谱图像分类", 遥感学报, no. 08 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116758361A (en) * | 2023-08-22 | 2023-09-15 | 中国铁路设计集团有限公司 | Engineering rock group remote sensing classification method and system based on spatial and spectral joint characteristics |
| CN116758361B (en) * | 2023-08-22 | 2023-10-27 | 中国铁路设计集团有限公司 | Engineering rock group remote sensing classification method and system based on spatial and spectral joint characteristics |
| CN118429726A (en) * | 2024-07-02 | 2024-08-02 | 中国科学院空天信息创新研究院 | Remote sensing image classification method, device, electronic equipment and storage medium |
| CN118429726B (en) * | 2024-07-02 | 2024-10-11 | 中国科学院空天信息创新研究院 | Remote sensing image classification method, device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113240017B (en) | 2023-09-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109993220B (en) | Multi-source remote sensing image classification method based on double-path attention fusion neural network | |
| CN111123257B (en) | Radar moving target multi-frame joint detection method based on graph space-time network | |
| CN110084159B (en) | Hyperspectral image classification method based on combined multistage spatial spectrum information CNN | |
| CN108710831B (en) | Small data set face recognition algorithm based on machine vision | |
| CN109727246B (en) | Comparative learning image quality evaluation method based on twin network | |
| CN108182456B (en) | Target detection model based on deep learning and training method thereof | |
| CN108564006B (en) | Polarized SAR terrain classification method based on self-learning convolutional neural network | |
| CN105786860B (en) | Data processing method and device in data modeling | |
| CN104537647B (en) | A kind of object detection method and device | |
| CN111291826B (en) | Pixel-by-pixel classification method of multisource remote sensing image based on correlation fusion network | |
| CN110082821B (en) | Label-frame-free microseism signal detection method and device | |
| CN111523521A (en) | Remote sensing image classification method for double-branch fusion multi-scale attention neural network | |
| CN111259853A (en) | High-resolution remote sensing image change detection method, system and device | |
| CN108960404B (en) | Image-based crowd counting method and device | |
| CN109344698A (en) | EO-1 hyperion band selection method based on separable convolution sum hard threshold function | |
| CN109598306A (en) | Hyperspectral image classification method based on SRCM and convolutional neural networks | |
| CN113240017A (en) | Multispectral and panchromatic image classification method based on attention mechanism | |
| CN109146944A (en) | A kind of space or depth perception estimation method based on the revoluble long-pending neural network of depth | |
| CN110619373B (en) | Infrared multispectral weak target detection method based on BP neural network | |
| CN109376753A (en) | A kind of the three-dimensional space spectrum separation convolution depth network and construction method of dense connection | |
| CN111161224A (en) | Casting internal defect grading evaluation system and method based on deep learning | |
| CN117576473A (en) | Hyperspectral image classification method based on mixed covariance attention and cross-layer fusion transducer | |
| CN112381179A (en) | Heterogeneous graph classification method based on double-layer attention mechanism | |
| CN112766340B (en) | Depth capsule network image classification method and system based on self-adaptive spatial mode | |
| CN111738114A (en) | Vehicle target detection method based on anchor-free accurate sampling remote sensing image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |