CN111695467B - Spatial spectrum full convolution hyperspectral image classification method based on super-pixel sample expansion - Google Patents

Spatial spectrum full convolution hyperspectral image classification method based on super-pixel sample expansion Download PDF

Info

Publication number
CN111695467B
CN111695467B CN202010485713.7A CN202010485713A CN111695467B CN 111695467 B CN111695467 B CN 111695467B CN 202010485713 A CN202010485713 A CN 202010485713A CN 111695467 B CN111695467 B CN 111695467B
Authority
CN
China
Prior art keywords
convolution
spatial
hyperspectral image
sample
hyperspectral
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010485713.7A
Other languages
Chinese (zh)
Other versions
CN111695467A (en
Inventor
王佳宁
李林昊
郭思颖
黄润虎
杨攀泉
焦李成
侯彪
张向荣
毛莎莎
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xidian University
Original Assignee
Xidian University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xidian University filed Critical Xidian University
Priority to CN202010485713.7A priority Critical patent/CN111695467B/en
Publication of CN111695467A publication Critical patent/CN111695467A/en
Application granted granted Critical
Publication of CN111695467B publication Critical patent/CN111695467B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G06V20/13—Satellite images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
    • G06V10/464—Salient features, e.g. scale invariant feature transforms [SIFT] using a plurality of salient features, e.g. bag-of-words [BoW] representations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G06V20/194—Terrestrial scenes using hyperspectral data, i.e. more or other wavelengths than RGB
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A40/00—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production
    • Y02A40/10—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production in agriculture

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Biomedical Technology (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Biophysics (AREA)
  • Software Systems (AREA)
  • Astronomy & Astrophysics (AREA)
  • Remote Sensing (AREA)
  • Probability & Statistics with Applications (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)
  • Investigating Or Analysing Materials By Optical Means (AREA)

Abstract

The invention discloses a spatial spectrum full convolution hyperspectral image classification method based on super-pixel sample expansion, which inputs hyperspectral images; acquiring a training set and a testing set; carrying out principal component analysis and dimension reduction on the hyperspectral image; performing entropy rate segmentation on the dimension reduction result; generating a pseudo tag sample; updating the training set; carrying out data preprocessing on the hyperspectral image; inputting a convolutional neural network; training a convolutional neural network to classify hyperspectral images; repeating the above operation and voting; and outputting hyperspectral classification results. According to the method, the entropy rate super-pixel segmentation result is utilized to expand the pseudo-label sample, the priori features of the hyperspectral image are fully utilized, the number of samples is increased, the problems of network overfitting and slow network convergence speed are solved, and the accuracy of hyperspectral image classification under the condition of rare marked samples is improved.

Description

Spatial spectrum full convolution hyperspectral image classification method based on super-pixel sample expansion
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a spatial spectrum full convolution hyperspectral image classification method based on super-pixel sample expansion.
Background
With the progress of scientific technology, hyperspectral remote sensing technology has been greatly developed. The hyperspectral data may be represented as a hyperspectral data cube, which is a three-dimensional data structure. The hyperspectral data can be regarded as a three-dimensional image, and one-dimensional spectrum information is added besides a common two-dimensional image. The space image describes the two-dimensional space characteristics of the earth surface, and the spectrum dimension reveals the spectrum curve characteristics of each pixel of the image, so that the organic fusion of the remote sensing data space dimension and spectrum dimension information is realized. The hyperspectral remote sensing image contains rich spectral information, can provide spatial domain information and spectral domain information, has the characteristic of 'map in one', can realize precise identification and detail extraction of ground objects, and provides favorable conditions for understanding objective world. Because of the unique characteristics of hyperspectral images, hyperspectral remote sensing technology has been widely used in different fields. In the civil field, hyperspectral remote sensing images have been used in urban environmental monitoring, surface soil monitoring, geological exploration, disaster assessment, agricultural yield estimation, crop analysis and the like. Hyperspectral remote sensing technology has been widely used in people's daily lives. Therefore, designing a practical and efficient hyperspectral image classification method has become an indispensable technological requirement in modern society.
Currently, researchers have proposed a number of classical classification methods for hyperspectral image classification, representative of which are Support Vector Machines (SVMs) and neural networks (CNNs). The SVM obtains a better classification result in small sample classification by maximizing class boundaries. The SVM is introduced into hyperspectral image classification by the aid of Melgani and L.Bruzzone at Classification of hyperspectral remote sensing images with support vector machines, and the best classification result is obtained at the moment, however, the SVM determines Ding He functions, empirical judgment is completely needed, and poor classification performance is caused by the fact that unsuitable kernel functions are selected. Meanwhile, with the rise of deep learning, convolutional neural networks are also applied to the field of hyperspectral image classification. However, since a large number of marked samples are needed to train the convolutional neural network as training samples, and the labeling cost of hyperspectral images is very high, how to solve the problem of small samples is a popular direction at present.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a spatial spectrum full convolution hyperspectral image classification method based on the expansion of a hyperspectral sample, which uses a segmentation result as prior information to improve the classification effect of hyperspectral images.
The invention adopts the following technical scheme:
a spatial spectrum full convolution hyperspectral image classification method based on super-pixel sample expansion comprises the following steps:
s1, inputting a hyperspectral image PavaU, and acquiring a training sample X from the hyperspectral image PavaU t And test sample X e ；
S2, carrying out normalization processing on the hyperspectral data set, taking n labels which are in the neighborhood of each training sample from the segmentation label matrix and are the same as the segmentation labels of the training sample, and adding the n labels as pseudo label samples into the training samples;
s3, respectively constructing a spectrum feature extraction module and a spatial spectrum feature extraction module, constructing a spectrum feature map and spatial spectrum feature map weighted fusion module, taking the spatial spectrum combined features as input, passing through two convolution layers and setting, and constructing a spatial spectrum combined full convolution neural network for hyperspectral classification;
s4, constructing a loss function of the full convolution neural network in the step S3, and training the neural network;
s5, obtaining a final classification result diagram through multiple training votes, and realizing image classification.
Specifically, step S1 specifically includes:
s101, recording a three-dimensional hyperspectral image Pavia U as
Figure BDA0002519100660000021
U, V, C are respectivelyThe space length, the space width and the spectrum channel number of the hyperspectral image comprise N pixel points, each pixel point has C spectrum wave bands, and N=U×V;
s102, randomly taking 30 samples of class labels 1 to 9 in X to form an initial training sample set X t The rest is taken as a test sample X e 。
Specifically, step S2 specifically includes:
s201, performing PCA dimension reduction treatment on the three-dimensional hyperspectral image, wherein the number of channels of the dimension reduced image is 1;
s202, performing entropy rate super-pixel segmentation on the image subjected to PCA dimension reduction, wherein the segmentation result is 50 blocks, and the obtained segmentation label matrix is
Figure BDA0002519100660000031
S203, setting a real label matrix
Figure BDA0002519100660000032
Dividing the tag matrix into->
Figure BDA0002519100660000033
(x 0 ,y 0 ) The true label of the training sample at is +.>
Figure BDA0002519100660000034
Dividing labels in the dividing map into +.>
Figure BDA0002519100660000035
The (x, y) centered 7 x 7 space is selected to satisfy
Figure BDA0002519100660000036
Generates a pseudo tag as +.>
Figure BDA0002519100660000037
The pseudo tag samples meeting the above criteria are expanded, at this time, the number of training samples becomes n+1 times of the original number, and the test samples remain unchanged.
Specifically, the step S3 specifically includes:
s301, constructing a spectral feature extraction module, wherein the spectral feature extraction module comprises three convolution layers and a merging layer, and a relu activation function and batch normalization processing are added behind each convolution layer;
s302, constructing a spatial spectrum feature extraction module, wherein the spatial spectrum feature extraction module comprises a 1X 1 convolution, a relu activation layer, a batch normalization layer, a multi-scale spatial feature fusion layer, a 3X 3 cavity convolution layer, a relu activation layer, a batch normalization layer, a 2X 2 average pooling layer and a merging layer;
s303, constructing a spectrum characteristic diagram and a spatial spectrum characteristic diagram weighted fusion module;
s304, taking the characteristics of spatial spectrum combination as input to pass through two convolution layers;
s305, performing PCA dimension reduction on the convolved feature map to 5 dimensions for subsequent CRF processing;
s306, performing Softmax operation on the convolved feature map to output a classification probability matrix, and outputting the dimension number with the largest value in the classification probability matrix as a prediction class label to obtain a classification result.
Further, in step S301, the batch normalization parameters are: momentum=0.8, the convolution kernel sizes are all 1, the step size is 1, the number of channels after all convolutions is 64, and the convolution results are added to obtain a spectrum characteristic diagram after three convolutions are continuous.
Further, in step S302, the first convolution layer uses 1×1 convolution, and the step size is 1; the position rate of the 3×3 hole convolution is 2, and the step size is 1; all convolution result channels number 64; all batch normalization parameters were: momentum=0.8, the merging layer is the feature map addition of three convolutional layers, and the channel number remains 64.
Further, in step S303, the spectral signature and the spatial signature are weighted and superimposed as follows:
C unite ＝λ spectral C spectral +λ spatial C spatial
wherein C is unite Lambda is the weighted feature map spectral And lambda (lambda) spatial Respectively trainable lights in a networkWeighting coefficients of spectral and spatial features, C spectral And C spatial A spectral signature and a spatial signature, respectively.
Specifically, in step S4, the loss function is:
L＝L 1 +L 2
Figure BDA0002519100660000041
where L is the final loss function, L 1 And L 2 Marked samples and pseudo-tag samples in the training set respectively,
Figure BDA0002519100660000043
and->
Figure BDA0002519100660000042
The label and the predictive label representing the ith training sample, j takes 1 or 2 to represent the sample as the original sample or the pseudo label sample.
Specifically, step S5 specifically includes:
s501, adding the feature map after convolution of the PCA in the step S3 and the dimension reduced to 5 dimensions and the classification result obtained by the normalized hyperspectral data input network in the step S4 into a conditional random field;
s502, obtaining a classification result of one training through a conditional random field;
and S503, repeating the pseudo sample expansion and network training operations for m times for the same training sample to obtain m times of classification results, and outputting the prediction class mark with the largest occurrence number of each pixel as a final prediction class mark.
Further, in step S501, the energy function of the conditional random field is as follows:
Figure BDA0002519100660000051
wherein, psi is u (y i ) Sum phi p (y i ,y j ) Respectively a unitary function partAnd a binary function portion.
Compared with the prior art, the invention has at least the following beneficial effects:
according to the spatial spectrum full convolution hyperspectral image classification method based on the hyperspectral image expansion, the pseudo-label sample is generated by utilizing the result guidance generated by hyperspectral image segmentation, the training sample is effectively expanded by utilizing the priori information of the hyperspectral image, and the better classification accuracy can be maintained under the condition of a small sample; the characteristic extraction is carried out by adopting a space spectrum combination mode, so that the spectrum and space spectrum characteristics of the hyperspectral image can be more fully extracted, and the classification accuracy of the hyperspectral image is improved; the spatial feature extraction module uses the cavity convolution with different condition rates to realize multi-scale feature fusion, and the spatial features of the hyperspectral image are extracted on a plurality of scales; the voter is added before the final classification result, so that the robustness of the whole structure is enhanced, and the classification result is more stable and reliable; by adopting the full convolution neural network, a full connection layer is not introduced, and the end-to-end image classification of hyperspectral images with any size can be realized; the full convolution neural network inputs the preprocessed hyperspectral data with the same size as the original hyperspectral image, so that the condition of high redundancy of training data caused by using the Patch of each pixel as input data commonly used in the existing method is avoided.
Furthermore, the entropy rate superpixel is set to obtain the prior information of the hyperspectral image, so that the hyperspectral image can be effectively utilized, samples similar to training samples are supplemented to the training set under the condition that classification labels are not needed to be known, and the difficulty that the hyperspectral image can be used for training and marked samples are scarce is effectively solved.
Furthermore, the full convolution neural network used for constructing the full convolution neural network for the spatial spectrum combination of hyperspectral classification has no full connection layer, so that hyperspectral images with any size can be very conveniently accepted as input. The mode of spatial spectrum combination can combine spectral information and spatial information in a hyperspectral image into new features that are better than single spatial features or spectral features.
Furthermore, the characteristic extraction of the spectrum module is completed by using a continuous 1×1 convolution layer on the premise of affecting the space information as little as possible, and the gradient information is kept by the residual module so that the model can be better converged.
Furthermore, the feature extraction of the spatial spectrum module uses the cavity convolution of different conditions, so that the receptive field can be enlarged and multi-scale spatial information can be extracted.
Further, aiming at the proposed pseudo tag sample expansion method, a loss function of the full convolution neural network is constructed, corresponding improvement is made, and the cross entropy of the pseudo tag sample is also added into the loss function, so that the network can be converged better.
Furthermore, aiming at instability caused by self-limitation of the proposed pseudo tag sample expansion method, the final classification diagram is obtained through multiple training votes, so that the robustness of the model is effectively improved.
In summary, the spatial spectrum full convolution hyperspectral image classification method based on the hyperspectral image expansion provided by the invention effectively utilizes the priori information of the hyperspectral image to realize the expansion of the pseudo sample, solves the problem of scarcity of the hyperspectral image marked sample, and simultaneously, the full convolution classification network of the spatial spectrum fully utilizes the multi-scale spatial characteristics and the spectral characteristics to realize higher classification precision.
The technical scheme of the invention is further described in detail through the drawings and the embodiments.
Drawings
FIG. 1 is a flow chart of an implementation of the present invention;
FIG. 2 is a flow diagram of a pseudo tag sample expansion implementation of the present invention;
FIG. 3 is a multi-scale spatial feature fusion module in accordance with the present invention.
Detailed Description
The invention provides a spatial spectrum full convolution hyperspectral image classification method based on super-pixel sample expansion, which inputs hyperspectral images; acquiring a training set and a testing set; carrying out principal component analysis and dimension reduction on the hyperspectral image; performing entropy rate super-pixel segmentation on the dimension reduction result; generating a pseudo tag sample; updating the training set; carrying out data preprocessing on the hyperspectral image; inputting a full convolution neural network; training a full convolution neural network to classify hyperspectral images; repeating the above operation and voting; and outputting hyperspectral classification results. According to the invention, the entropy rate super-pixel segmentation result is utilized to expand the pseudo-label sample, the space prior information of the hyperspectral image is fully utilized, the number of samples is increased, the problem of network over-fitting is solved, and the accuracy rate, the classification efficiency and the classification performance of hyperspectral image classification under the condition of small samples are effectively improved.
Referring to fig. 1, the spatial spectrum full convolution hyperspectral image classification method based on super-pixel sample expansion of the invention comprises the following steps:
s1, inputting a hyperspectral image PavaU, wherein the hyperspectral three-dimensional image is data used in the experiment of the invention, and acquiring a training sample X from the PavaU hyperspectral image t And test sample X e ；
S101, recording a three-dimensional hyperspectral image Pavia U as
Figure BDA0002519100660000071
Wherein U, V, C are the spatial length, spatial width, and spectral channel number, respectively, of a hyperspectral image comprising N pixels, each pixel having C spectral bands, where n=u×v. N= 207400 samples in the PaviaU dataset, u=610, v=340, c=103. Samples with class labels 1 to 9 total 42776. Normalizing X, and keeping the data value at [0,1 ]]The specific steps are as follows:
Figure BDA0002519100660000072
；
s102, randomly taking 30 samples with class labels of 1 to 9 in X to form an initial training sample set X t The rest is taken as a test sample X e 。
S2, generating a pseudo tag sample by generating a segmentation tag by utilizing entropy super pixels after normalizing the hyperspectral data set;
s201, performing PCA dimension reduction treatment on the three-dimensional hyperspectral image, wherein the number of channels of the dimension reduced image is 1;
s202, performing entropy rate super-pixel segmentation on the image subjected to PCA dimension reduction, wherein the segmentation result is 50 blocks, and the obtained segmentation label matrix is
Figure BDA0002519100660000081
S203, setting a real label matrix
Figure BDA0002519100660000082
Dividing the tag matrix into->
Figure BDA0002519100660000083
(x 0 ,y 0 ) The true label of the training sample at is +.>
Figure BDA0002519100660000084
Dividing labels in the dividing map into +.>
Figure BDA0002519100660000085
The (x, y) centered 7 x 7 space is selected to satisfy
Figure BDA0002519100660000086
For which pseudo tag is generated +.>
Figure BDA0002519100660000087
The pseudo tag samples meeting the above criteria are expanded, and at this time, the number of training samples becomes n+1 times the original number, and the test samples remain unchanged, as shown in fig. 2.
S3, constructing a space spectrum combined full convolution neural network for hyperspectral classification;
s301, a spectrum characteristic extraction module consists of three convolution layers, wherein a relu activation function and batch normalization processing are added behind each convolution layer.
The batch normalization processing parameters are as follows:
momentum=0.8, the convolution kernel sizes are all 1, the step size is 1, the number of channels after all convolutions is 64, and the convolution results are added to obtain a spectrum characteristic diagram after three convolutions are continuous.
S302, a spatial spectrum feature extraction module consists of three convolution layers, wherein the convolution kernel size of a first convolution layer is 1, and the step length is 1; the second convolution layer is used for realizing multi-scale feature fusion, and the structure is shown in figure 3, and is obtained by adding three cavity convolutions with the convolution kernel size of 3, the adaptation rate of 2,3 and 4 and the step length of 1; the third convolutional layer is followed by a 2 x 2 average pooling layer.
After the convolution is completed, performing a relu activation function and batch normalization treatment; the convolution kernel size of the third convolution layer is 3, the adaptation rate is 2, and the step size is 1.
All batch normalization parameters were:
momentum=0.8, the number of channels of all convolution results is 64, and the convolution results are added after three consecutive convolutions to obtain a spatial spectrum characteristic diagram.
S303, weighting and superposing the spectrum characteristic diagram and the space spectrum characteristic diagram according to the following formula:
C unite ＝λ spectral C spectral +λ spatial C spatial
wherein C is unite For weighted feature graphs, the channel is still 64, lambda spectral And lambda (lambda) spatial Weighting coefficients for trainable spectral and spatial features, respectively, in a network, C spectral And C spatial A spectrum characteristic diagram and a space spectrum characteristic diagram respectively;
s304, performing relu activation after the characteristic of spatial spectrum combination is taken as input and passes through two 1X 1 convolution layers, wherein the convolution kernel sizes are 1, the step sizes are 1, the number of channels of a first convolution result is 64, and the second convolution result is 128;
s305, performing PCA dimension reduction on the convolved feature map to 5 dimensions for subsequent CRF processing;
s306, performing Softmax operation on the convolved feature map to output a 610×340×9 classification probability matrix, and outputting the dimension number with the largest value in 9 dimensions as a prediction category label to obtain a classification result with the size of 610×340.
S4, constructing a loss function of the full convolution neural network, and training the neural network;
s401, calculating cross entropy of the training sample prediction label and the training sample label after expansion by using the cross entropy of the loss function, wherein the cross entropy is shown in the following formula:
L＝L 1 +L 2
Figure BDA0002519100660000091
where L is the final loss function, L 1 And L 2 Marked samples and pseudo-tag samples in the training set respectively,
Figure BDA0002519100660000093
and->
Figure BDA0002519100660000092
The label and the predictive label representing the ith training sample, j takes 1 or 2 to represent that the sample is an original sample or a pseudo label sample;
s402, inputting the normalized hyperspectral data into a network, and iterating 1000 times to generate a predictive label graph.
S5, training voting for many times to obtain a final classification result diagram.
S501, adding the outputs of step S305 and step S402 to a conditional random field, the energy function of which is shown below,
Figure BDA0002519100660000101
ψ u (y i ) Sum phi p (y i ,y j ) A unitary function part and a binary function part, respectively.
In the invention, the calculation formula of the unitary part is psi u (y i )＝-log P(y i ) Wherein P (y i ) Is the label assignment probability for pixel i given by the proposed full convolution network.
The binary function part is defined as:
Figure BDA0002519100660000102
wherein if y i ＝y j ,μ(y i ,y j ) =1, otherwise zero; k (k) m Is a gaussian kernel; f (f) i And f j Is a feature vector of pixels i and j in any feature space; omega m Is the corresponding weight; to fully exploit the deep spectral space features, C in S305 unite Is used as a feature of each pixel. The complete form of the gaussian kernel is written as:
Figure BDA0002519100660000103
；
s502, obtaining a classification result of one training through a conditional random field;
and S503, repeating the operation for m times on the same training sample to obtain m times of classification results, and outputting the prediction class mark with the largest occurrence number of each pixel as a final prediction class mark.
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The three evaluation indexes of the simulation experiment are specifically as follows:
the total accuracy OA indicates the proportion of correctly classified samples to all samples, and the larger the value, the better the classification effect. The average accuracy AA represents the average value of the classification accuracy of each class, and the larger the value is, the better the classification effect is. The chi-square coefficient Kappa indicates different weights in the confusion matrix, and the larger the value is, the better the classification effect is.
The prior art comparison and classification method used in the invention is as follows:
the hyperspectral image classification method, abbreviated as the depth feature fusion DFFN method, is proposed by Song et al in "Hyperspectral Image Classification With Deep Feature Fusion Network, IEEE Trans. Geosci. Remote Sens., vol.56, no.6, pp.3173-3184, june 2018".
Table 1 is a quantitative analysis table of the classification results of the present invention (select PaviaU dataset, 30 labeled samples per class as training set:
Figure BDA0002519100660000111
Figure BDA0002519100660000121
in summary, according to the spatial spectrum full-convolution hyperspectral image classification method based on the hyperspectral sample expansion, the prior information of the hyperspectral image is effectively utilized to realize the pseudo-sample expansion, the problem that the hyperspectral image has marked samples is solved, and meanwhile, the full-convolution classification network of the spatial spectrum fully utilizes the multi-scale spatial features and the spectral features to realize higher classification precision.
The above is only for illustrating the technical idea of the present invention, and the protection scope of the present invention is not limited by this, and any modification made on the basis of the technical scheme according to the technical idea of the present invention falls within the protection scope of the claims of the present invention.

Claims (9)

1. The spatial spectrum full convolution hyperspectral image classification method based on the super-pixel sample expansion is characterized by comprising the following steps of:
s1, inputting a hyperspectral image PavaU, and acquiring a training sample X from the hyperspectral image PavaU t And test sample X e ；
S2, carrying out normalization processing on the hyperspectral data set, taking n labels which are in the neighborhood of each training sample from the segmentation label matrix and are the same as the segmentation labels of the training sample as pseudo label samples, and adding the pseudo label samples into the training samples, wherein the method specifically comprises the following steps of:
s201, performing PCA dimension reduction treatment on the three-dimensional hyperspectral image, wherein the number of channels of the dimension reduced image is 1;
s202, performing entropy rate super-pixel segmentation on the image subjected to PCA dimension reduction, wherein the segmentation result is 50 blocks, and the obtained segmentation label matrix is
Figure FDA0004051615430000011
S203, setting a real label matrix
Figure FDA0004051615430000012
Dividing the tag matrix into->
Figure FDA0004051615430000013
(x 0 ,y 0 ) The true label of the training sample at is +.>
Figure FDA0004051615430000014
Dividing labels in the dividing map into +.>
Figure FDA0004051615430000015
The 7X 7 space centered on (x, y) is chosen to be satisfied +.>
Figure FDA0004051615430000016
Generates a pseudo tag as +.>
Figure FDA0004051615430000017
Meets the above standardThe pseudo tag samples are expanded, the number of training samples is n+1 times of the number of the original training samples, and the test samples are kept unchanged;
s3, respectively constructing a spectrum feature extraction module and a spatial spectrum feature extraction module, constructing a spectrum feature map and spatial spectrum feature map weighted fusion module, taking the spatial spectrum combined features as input, passing through two convolution layers and setting, and constructing a spatial spectrum combined full convolution neural network for hyperspectral classification;
s4, constructing a loss function of the full convolution neural network in the step S3, and training the neural network;
s5, obtaining a final classification result diagram through multiple training votes, and realizing image classification.
2. The spatial spectrum full convolution hyperspectral image classification method based on the extension of the superpixel samples as claimed in claim 1, wherein step S1 specifically comprises:
s101, recording a three-dimensional hyperspectral image Pavia U as
Figure FDA0004051615430000018
U, V, C are the space length, space width and spectrum channel number of the hyperspectral image respectively, the hyperspectral image comprises N pixel points, each pixel point has C spectrum wave bands, and N=U×V;
s102, randomly taking 30 samples of class labels 1 to 9 in X to form an initial training sample set X t The rest is taken as a test sample X e 。
3. The spatial spectrum full convolution hyperspectral image classification method based on the extension of the superpixel samples as claimed in claim 1, wherein the step S3 is specifically:
s301, constructing a spectral feature extraction module, wherein the spectral feature extraction module comprises three convolution layers and a merging layer, and a relu activation function and batch normalization processing are added behind each convolution layer;
s302, constructing a spatial spectrum feature extraction module, wherein the spatial spectrum feature extraction module comprises a 1X 1 convolution, a relu activation layer, a batch normalization layer, a multi-scale spatial feature fusion layer, a 3X 3 cavity convolution layer, a relu activation layer, a batch normalization layer, a 2X 2 average pooling layer and a merging layer;
s303, constructing a spectrum characteristic diagram and a spatial spectrum characteristic diagram weighted fusion module;
s304, taking the characteristics of spatial spectrum combination as input to pass through two convolution layers;
s305, performing PCA dimension reduction on the convolved feature map to 5 dimensions for subsequent CRF processing;
s306, performing Softmax operation on the convolved feature map to output a classification probability matrix, and outputting the dimension number with the largest value in the classification probability matrix as a prediction class label to obtain a classification result.
4. The method for classifying a spatial-spectral full-convolution hyperspectral image based on super-pixel sample expansion as claimed in claim 3, wherein in step S301, the batch normalization processing parameters are: momentum=0.8, the convolution kernel sizes are all 1, the step size is 1, the number of channels after all convolutions is 64, and the convolution results are added to obtain a spectrum characteristic diagram after three convolutions are continuous.
5. The method for classifying a spatial spectrum full convolution hyperspectral image based on super pixel sample expansion as claimed in claim 3, wherein in step S302, the first convolution layer uses 1 x 1 convolution with a step size of 1; the position rate of the 3×3 hole convolution is 2, and the step size is 1; all convolution result channels number 64; all batch normalization parameters were: momentum=0.8, the merging layer is the feature map addition of three convolutional layers, and the channel number remains 64.
6. The method for classifying a spatial spectrum full convolution hyperspectral image based on the extension of a superpixel sample according to claim 3, wherein in step S303, a spectral feature map and a spatial spectrum feature map are weighted and superimposed as follows:
C unite ＝λ spectral C spectral +λ spatial C spatial
wherein C is unite Lambda is the weighted feature map spectral And lambda (lambda) spatial Weighting coefficients for trainable spectral and spatial features, respectively, in a network, C spectral And C spatial A spectral signature and a spatial signature, respectively.
7. The method for classifying a spatial-spectral full-convolution hyperspectral image based on super-pixel sample expansion according to claim 1, wherein in step S4, the loss function is:
L＝L 1 +L 2
Figure FDA0004051615430000031
where L is the final loss function, L 1 And L 2 Marked samples and pseudo-labeled samples in the training set, y j (i) And
Figure FDA0004051615430000032
the label and the predictive label representing the ith training sample, j takes 1 or 2 to represent the sample as the original sample or the pseudo label sample.
8. The spatial spectrum full convolution hyperspectral image classification method based on the extension of the superpixel samples as claimed in claim 1, wherein step S5 specifically comprises:
s501, adding the feature map after convolution of the PCA in the step S3 and the dimension reduced to 5 dimensions and the classification result obtained by the normalized hyperspectral data input network in the step S4 into a conditional random field;
s502, obtaining a classification result of one training through a conditional random field;
and S503, repeating the pseudo sample expansion and network training operations for m times for the same training sample to obtain m times of classification results, and outputting the prediction class mark with the largest occurrence number of each pixel as a final prediction class mark.
9. The method for classifying a spatial-spectral full-convolution hyperspectral image based on super-pixel sample expansion according to claim 8, wherein in step S501, the energy function of the conditional random field is as follows:
Figure FDA0004051615430000033
wherein, psi is u (y i ) Sum phi p (y i ,y j ) A unitary function part and a binary function part, respectively.
CN202010485713.7A 2020-06-01 2020-06-01 Spatial spectrum full convolution hyperspectral image classification method based on super-pixel sample expansion Active CN111695467B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010485713.7A CN111695467B (en) 2020-06-01 2020-06-01 Spatial spectrum full convolution hyperspectral image classification method based on super-pixel sample expansion

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010485713.7A CN111695467B (en) 2020-06-01 2020-06-01 Spatial spectrum full convolution hyperspectral image classification method based on super-pixel sample expansion

Publications (2)

Publication Number Publication Date
CN111695467A CN111695467A (en) 2020-09-22
CN111695467B true CN111695467B (en) 2023-05-30

Family

ID=72479042

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010485713.7A Active CN111695467B (en) 2020-06-01 2020-06-01 Spatial spectrum full convolution hyperspectral image classification method based on super-pixel sample expansion

Country Status (1)

Country Link
CN (1) CN111695467B (en)

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112232137A (en) * 2020-09-24 2021-01-15 北京航空航天大学 Hyperspectral image processing method and device
CN112699756B (en) * 2020-12-24 2023-08-25 中国农业科学院农业信息研究所 Hyperspectral image-based tea origin identification method and system
CN112733769B (en) * 2021-01-18 2023-04-07 西安电子科技大学 Hyperspectral image classification method based on multiband entropy rate superpixel segmentation
CN113052014B (en) * 2021-03-09 2022-12-23 西北工业大学深圳研究院 Hyperspectral image classification method based on double-layer space manifold representation
CN112949592B (en) * 2021-03-31 2022-07-22 云南大学 Hyperspectral image classification method and device and electronic equipment
CN113222867B (en) * 2021-04-16 2022-05-20 山东师范大学 Image data enhancement method and system based on multi-template image
CN113239755B (en) * 2021-04-28 2022-06-21 湖南大学 Medical hyperspectral image classification method based on space-spectrum fusion deep learning
CN113327231B (en) * 2021-05-28 2022-10-14 北京理工大学重庆创新中心 Hyperspectral abnormal target detection method and system based on space-spectrum combination
CN113516194B (en) * 2021-07-20 2023-08-08 海南长光卫星信息技术有限公司 Semi-supervised classification method, device, equipment and storage medium for hyperspectral remote sensing images
WO2023000160A1 (en) * 2021-07-20 2023-01-26 海南长光卫星信息技术有限公司 Hyperspectral remote sensing image semi-supervised classification method, apparatus, and device, and storage medium
CN113642655B (en) * 2021-08-18 2024-02-13 杭州电子科技大学 Small sample image classification method based on support vector machine and convolutional neural network
CN113723255B (en) * 2021-08-24 2023-09-01 中国地质大学（武汉） Hyperspectral image classification method and storage medium
CN113673607A (en) * 2021-08-24 2021-11-19 支付宝(杭州)信息技术有限公司 Method and device for training image annotation model and image annotation
CN113822209B (en) * 2021-09-27 2023-11-14 海南长光卫星信息技术有限公司 Hyperspectral image recognition method and device, electronic equipment and readable storage medium
CN113902013A (en) * 2021-10-09 2022-01-07 黑龙江雨谷科技有限公司 Hyperspectral classification method based on three-dimensional convolutional neural network and superpixel segmentation
CN114049567B (en) * 2021-11-22 2024-02-23 齐鲁工业大学 Adaptive soft label generation method and application in hyperspectral image classification
CN114332534B (en) * 2021-12-29 2024-03-29 山东省科学院海洋仪器仪表研究所 Hyperspectral image small sample classification method

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106097355A (en) * 2016-06-14 2016-11-09 山东大学 The micro-Hyperspectral imagery processing method of gastroenteric tumor based on convolutional neural networks
CN109948693B (en) * 2019-03-18 2021-09-28 西安电子科技大学 Hyperspectral image classification method based on superpixel sample expansion and generation countermeasure network
CN110321963B (en) * 2019-07-09 2022-03-04 西安电子科技大学 Hyperspectral image classification method based on fusion of multi-scale and multi-dimensional space spectrum features

Also Published As

Publication number Publication date
CN111695467A (en) 2020-09-22

Similar Documents

Publication Publication Date Title
CN111695467B (en) Spatial spectrum full convolution hyperspectral image classification method based on super-pixel sample expansion
CN110443143B (en) Multi-branch convolutional neural network fused remote sensing image scene classification method
CN108596248B (en) Remote sensing image classification method based on improved deep convolutional neural network
CN111797779A (en) Remote sensing image semantic segmentation method based on regional attention multi-scale feature fusion
CN111274869B (en) Method for classifying hyperspectral images based on parallel attention mechanism residual error network
CN105138973B (en) The method and apparatus of face authentication
CN110321967B (en) Image classification improvement method based on convolutional neural network
CN111414942A (en) Remote sensing image classification method based on active learning and convolutional neural network
CN112347970B (en) Remote sensing image ground object identification method based on graph convolution neural network
CN113674334B (en) Texture recognition method based on depth self-attention network and local feature coding
CN109344891A (en) A kind of high-spectrum remote sensing data classification method based on deep neural network
CN114842264B (en) Hyperspectral image classification method based on multi-scale spatial spectrum feature joint learning
CN110188827A (en) A kind of scene recognition method based on convolutional neural networks and recurrence autocoder model
CN110414616B (en) Remote sensing image dictionary learning and classifying method utilizing spatial relationship
CN111626267B (en) Hyperspectral remote sensing image classification method using void convolution
CN113947725B (en) Hyperspectral image classification method based on convolution width migration network
CN111639697B (en) Hyperspectral image classification method based on non-repeated sampling and prototype network
CN115457311B (en) Hyperspectral remote sensing image band selection method based on self-expression transfer learning
CN114155443A (en) Hyperspectral image classification method based on multi-receptive-field attention network
CN109034213B (en) Hyperspectral image classification method and system based on correlation entropy principle
CN114694039A (en) Remote sensing hyperspectral and laser radar image fusion classification method and device
CN116630700A (en) Remote sensing image classification method based on introduction channel-space attention mechanism
CN111310820A (en) Foundation meteorological cloud chart classification method based on cross validation depth CNN feature integration
Dai et al. Research on hyper-spectral remote sensing image classification by applying stacked de-noising auto-encoders neural network
CN114399687A (en) Semi-supervised self-training hyperspectral remote sensing image classification method based on spatial correction

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant