CN109359623B - Hyperspectral image transfer classification method based on deep joint distribution adaptation network - Google Patents

Abstract

A hyperspectral image migration classification method based on a deep joint distribution adaptation network. Features; construct an edge probability distribution adaptation network to adapt the edge probability distribution of hyperspectral images in the source and target domains; according to the principle of one-to-many classification, select the training samples of the source domain and a small number of hyperspectral images in the target domain; construct The conditional probability distribution adaptation network is used to adapt the conditional probability distribution of the hyperspectral images in the source domain and the target domain; perform one-to-many classification on the hyperspectral images in the target domain. The invention proposes a deep joint distribution adaptation network, which realizes the feature adaptation of the hyperspectral images in the source domain and the target domain, and reduces the joint probability distribution difference between the two; Intra- and inter-class discrimination, which in turn improves the accuracy of hyperspectral image transfer classification.

Description

Hyperspectral image transfer classification method based on deep joint distribution adaptation network

technical field

The invention belongs to the technical field of remote sensing image processing, in particular to a hyperspectral image migration classification based on a depth joint distribution adaptation network.

Background technique

Hyperspectral images have high spectral resolution, wide band coverage, and rich spectral detail information of ground objects, which is conducive to fine ground object analysis. Hyperspectral image classification is an important part of hyperspectral image interpretation, which is widely used in mineral exploration, vegetation survey, agricultural monitoring and other fields. Due to the large amount of hyperspectral image data and the existence of redundancy, the same target has spectral differences in different data, which affects the effect of classification.

Hyperspectral image classification is the process of classifying the acquired target spectral signal and other features, and mainly adopts the supervised classification method. Supervised classification models require a large number of training samples in order to obtain excellent classification results. Usually, training samples are obtained by manual annotation, which requires a large labor cost. In practical applications, labeled samples of new remote sensing data are often not easy to obtain. To achieve automatic classification, new remote sensing data should be classified using classifiers trained on other data. However, in the process of hyperspectral image classification, different factors such as sensors and band coverage will affect the data transfer ability of the classifier. A classifier trained on one dataset has higher classification accuracy on that data, but it is difficult to achieve the same effect on other data.

Aiming at the problem of limited data transfer ability in hyperspectral image transfer classification, a domain adaptation model based on transfer learning is proposed to reduce the impact of data differences from different sensors. For example, "Kernel-based domain-invariant feature selection in hyperspectral images for transferlearning" published by Persello C. et al. in IEEE Transactions on Geoscience and Remote Sensing, Vol. 54, Issue 5, in 2016, proposes a The variable feature selection method measures the stability of data migration by calculating the distance of the conditional probability distribution between the source domain and the target domain in the regenerated kernel Hilbert space. The experiment proves the effectiveness of the feature selection of this method. Zhou X. et al. published "Deep feature alignment neural networks for domain adaptation of hyperspectral data" in IEEE Transactions on Geoscience and RemoteSensing, Vol. 56, No. 10 in 2018, and proposed a deep convolutional recurrent neural network to perform source domain and Feature learning and domain adaptation of target-domain hyperspectral images. Experiments show that this method can improve the accuracy of target-domain hyperspectral image classification by using information from source domains. The above methods do not analyze the joint probability distribution of hyperspectral images in the source and target domains, and do not effectively use the joint probability distribution of the data to further improve the transfer classification effect.

SUMMARY OF THE INVENTION

The present invention aims to provide a model capable of obtaining a better migration classification effect, and proposes a hyperspectral image migration classification method based on a deep joint distribution adaptation network.

Technical scheme of the present invention:

The classification method of hyperspectral image transfer based on deep joint distribution adaptation network, the steps are as follows:

(1) Input the hyperspectral images of the source domain and the target domain, and perform feature normalization and dimension unification:

(1a) Normalize the spectral features of the two hyperspectral images in the source domain and the target domain, respectively, so that they are distributed between 0 and 1;

(1b) If the spectral dimensions of the two hyperspectral images are different, zero-pad the hyperspectral image with low dimension to make it dimensionally unified with the image with high dimension;

(2) Combine the features of source and target hyperspectral images:

(2a) Quantize the spectral features of the source domain and target domain hyperspectral images respectively;

(2b) Combine the spectral features of the source domain and the target domain into a vector set;

(3) Construct a three-layer edge probability distribution adaptation network to adapt the edge probability distribution of the source domain and target domain hyperspectral images:

(3a) Use a linear denoising encoder and a nonlinear encoder to construct a first-layer edge probability distribution adaptation network, solve the weights of the linear denoising encoder, and adapt the spectral features of the source domain and the target domain;

(3b) Use a linear denoising encoder and a nonlinear encoder to construct a second-layer edge probability distribution adaptation network, solve the weights of the linear denoising encoder, and adapt the spectral features of the source domain and the target domain;

(3c) Use a linear denoising encoder and a nonlinear encoder to construct a third-layer edge probability distribution adaptation network, solve the weights of the linear denoising encoder, and adapt the spectral features of the source domain and the target domain;

(4) According to the principle of one-to-many classification, select the training samples of hyperspectral images in the source domain and target domain:

For the C class classification, select the source domain hyperspectral image samples and p% target domain hyperspectral image samples, and form C training sample sets according to the one-to-many classification principle. class, and the sample features are the features of the network output through the edge probability distribution adaptation;

(5) Construct C three-layer conditional probability distribution adaptation networks to perform conditional probability distribution adaptation of source domain and target domain hyperspectral images:

(5a) Respectively initialize the weights and bias parameters of C three-layer conditional probability distribution adaptation networks, and use each training sample set to pre-train C networks layer by layer;

(5b) After pre-training, the implicit output of the third-layer conditional probability distribution adaptation network is used as the optimized sample feature;

(5c) In each network, connect the softmax classifier after the third-layer conditional probability distribution is adapted to the network, input the optimized features and category labels of the training samples into the classifier, and optimize the weight and bias parameters of the softmax classifier;

(5d) Fine-tune the C conditional probability distribution adaptation networks from the top to the bottom to further optimize the parameters of each network;

(6) One-to-many classification of target domain hyperspectral images:

(6a) Input the test sample set of the target domain hyperspectral image into C conditional probability distribution adaptation networks respectively, and the optimized feature of the network is the implicit output of the third-layer conditional probability distribution adaptation network;

(6b) Use the trained C softmax classifiers to classify the test samples, and obtain the predicted probabilities of belonging and not belonging to each category;

(6c) Compare the size of the predicted probability belonging to each category, and take the category corresponding to the maximum probability as the predicted label of each sample;

(6d) According to the predicted label vector and the spatial position of the test sample, output the classification result map of the hyperspectral image in the target domain.

Compared with the prior art, the present invention mainly has the following advantages:

First, the present invention proposes a deep joint distribution adaptation network, which uses the edge probability distribution adaptation network and the conditional probability distribution adaptation network respectively to realize the feature adaptation of the hyperspectral images in the source domain and the target domain, and reduce the two. The difference in the joint probability distribution of .

Second, the present invention uses the source domain samples and a small amount of target domain samples to train and learn the classifier, reducing the requirement for training samples in the target domain; The intra- and inter-class discrimination is improved, and the accuracy of transfer classification is improved.

Description of drawings

Fig. 1 is the realization flow chart of the hyperspectral image migration classification;

Figure 2 shows the hyperspectral data of the source domain Pavia University;

Figure 3 is the hyperspectral data of the Pavia Center in the target domain;

Fig. 4 is the classification result graph of using one-to-many softmax classifier without migration;

FIG. 5 is a diagram of the classification result using the method of the present invention.

Detailed ways

The present invention will be described in detail below with reference to specific examples and accompanying drawings.

According to Figure 1, the hyperspectral image migration classification method based on the deep joint distribution adaptation network includes the following steps:

(1) Read the hyperspectral images of the source domain and the target domain, and perform feature normalization and dimension unification:

(1a) Linearly normalize the spectral features of the hyperspectral images in the source and target domains, respectively, so that they are distributed between 0 and 1;

(1b) If the spectral dimensions of the hyperspectral images in the source domain and the target domain are different, zero-padded the hyperspectral image with low dimension to make it dimensionally unified with the image with high dimension;

(2) Combine the features of source and target hyperspectral images:

(2a) The spectral feature vector of the source domain hyperspectral image is quantized as X _S , and the spectral feature vector of the target domain hyperspectral image is quantized as X _T ;

(2b) Combining the spectral features of the source domain and the target domain into a vector set X=[X _S X _T ];

(3) Construct a three-layer conditional probability distribution adaptation network to perform edge probability distribution adaptation of hyperspectral images in the source and target domains:

(3a) Use a linear denoising encoder and a nonlinear encoder to construct the first-layer edge probability distribution adaptation network:

优化权重W的目标方程如下：For the sample x _i , by randomly setting the features of each dimension to zero, M disturbed versions are obtained, of which the m-th disturbed version is x′ _{i,m ;} Implicit to restore the original sample,

The objective equation for optimizing the weight W is as follows:

M个被扰动的样本矩阵组合X′＝[X′₁,X′₂,…,X′_M]，这样上式目标方程转化为Among them, the first term is the average reconstruction error, the second term is the edge penalty term, λ represents the balance factor, N _S and N _T represent the number of samples of source domain hyperspectral and target domain hyperspectral respectively, N is the total number of samples, That is, N=N _S +N _T ; let the perturbed sample matrix X′ _m =[x′ _1,m ,x′ _2,m ,...,x′ _N,m ], M times the original sample matrix

M perturbed sample matrix combinations X′=[X′ ₁ ,X′ ₂ ,...,X′ _M ], so the target equation above is transformed into

where D represents the difference index matrix, defined as

In this way, the linear denoising encoder weight W has the following closed solution:

After that, a nonlinear encoder is used to perform nonlinear mapping on the feature, and the formula is as follows

表示第一层边缘概率分布适配网络的输出；here,

Represents the output of the first-layer edge probability distribution adaptation network;

(3b) According to the above step (3a), use a linear denoising encoder and a nonlinear encoder to construct a second-layer edge probability distribution adaptation network, solve the weight of the linear denoising encoder, and compare the source domain and target. Domain spectral features are adapted to obtain the output of the second layer network

(3c) According to the above step (3a), use a linear denoising encoder and a nonlinear encoder to construct a third-layer edge probability distribution adaptation network, solve the weight of the linear denoising encoder, and compare the source domain and target. Domain spectral features are adapted to obtain the output of the third-layer network

(4) According to the principle of one-to-many classification, select the training samples of hyperspectral images in the source domain and target domain:

For the classification of 9 categories, select 500 samples of each category of source domain hyperspectral images and 1% samples of each category of target domain hyperspectral images, and form 9 training sample sets according to the principle of one-to-many classification. This class (represented as 1) and does not belong to this class (represented as 0), the sample feature is the output of the third layer network

(5) Construct 9 three-layer conditional probability distribution adaptation networks to perform conditional probability distribution adaptation of source domain and target domain hyperspectral images:

(5a) First initialize the weight and bias parameters of the three-layer conditional probability distribution adaptation network, and then use the c-th training sample set to pre-train each layer of the network layer by layer; the encoding process and decoding of the k-th layer of conditional probability distribution adaptation network The process is as follows:

和

分别表示第k层网络的输入、隐含表示和输出，f(·)和g(·)分别为编码和解码的非线性函数，

和

分别为编码过程的权重和偏置，

和

分别为解码过程的权重和偏置，*表示S(源域)或者T(目标域)；in,

and

represent the input, implicit representation and output of the k-th layer network, respectively, f( ) and g( ) are the nonlinear functions of encoding and decoding, respectively,

and

are the weights and biases of the encoding process, respectively,

and

are the weights and biases of the decoding process, respectively, * means S (source domain) or T (target domain);

The objective equations for the pretrained weights and bias parameters are:

N_train表示训练样本数量，λ′表示权重惩罚因子；利用反向传播算法求解上式；Among them, the first item represents the average reconstruction error on the training samples, and the second item is the weight penalty item to prevent the weight from being too large.

N _train represents the number of training samples, λ′ represents the weight penalty factor; use the back propagation algorithm to solve the above formula;

作为优化后的样本特征；(5b) After pre-training, the third-layer conditional probability distribution adapts to the implicit output of the network

as the optimized sample features;

(5c) Connect the softmax classifier after the third-layer conditional probability distribution is adapted to the network, input the optimized features and category labels of the training samples into the classifier, and optimize the weight and bias parameters of the softmax classifier;

(5d) Fine-tune the conditional probability distribution adaptation network from the top layer to the bottom layer to further optimize the parameters of each network; the objective equation for reverse fine-tuning the weight and bias parameters of the entire network is:

和

分别为第c类源域高光谱和目标域高光谱的样本数；同样，利用反向传播算法求解上式，得到微调后的各层条件概率分布适配网络和softmax分类器的参数；Among them, the first term is the total reconstruction error of the coding layer, and the second term is the conditional penalty term, which is used to reduce the difference between the conditional probability distributions of the source domain and the target domain samples;

and

are the sample numbers of the c-th source domain hyperspectral and target domain hyperspectral samples; similarly, the backpropagation algorithm is used to solve the above equation, and the fine-tuned parameters of the conditional probability distribution adaptation network and softmax classifier of each layer are obtained;

(5e) According to the steps of (5a)-(5d) above, 9 three-layer conditional probability distribution adaptation networks are obtained by training 9 training sample sets respectively, so as to realize the conditional probability distribution adaptation of the source domain and target domain hyperspectral images ;

(6) One-to-many classification of target domain hyperspectral images:

(6a) Input the test sample set of the target domain hyperspectral image into the c-th conditional probability distribution adaptation network, and the optimized feature of the network is the implicit output of the third-layer conditional probability distribution adaptation network

(6b) Use the trained softmax classifier to classify the test samples to obtain the predicted probability of belonging to the c-th class:

和

分别对应softmax分类器的部分权重和偏置；in,

and

Corresponding to the partial weights and biases of the softmax classifier, respectively;

(6c) According to the steps of (6a)-(6b) above, input the test sample set of the hyperspectral image in the target domain into 9 conditional probability distribution adaptation networks and softmax classifiers, respectively, to obtain the probability that the sample belongs to each category;

(6d) Compare the predicted probability of each category, and take the category corresponding to the maximum probability as the predicted label of each sample, as follows

(6e) According to the predicted label vector and the spatial position of the test sample, output the classification result map of the hyperspectral image in the target domain.

The technical effects of the present invention are described below through simulation experiments:

1. Simulation conditions and content

The experimental data of the present invention are Pavia University hyperspectral data and Pavia Center hyperspectral data, both of which are acquired by ROSIS sensors. As shown in 2, it is the source domain Pavia University hyperspectral data, with a size of 610 × 340 pixels and 103 bands. Figure 2(a) is the composite image of its 57th, 34th, and 3rd bands, and Figure 2(b) is the corresponding Real-world feature marker map with 9 categories including asphalt, grass, gravel, tree, sheet metal, land, asphalt, brick and shade. As shown in 3, it is the hyperspectral data of the target domain Pavia Center, with a size of 1096 × 715 pixels and 102 bands. Figure 3(a) is the composite image of its 57th, 34th, and 3rd bands, and Figure 3(b) corresponds to Real-world landmark map with 9 categories, including water, trees, grass, bricks, land, asphalt, asphalt, tiles, and shadows. Fig. 4 is the result chart of the classification result of Pavia Center hyperspectral image without migration using one-to-many softmax classifier, Fig. 5 is the result chart of the migration classification of Pavia Center hyperspectral image using the method of the present invention, Table 1 shows the source domain and the target Correspondence of domain training samples and their classification accuracy comparison. In the simulation experiment, both the present invention and the comparison method are implemented by programming in Matlab R2017a.

2. Analysis of simulation results

Table 1 Classification accuracy comparison

It can be seen from Table 1 that the migration classification using the deep joint distribution adaptation network of the present invention obtains higher classification accuracy than the classification without migration, which proves the effectiveness of the method of the present invention in the migration classification. Comparing Fig. 4 and Fig. 5, it can be seen that the classification result map of the method of the present invention has fewer misclassification points, and is closer to the real object marking map. In conclusion, the method of the present invention can effectively improve the effect of hyperspectral image transfer classification.

Claims (1)

1. a kind of hyperspectral image migration classification method based on depth joint distribution adaptation network, is characterized in that, step is as follows: (1) Read the hyperspectral images of the source domain and the target domain, and perform feature normalization and dimension unification: (1a) Linearly normalize the spectral features of the hyperspectral images in the source and target domains, respectively, so that they are distributed between 0 and 1; (1b) If the spectral dimensions of the hyperspectral images in the source domain and the target domain are different, zero-padded the hyperspectral image with low dimension to make it dimensionally unified with the image with high dimension; (2) Combine the features of source and target hyperspectral images: (2a) The spectral feature vector of the source domain hyperspectral image is quantized as X _S , and the spectral feature vector of the target domain hyperspectral image is quantized as X _T ; (2b) Combining the spectral features of the source domain and the target domain into a vector set X=[X _S X _T ]; (3) Construct a three-layer conditional probability distribution adaptation network to perform edge probability distribution adaptation of hyperspectral images in the source and target domains: (3a) Use a linear denoising encoder and a nonlinear encoder to construct the first-layer edge probability distribution adaptation network: For the sample x _i , by randomly setting the features of each dimension to zero, M disturbed versions are obtained, of which the m-th disturbed version is x′ _{i,m ;} Implicit to restore the original sample,

The objective equation for optimizing the weight W is as follows:

Among them, the first term is the average reconstruction error, the second term is the edge penalty term, λ represents the balance factor, N _S and N _T represent the number of samples of source domain hyperspectral and target domain hyperspectral respectively, N is the total number of samples, That is, N=N _S +N _T ; let the perturbed sample matrix X′ _m =[x′ _1,m ,x′ _2,m ,...,x′ _N,m ], M times the original sample matrix

M perturbed sample matrix combinations X′=[X′ ₁ ,X′ ₂ ,...,X′ _M ], so the target equation above is transformed into

where D represents the difference index matrix, defined as

In this way, the linear denoising encoder weight W has the following closed solution:

After that, a nonlinear encoder is used to perform nonlinear mapping on the feature, and the formula is as follows

here,

Represents the output of the first-layer edge probability distribution adaptation network; (3b) According to the above step (3a), use a linear denoising encoder and a nonlinear encoder to construct a second-layer edge probability distribution adaptation network, solve the weight of the linear denoising encoder, and compare the source domain and target. Domain spectral features are adapted to obtain the output of the second layer network

(3c) According to the above step (3a), use a linear denoising encoder and a nonlinear encoder to construct a third-layer edge probability distribution adaptation network, solve the weight of the linear denoising encoder, and compare the source domain and target. Domain spectral features are adapted to obtain the output of the third-layer network

(4) According to the principle of one-to-many classification, select the training samples of hyperspectral images in the source domain and target domain: For the classification of 9 categories, select 500 samples of each category of source domain hyperspectral images and 1% samples of each category of target domain hyperspectral images, and form 9 training sample sets according to the principle of one-to-many classification. The category to which it belongs and the category it does not belong to, the category it belongs to is represented as 1, the category it does not belong to is represented as 0, and the sample feature is the output of the third layer network.

(5) Construct 9 three-layer conditional probability distribution adaptation networks to perform conditional probability distribution adaptation of source domain and target domain hyperspectral images: (5a) First initialize the weight and bias parameters of the three-layer conditional probability distribution adaptation network, and then use the c-th training sample set to pre-train each layer of the network layer by layer; the encoding process and decoding of the k-th layer of conditional probability distribution adaptation network The process is as follows:

in,

and

represent the input, implicit representation and output of the k-th layer network, respectively, f( ) and g( ) are nonlinear functions for encoding and decoding, respectively, W ₁ ^k and

are the weights and biases of the encoding process, respectively,

and

are the weights and biases of the decoding process, respectively, * means S (source domain) or T (target domain); The objective equations for the pretrained weights and bias parameters are:

Among them, the first item represents the average reconstruction error on the training samples, and the second item is the weight penalty item to prevent the weight from being too large.

N _train represents the number of training samples, λ′ represents the weight penalty factor; use the back propagation algorithm to solve the above formula; (5b) After pre-training, the third-layer conditional probability distribution adapts to the implicit output of the network

as the optimized sample features; (5c) Connect the softmax classifier after the third-layer conditional probability distribution is adapted to the network, input the optimized features and category labels of the training samples into the classifier, and optimize the weight and bias parameters of the softmax classifier; (5d) Fine-tune the conditional probability distribution adaptation network from the top layer to the bottom layer to further optimize the parameters of each network; the objective equation for reverse fine-tuning the weight and bias parameters of the entire network is:

Among them, the first term is the total reconstruction error of the coding layer, and the second term is the conditional penalty term, which is used to reduce the difference between the conditional probability distributions of the source domain and the target domain samples;

and

are the sample numbers of the c-th source domain hyperspectral and target domain hyperspectral samples respectively; similarly, the backpropagation algorithm is used to solve the above equation, and the fine-tuned parameters of the conditional probability distribution adaptation network and softmax classifier of each layer are obtained; (5e) According to the steps of (5a)-(5d) above, 9 three-layer conditional probability distribution adaptation networks are obtained by training 9 training sample sets respectively, so as to realize the conditional probability distribution adaptation of the source domain and target domain hyperspectral images ; (6) One-to-many classification of target domain hyperspectral images: (6a) Input the test sample set of the target domain hyperspectral image into the c-th conditional probability distribution adaptation network, and the optimized feature of the network is the implicit output of the third-layer conditional probability distribution adaptation network

(6b) Use the trained softmax classifier to classify the test samples to obtain the predicted probability of belonging to the c-th class:

in,

and

Corresponding to the partial weights and biases of the softmax classifier, respectively; (6c) According to the steps of (6a)-(6b) above, input the test sample set of the target domain hyperspectral image into 9 conditional probability distribution adaptation networks and softmax classifiers, respectively, to obtain the probability that the samples belong to each category; (6d) Compare the predicted probability of each category, and take the category corresponding to the maximum probability as the predicted label of each sample, as follows

(6e) According to the predicted label vector and the spatial position of the test sample, output the classification result map of the hyperspectral image in the target domain.

Images