CN111401455B - Remote sensing image deep learning classification method and system based on Capsules-Unet model - Google Patents

Abstract

The invention discloses a remote sensing image deep learning classification method and system based on the Capsules-Unet model, comprising: performing data preprocessing on remote sensing image data, and dividing the preprocessed remote sensing image data into training set data and verification set data; Take the Unet model as the basic network architecture, fuse the capsule (Capsules) model, and set up the Capsules-Unet model; use the training set data and the verification set data to train the Capsules-Unet model, and obtain the trained Capsules -Unet model; using the trained Capsules-Unet model to classify the remote sensing image data to be classified. The invention establishes a remote sensing image deep learning classification Capsules-Unet model capable of encapsulating multi-dimensional features of ground objects, improves the dynamic routing algorithm of the existing capsule model, and classifies high-resolution remote sensing images more accurately and efficiently.

Description

A remote sensing image deep learning classification method and system based on Capsules-Unet model

Technical Field

The present invention relates to the field of remote sensing image classification, and in particular to a remote sensing image deep learning classification method and system based on a Capsules-Unet model.

Background Art

Classification is a basic problem in the field of remote sensing. Image classification represented by deep convolutional neural networks (CNN) has become a major trend. Compared with traditional remote sensing image classification methods, deep convolutional neural networks do not require hand-crafted features. It is usually composed of multiple consecutive layers that can automatically learn extremely complex hierarchical features from massive data, thus avoiding the problem of highly dependent on expert knowledge to design features. Common convolutional neural networks such as fully convolutional networks (FCN), U-net, generative adversarial networks (GANs) and other cross-connected neural networks have become ideal models for various image classification tasks and have shown great potential in remote sensing applications. For example, in the classification of complex urban landforms, the classification accuracy can reach more than 90% by automatically extracting multi-scale features using convolutional neural networks. The model structure is improved to address the problem that the upsampling method of convolutional neural networks is not fine enough, and the model's ability to distinguish land objects is greatly improved through optimization methods such as data enhancement, multi-scale fusion, post-processing (CRF, voting, etc.), and additional features (elevation information, vegetation index, spectral features).

Despite the great success of convolutional neural networks, there are still some challenging problems in remote sensing classification. The main reasons are as follows:

(1) Remote sensing images are more complex than natural images. Remote sensing images contain various types of objects that vary greatly in size, color, position, and orientation. Spectral characteristics alone may not be sufficient to distinguish objects, and identification features such as spatial position are also required. Therefore, how to combine rich spectral information and spatial information as complementary clues to significantly improve the performance of deep learning in remote sensing classification is a hot topic in current research.

(2) In remote sensing applications, there is often a lack of a large number of labeled data sets. This is not only related to the number of data sets, but also to the difficulty in defining the category labels in the data sets. Currently, complex deep networks require the configuration of a large number of hyperparameters, making the entire network too complex to be optimized. Therefore, training deep neural networks with a small number of data sets will inevitably lead to overfitting.

(3) The current “end-to-end” learning strategy makes it difficult to explain the excellent performance of deep learning in classification tasks. Apart from the final network output, it is difficult to understand the prediction logic hidden inside the network. This brings difficulties to the further mining and processing of remote sensing image classification.

Considering the above points, researchers have proposed many ways to deal with these challenges. Recently, Sabour et al. designed a new neuron, namely "capsule", to replace the traditional scalar neuron to construct a capsule network (CapsNet). A capsule is a carrier that encapsulates multiple neurons. Its output is a high-dimensional vector that can represent various attribute information of the entity, such as posture, lighting, and deformation. The modulus of the vector is used to represent the probability of the entity appearing. The larger the modulus of the high-dimensional vector, the greater the possibility of the entity's existence. The vector can be used to represent the direction and relative relationship of the object in space, which greatly makes up for the shortcomings of the convolutional neural network. The training algorithm of the capsule network is mainly a dynamic routing mechanism between capsules in consecutive layers of the network. The dynamic routing mechanism can improve the robustness of model training and prediction while reducing the number of samples required.

Summary of the invention

The technical problem to be solved by the present invention is to establish a deep learning classification model for remote sensing images that can encapsulate the multi-dimensional features of ground objects so as to more accurately classify high-resolution remote sensing images; to improve the dynamic routing algorithm of the existing capsule model, reduce its computational memory burden and reduce data parameters, so as to more efficiently classify high-resolution remote sensing images.

According to one aspect of the present invention, a deep learning classification method for remote sensing images based on a Capsules-Unet model is provided, comprising: step S1: performing data preprocessing on remote sensing image data, and dividing the preprocessed remote sensing image data into training set data and verification set data; step S2: using the Unet model as the basic network architecture, integrating the capsule model, and establishing a Capsules-Unet model; step S3: using the training set data and the verification set data to train the Capsules-Unet model to obtain the trained Capsules-Unet model; step S4: using the trained Capsules-Unet model to classify the remote sensing image data to be classified.

Optionally, step S3 also includes a step of using the verification set data to verify the Capsules-Unet model, including setting the training iteration to stop when the error is less than a given threshold or the maximum number of iterations is met, thereby completing the training of the Capsules-Unet model.

Optionally, the Capsules-Unet model includes: a feature extraction module, which includes an input convolution layer and a convolution capsule layer (ConCaps), the input convolution layer is used to extract low-level features of the input remote sensing image; the convolution capsule layer (ConCaps) is used to convert the low-level features extracted by the convolution layer into capsules through convolution filtering; a contraction path module, which includes multiple primary capsule layers (PrimaryCaps) for downsampling the capsules obtained by the feature extraction module; an expansion path module, which includes multiple primary capsule layers (PrimaryCaps) and multiple deconvolution capsule layers (DeconCaps), the multiple primary capsule layers (PrimaryCaps) and the multiple deconvolution capsule layers (DeconCaps) are interlaced with each other, and are used to upsample the capsules from the contraction path module; and also includes an output primary capsule layer for convolution processing the data obtained by the upsampling processing in the expansion path module and outputting it to the classification module; a skip connection layer (skip connection layer); layer), the expansion path module cuts and copies the low-level features in the contraction path module through a skip connection layer for upsampling in the expansion path module; a classification module, which includes a classification capsule layer (Class Capsule), the classification capsule layer includes a plurality of capsules, and the activation vector modulus of each capsule in the plurality of capsules is used to calculate the probability of the existence of an instance of each class.

Optionally, in step S3 and step S4, an improved local constrained dynamic routing algorithm is adopted in the Capsules-Unet model, so that when the data of the child capsule is routed to the data of the parent capsule of the next layer, the same transformation matrix is adopted for the child capsule and the parent capsule of the same type.

Optionally, in step S3 and step S4, an improved local constrained dynamic routing algorithm is adopted in the Capsules-Unet model, so that when the data of a child capsule is routed to the data of a parent capsule of the next layer, the child capsule is routed to the parent capsule only in a defined local window.

According to another aspect of the present invention, a remote sensing image deep learning classification system based on a Capsules-Unet model is provided, comprising: a data preprocessing unit, used to perform data preprocessing on remote sensing image data, and divide the preprocessed remote sensing image data into training set data and verification set data; a model building unit: using the Unet model as the basic network architecture, integrating the capsule (Capsules) model, and building a Capsules-Unet model; a model training unit: using the training set data and the verification set data to train the Capsules-Unet model to obtain the trained Capsules-Unet model; a classification unit: using the trained Capsules-Unet model to classify the remote sensing image data to be classified.

Optionally, the model training unit also includes a model verification subunit, which is used to verify the Capsules-Unet model using the verification set data, including setting the training iteration to stop when the error is less than a given threshold or the maximum number of iterations is met, thereby completing the training of the Capsules-Unet model.

Optionally, the Capsules-Unet model in the above system includes: a feature extraction module, which includes an input convolution layer and a convolution capsule layer (ConCaps), wherein the input convolution layer is used to extract low-level features of the input remote sensing image; the convolution capsule layer (ConCaps) is used to convert the low-level features extracted by the convolution layer into capsules through convolution filtering; a contraction path module, which includes multiple primary capsule layers (PrimaryCaps) for downsampling the capsules obtained by the feature extraction module; an expansion path module, which includes multiple primary capsule layers (PrimaryCaps) and multiple deconvolution capsule layers (DeconCaps), wherein the multiple primary capsule layers (PrimaryCaps) and the multiple deconvolution capsule layers (DeconCaps) are interlaced with each other and are used to upsample the capsules from the contraction path module; and also includes an output primary capsule layer for convolutionally processing the data obtained by the upsampling processing in the expansion path module and outputting it to the classification module; a skip connection layer (skip connection layer); layer), the expansion path module cuts and copies the low-level features in the contraction path module through a skip connection layer for upsampling in the expansion path module; a classification module, which includes a classification capsule layer (Class Capsule), the classification capsule layer includes a plurality of capsules, and the activation vector modulus of each capsule in the plurality of capsules is used to calculate the probability of the existence of an instance of each class.

Optionally, in the model training unit and the classification unit, an improved local constrained dynamic routing algorithm is adopted in the Capsules-Unet model, so that when the data of the child capsule is routed to the data of the parent capsule of the next layer, the same transformation matrix is adopted for the child capsule and the parent capsule of the same type.

Optionally, in the model training unit and the classification unit, an improved local constrained dynamic routing algorithm is adopted in the Capsules-Unet model, so that when the data of a child capsule is routed to the data of a parent capsule of the next layer, the child capsule is routed to the parent capsule only in a defined local window.

The beneficial technical effects of the technical solution of the present invention are: establishing a Capsules-Unet model for deep learning classification of remote sensing images that can encapsulate the multi-dimensional features of ground objects, and more accurately classifying high-resolution remote sensing images; improving the dynamic routing algorithm of the existing capsule model, reducing its computational memory burden and reducing data parameters, and more efficiently classifying high-resolution remote sensing images.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a flow chart of a remote sensing image deep learning classification method according to the present invention.

FIG2 is a flowchart of a remote sensing image deep learning classification method according to an embodiment of the present invention.

FIG3 is a schematic diagram of the Capsules-Unet model structure of the present invention.

FIG. 4 is a further detailed structural diagram of the Capsules-Unet model structure of FIG. 3 of the present invention.

FIG5 is a schematic diagram of the structure of a remote sensing image deep learning classification system according to the present invention.

FIG6 is a schematic diagram of an improved local constraint dynamic routing algorithm of the Capsules-Unet model of the present invention.

FIG. 7 is a schematic diagram of the process of updating the coupling coefficient of the improved local constraint dynamic routing algorithm of the Capsules-Unet model of the present invention.

DETAILED DESCRIPTION

Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of systems consistent with some aspects of the present invention as detailed in the appended claims.

As shown in Figure 1, according to one aspect of the present invention, the present invention provides a remote sensing image deep learning classification method based on the Capsules-Unet model, including: step S1: performing data preprocessing on the remote sensing image data, and dividing the preprocessed remote sensing image data into training set data and verification set data; step S2: using the Unet model as the basic network architecture, integrating the capsule (Capsules) model, and establishing the Capsules-Unet model; step S3: using the training set data and the verification set data to train the Capsules-Unet model to obtain the trained Capsules-Unet model; step S4: using the trained Capsules-Unet model to classify the remote sensing image data to be classified.

Data preprocessing

The high-resolution remote sensing image data of the present invention comes from the Vaihingen and Potsdam data sets of ISPRS 2D semantic labeling. The ISPRS Vaihingen data set contains 33 orthorectified images of different sizes and ground truth data, all of which are taken from the Vaihingen area of Germany. Its ground sampling interval is 9 cm and it consists of three channels (IRRG) of near infrared, infrared and green. There are 6 types of ground objects in the data, namely impermeable layer, building, low vegetation, tree, vehicle and others (background). During model training, the images with real label data are divided into training set and verification set, 31 images are selected as training set and the remaining 2 images are selected as verification set.

The Potsdam dataset contains 38 orthorectified images of different sizes and the corresponding digital surface models (DSM). The DSM is an array of the same size as the input image, providing an elevation value at each pixel. The images were taken in the Potsdam area of Germany, with a ground sampling interval of 5 cm and composed of four channels: near infrared, infrared, green, and blue (IRRGB). In the experiment, in order to increase the number of bands in the Potsdam dataset, the NDVI index was calculated using the infrared light band. The dataset provides real labeled data of 24 images for model training and validation. The Potsdam dataset has the same ground object categories as the Vaihingen dataset. During model training, 23 images were selected as the training set, and the remaining 1 was used as the validation set. The image numbers of the labeled data divided into training samples and test samples in the two datasets are shown in Table 1.

Table 1. Image numbers of labeled data divided into training samples and test samples

See Figure 2. In the data preprocessing step, the original image is sampled into 64×64 pixels using a random sliding window sampling method. 80% of the samples in the sample library are used as training samples, and 20% are used as test samples. In order to increase the diversity and variability of the samples, the model inputs a large amount of labeled training data into the model through normalization, random sampling and data augmentation.

Capsules-Unet model establishment

The present invention is based on a new neuron, namely "capsule", designed by Sabour et al. to replace the traditional scalar neuron to construct a capsule network. The capsule network consists of multiple layers of capsules. Each capsule contains two components: weight and coupling coefficient. Each weight matrix W represents a linear transformation, carrying the spatial relationship and other important relationships between low-level features and high-level features, such as posture (position, size, direction), deformation, speed, hue, texture, etc. The coupling coefficient c _ij determines which high-level capsule the output of a low-level capsule is sent to. Unlike the weight, c _ij is updated by dynamic routing. Therefore, the coupling coefficient essentially determines how information flows between capsules. The sum of c _ij between each low-level capsule and all potential high-level capsules is 1. The capsule network is built on the basis of capsules and aims to overcome the shortcomings of traditional CNNs that cannot recognize the posture information of entities and the lack of part-whole relationship between objects.

The present invention proposes a high spatial resolution image classification model Capsules-Unet based on the U-net model. The present invention uses the concept and structure of capsules to design a classification model, in order to improve the classification performance through the viewpoint invariance and interoperability mechanism of the capsule model.

Referring to FIG. 3 and FIG. 4 , the Capsules-Unet model of the present invention comprises: a feature extraction module 1, which comprises an input convolution layer 11 and a convolution capsule layer (ConCaps) 12, wherein the input convolution layer 11 is used to extract low-level features of an input remote sensing image; the convolution capsule layer (ConCaps) 12 is used to convert the low-level features extracted by the convolution layer 11 into capsules through convolution filtering; a contraction path module 2, which comprises a plurality of primary capsule layers 21 (PrimaryCaps) for performing convolution filtering on the capsules obtained by the feature extraction module 1. downsampling processing; an expansion path module 3, which includes a plurality of primary capsule layers (PrimaryCaps) 31 and a plurality of deconvolution capsule layers (DeconCaps) 32, the plurality of primary capsule layers (PrimaryCaps) and the plurality of deconvolution capsule layers (DeconCaps) are staggered to form an upsampling processing on the capsules from the contraction path module; it also includes an output primary capsule layer 33, which is used to convolve the data obtained by the upsampling processing in the expansion path module 3 and output it to the classification module 5; a skip connection layer (skip layer) 4, the expansion path module 3 cuts and copies the low-level features in the contraction path module 2 through the skip connection layer 4, and is used for upsampling processing in the expansion path module 3; a classification module 5, which includes a classification capsule layer (Class Capsule) 52, the classification capsule layer 52 includes a plurality of capsules, and the activation vector modulus of each capsule in the plurality of capsules is used to calculate the probability of the existence of an instance of each class.

According to an optional embodiment of the present invention, the input convolution layer 11 includes 16 5×5 convolution kernels with a stride of 1, and its input is a 64×64×3 image and its output is a 64×64×1×16 tensor. The functions of the primary capsule layers (PrimaryCaps) 21, 31 are similar to the convolution layers of CNN in many aspects. However, they take capsules as input and use a local constrained dynamic routing algorithm to determine the output, and the output result is also a capsule. The deconvolution capsule layer (DeconCaps) 32 uses transposed convolution to compensate for the global connectivity loss caused by local constrained dynamic routing. The classification capsule layer (Class Capsule) 52 may include k capsules (corresponding to k classification categories). The activation vector modulus of each capsule calculates the probability of the existence of an instance of each class. Each capsule in the previous layer is fully connected to the capsule in this layer.

As shown in Figure 4, the input is a 64×64 multi-band remote sensing image. First, starting from the input convolution layer 11 of the feature extraction module 1, 16 convolution kernels of size 5x5 and stride 1 are used, and the output is a tensor of 64×64×1×16. The input convolution layer 11 produces 16 feature maps with the same spatial dimensions. The 16 basic features detected by the input convolution layer 11 are converted into the capsule output of the first set of feature combinations through the capsule convolution kernels of the 2 5×5 convolution capsule layers 12 with a stride of 2, and the output size is 32×32×2×16. The feature extraction module 1 can capture the difference features of the input data and input them into the subsequent modules. Next, with the basic architecture of the Unet model, its architecture consists of a contraction path module 2 and an expansion path module 3. The contraction path module 2 consists of 4 consecutive primary capsule layers (PrimaryCaps) 21 to downsample the image. That is, capsule convolution of 5×5×4×16 (4 capsules, 16 feature maps) with a stride of 1 + capsule convolution of 5×5×4×16 (4 capsules, 16 feature maps) with a stride of 2 + capsule convolution of 5×5×8×16 (8 capsules, 16 feature maps) with a stride of 1 + capsule convolution of 5×5×8×32 (8 capsules, 32 feature maps) with a stride of 2. When reaching the bottom layer, the image size is 8×8×8×32. By adding capsule layers instead of convolutional layers to Unet, part-whole context information is retained. The extended path module 3 is composed of 3 primary capsule layers (PrimaryCaps) 31 and 3 deconvolution capsule layers (DeconCaps) 32 alternately, and also includes an output primary capsule layer 33. That is, capsule convolution of 5×5×8×32 (8 capsules, 32 feature maps) with a step size of 1 + deconvolution capsule of 4×4×8×32 (8 capsules, 32 feature maps) + capsule convolution of 5×5×4×32 (4 capsules, 32 feature maps) with a step size of 1 + deconvolution capsule of 4×4×4×16 (4 capsules, 16 feature maps) + capsule convolution of 5×5×4×16 (4 capsules, 16 feature maps) with a step size of 1 + deconvolution capsule of 4×4×2×16 (2 capsules, 16 feature maps). When reaching the top layer, that is, after the third deconvolution, the image becomes 64×64×2×16 in size, and then the data after upsampling processing in the extension path module 3 is convolved through the output main capsule 33 and output to the classification module 5. The features in the contraction path module 2 are cropped and copied through the skip connection layer 4 so as to be correspondingly upsampled in the expansion path module 3. The classification module 5 includes a classification capsule layer 52, and the output main capsule layer 33 performs convolution processing on the data after upsampling processing in the expansion path module 3 and outputs it to the classification capsule layer 52; the classification capsule layer 52 includes a plurality of capsules, and the activation vector modulus of each capsule in the plurality of capsules is used to calculate the probability of the existence of an instance of each class.

Model training and validation

The computing environment of the embodiment of the present invention is based on NVIDIA Quadro P600 GPUs and Keras deep learning platform. The Capsules-Unet model is trained on a computer with 3.7GHz 8-core CPU and 32GB memory.

Referring to Figure 2, during the model training phase, all training set data are input into the Capsules-Unet model, and the validation set data is used to judge the operation results of the Capsules-Unet model under different parameter values. The maximum number of training cycles is set to 10,000. The batch input size of each training step is 30, that is, 30 samples are input each time to fit the Capsules-Unet model. The initial learning rate is 0.001. The Adam optimization method is used to update all parameters of the Capsules-Unet model. When the error is less than a given threshold or the maximum number of iterations is met, the above iteration stops.

Classification

In the classification stage, the trained Capsules-Unet model is used to classify the data to be classified to generate preliminary category prediction data. The prediction data is then input into the classification capsule layer (ClassCapsule) 52 to generate the final feature category according to the probability of the predicted category.

For classification tasks involving k classes, the Class Capsule layer 52 has k capsules, each capsule representing a class. Since the length of the capsule vector output indicates the presence of a visual entity, the length of each capsule in the last layer (||v _c ||) represents the probability of class capsule k. For each predefined class, there is a contribution term L _k in the marginal loss, which is similar to Softmax applied to multi-classification tasks.

L _k =T _k max(0, m ⁺ -||v _c ||) ² +λ _margin (1-T _k )max(0, ||v _c ||-m ^- ) (1)

Where m ⁺ = 0.9, m ^- = 0.1 and λ _margin = 0.5. _{T k} is the indicator function of the classification, that is, 1 when k exists and 0 when it does not exist. ||v _c || represents the output probability of the capsule; m ⁺ is the upper bound, which penalizes false positives, that is, predicting that class k exists but does not exist; m ^- is the lower bound, which penalizes false negatives, that is, predicting that class k does not exist but does exist; λ _margin is the proportional coefficient, which adjusts the proportion of the two.

Referring to FIG5 , according to another aspect of the present invention, a remote sensing image deep learning classification system based on a Capsules-Unet model is provided, including: a data preprocessing unit 101, for performing data preprocessing on remote sensing image data, and dividing the preprocessed remote sensing image data into training set data and verification set data; a model building unit 102: using the Unet model as the basic network architecture, integrating the capsule (Capsules) model, and establishing a Capsules-Unet model; a model training unit 103: using the training set data and the verification set data to train the Capsules-Unet model to obtain the trained Capsules-Unet model; a classification unit 104: using the trained Capsules-Unet model to classify the remote sensing image data to be classified.

According to an optional embodiment of the present invention, the model training unit 103 also includes a model verification subunit, which is used to verify the Capsules-Unet model using the verification set data.

Local Constraint Dynamic Routing Algorithm

The original capsule network and dynamic routing algorithm require a lot of memory and the model operation is extremely time-consuming. This is because when the dynamic routing algorithm determines the coefficients of the child capsule routing to the parent capsule in the next layer, an additional intermediate representation is required to store the output of the child capsule in a given layer. Therefore, in order to solve the problems of excessive memory burden and parameter explosion, the present invention proposes an improved local constraint dynamic routing algorithm based on the original dynamic routing algorithm.

According to an embodiment of the improved local constrained dynamic routing algorithm of the present invention, when the data of the child capsule (capsule of the current layer) of the Capsules-Unet model is routed to the data of the parent capsule (capsule of the next layer), the same conversion matrix is used for the child capsule and the parent capsule of the same type.

对于每一个

都存在一个h^l×w^l _×z ^l的子胶囊

其中h^l×w^l是l-1层输出的大小。在网络的l+1层有一组胶囊类型

对于每一个

存在h^l+1×w^l+1×z^l+1的父胶囊

其中h^l+1×w^l+1是l层输出的大小。As shown in Figure 6. There is a set of capsule types in layer l

For each

There exists a subcapsule of h ^l ×w ^l _×z ^l

Where h ^l × w ^l is the size of the output of layer l-1. In the l+1 layer of the network there is a set of capsule types

For each

There exists a parent capsule of h ^l+1 ×w ^l+1 ×z ^l+1

where h ^l+1 ×w ^l+1 is the size of the output of layer l.

该集合被定义为在第l层中以(x,y)为中心的核内，变换矩阵

和子胶囊的输出

之间的矩阵乘法，即对于任意的

存在

因此，我们可以看到每个

都有形状k_h×k_w×z^l，其中kh×kw是自定义内核的大小。对于所有的胶囊类型T^l，每个

的形状都是kh×kw×z^l×|T^l+1|×z^l+1，其中|T^l+1|是l+1层中的父胶囊类型数。值得注意的是每个

与空间位置(x,y)无关，因为相同的变换矩阵在指定胶囊类型的所有空间位置上共享。简单来讲，局部约束动态路由是在底层的每一个胶囊内做卷积，每一个胶囊都卷积出与高层的所有胶囊维度相同的张量，而后对于每一个底层胶囊卷积的结果做路由选择。这是本文可以利用矩阵共享来显著减少参数数量的原因。Taking the convolutional capsule layer 12 as an example, each parent capsule p _xy ∈ P receives a set of “prediction vectors”,

This set is defined as the transformation matrix within the kernel centered at (x,y) in layer l.

Output of the and sub-capsules

The matrix multiplication between

exist

Therefore, we can see that each

have the shape k _h × k _w × z ^l , where kh × kw is the size of the custom kernel. For all capsule types T ^l , each

The shape of is kh×kw×z ^l ×|T ^l+1 |×z ^l+1 , where |T ^l+1 | is the number of parent capsule types in the l+1 layer. It is worth noting that each

It has nothing to do with the spatial position (x, y) because the same transformation matrix is shared across all spatial positions of a given capsule type. Simply put, local constrained dynamic routing is to perform convolutions in each capsule at the bottom layer, and each capsule convolves a tensor with the same dimension as all capsules at the top layer, and then performs routing selection for the result of the convolution of each bottom-layer capsule. This is why this paper can use matrix sharing to significantly reduce the number of parameters.

上的加权和，其中

是由局部约束动态路由算法确定的路由系数。这些路由系数由“RoutingSoftmax”计算，To determine the final input for each parent capsule _pxy∈P , these “prediction vectors” are calculated

The weighted sum of

are the routing coefficients determined by the local constrained dynamic routing algorithm. These routing coefficients are calculated by "RoutingSoftmax",

是子胶囊路由到父胶囊p_xy的先验概率，初始化为0，

的迭代更新方式如下：in

is the prior probability of the child capsule being routed to the parent capsule p _xy , initialized to 0,

The iterative update method is as follows:

的计算与当前的输入图像无关，它取决于两个胶囊的位置和类型，然后通过测量上述层中的每个胶囊的当前输出v_xy与预测向量

之间的一致性来迭代地改进初始耦合系数。

The calculation of is independent of the current input image, it depends on the position and type of the two capsules, and then by measuring the current output v _xy of each capsule in the above layer with the predicted vector

The initial coupling coefficient is iteratively improved by comparing the consistency between them.

In the Capsulse-Unet model, because it is transmitted in the form of vectors in the previous layer network, the direction of the "capsule" must be processed when activating. The activation function of the Capsulse-Unet model is named Squashing, and the expression is shown in formula (4):

Where p _xy and v _xy represent the input vector of capsule j and the output vector at the spatial position (x, y) respectively. p _xy is actually the weighted sum of all vectors output to capsule j in the previous layer. The physical meaning of the first part of the formula is the scaling of the input vector p _xy , and the second part represents the unit vector of p _xy . The squashing function ensures that the value range of the input vector is between 0 and 1 while also preserving the direction of the input vector. When ||p _xy || is zero, the value of v _xy is close to 0; when ||p _xy || is infinite, v _xy is infinitely close to 1.

的值都被初始化为零，因此胶囊i到层l+1中所有胶囊的耦合系数都相等，然后对接收到的所有输入

进行加权求和得到p_xy，其中权值为各耦合系数

接着p_xy继续按照公式(4)中的Squashing函数进行非线性变换得v_xy，最后遵循

的原则对

进行更新，执行完r次的迭代后才能返回胶囊的输出v_xy。通常，实验中的最佳迭代次数为3。The dynamic process of updating the coupling coefficients between capsules by local constrained dynamic routing is shown in Figure 7. In the first iteration of dynamic routing, due to the temporary variable

The values of are initialized to zero, so the coupling coefficients from capsule i to all capsules in layer l+1 are equal, and then for all received inputs

Perform weighted summation to obtain p _xy , where the weights are the coupling coefficients

Then p _xy continues to be nonlinearly transformed according to the Squashing function in formula (4) to obtain v _xy , and finally follows

The principle of

After updating, the output v _xy of the capsule can be returned after r iterations. Usually, the best number of iterations in the experiment is 3.

According to another embodiment of the improved local constraint dynamic routing algorithm of the present invention, in the model training and classification steps, when the data of the child capsule of the Capsules-Unet model is routed to the data of the parent capsule of the next layer, the child capsule is routed to the parent capsule only in a defined local window. As shown in FIG5 , only the local window area k _h × k _w × z ^l of the capsule of the lth layer is sampled, rather than the whole capsule being operated, thereby greatly reducing the amount of operation, reducing its operation memory burden and reducing data parameters. Although the local window sampling may cause some data information to be distorted, the jump connection layer in the Capsules-Unet model can cut and copy the low-level features in the contraction path module 2, thereby compensating for some data information distortion to a certain extent.

Evaluation Method

In the present invention, the overall accuracy (OA) and Kappa coefficient are used to evaluate the classification effect of the method of the present invention. The overall accuracy is the percentage of correctly classified images in all test sets. The Kappa coefficient is another widely used evaluation criterion, which is based on the confusion matrix to evaluate the accuracy of remote sensing classification. The equation is as follows:

where N is the total number of samples, n is the number of categories, _Mij is the (i, j)th value of the confusion matrix C[41], and Mi ₊ and M _+i represent the sum of the i-th row and i-th column of C, respectively.

Comparative Analysis

The present invention compares the Capsules-Unet model with CapsNet and Unet, and obtains the following results: Both the Capsules-Unet model and CapsNet achieve good results in the classification of impermeable layers, buildings and low vegetation. These four types of objects have good regional connectivity and clear object edges. However, vehicles and other unique categories with smaller coverage are not well identified, and the mixed classification phenomenon with buildings is very obvious. In the Vaihingen data set, the homogeneous area is small, and it is not accurate enough when performing feature extraction. Although the Capsules-Unet model and the CapsNet model proposed in the present invention both use the "capsule" form to retain the spatial information between objects, when the homogeneous area is small, the spatial information of small target areas such as cars is often limited. The Unet model has a good effect on the classification of impermeable layers and low vegetation, but the boundaries of buildings are unclear.

Table 2 lists the classification accuracy of the three models on the Vaihingen dataset. It can be seen from the table that the method of the present invention achieves very high classification performance. From the perspective of overall accuracy (OA), the overall accuracy (OA) of Capsules-Unet is 1.22% higher than that of CapsNet and 1.89% higher than that of Unet. The Kappa coefficients of Capsules-Unet, CapsNet and Unet models are 0.74, 0.74 and 0.72, respectively. The classification accuracy of Capsules-Unet in the Vaihingen dataset is slightly better than that of CapsNet and Unet.

Table 2. Accuracy evaluation results of Vaihingen dataset and Potsdam dataset

The features and benefits of the present invention are explained by reference to the embodiments. Accordingly, the present invention should expressly not be limited to these exemplary embodiments which illustrate some possible non-limiting combinations of features, which may exist alone or in other combinations of features.

The above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims (8)

1. A remote sensing image deep learning classification method based on the Capsules-Unet model, comprising: Step S1: preprocessing the remote sensing image data, dividing the preprocessed remote sensing image data into training set data and verification set data; Step S2: Taking the Unet model as the basic network architecture, integrating the capsule model, and establishing the Capsules-Unet model; The Capsules-Unet model includes: A feature extraction module, comprising an input convolution layer and a convolution capsule layer, wherein the input convolution layer is used to extract low-level features of an input remote sensing image; the convolution capsule layer is used to convert the low-level features extracted by the convolution layer into capsules through convolution filtering; A contraction path module, comprising a plurality of main capsule layers, for downsampling the capsules obtained by the feature extraction module; an expansion path module, comprising a plurality of main capsule layers and a plurality of deconvolution capsule layers, the plurality of main capsule layers and the plurality of deconvolution capsule layers being interlaced with each other, and configured to upsample capsules from the contraction path module; and an output main capsule layer, configured to convolute the data obtained by the upsampling process in the expansion path module and output the convolution process to the classification module; A skip connection layer, wherein the expansion path module cuts and copies low-level features in the contraction path module through the skip connection layer for upsampling processing in the expansion path module; A classification module, comprising a classification capsule layer, wherein the classification capsule layer comprises a plurality of capsules, wherein the activation vector modulus of each capsule in the plurality of capsules is used to calculate the probability of the existence of an instance of each class; Step S3: training the Capsules-Unet model using the training set data and the validation set data to obtain the trained Capsules-Unet model; Step S4: Classify the remote sensing image data to be classified using the trained Capsules-Unet model. 2. The remote sensing image deep learning classification method according to claim 1 is characterized in that: in step S3, it also includes a step of using the verification set data to verify the Capsules-Unet model, including setting the training iteration to stop when the error is less than a given threshold or meets the maximum number of iterations, thereby completing the training of the Capsules-Unet model. 3. The remote sensing image deep learning classification method according to claim 1 is characterized in that: in step S3 and step S4, an improved local constraint dynamic routing algorithm is adopted in the Capsules-Unet model, so that when the data of the child capsule is routed to the data of the parent capsule of the next layer, the same transformation matrix is used for the child capsule and the parent capsule of the same type. 4. The remote sensing image deep learning classification method according to claim 1 or 3, characterized in that: in step S3 and step S4, an improved local constrained dynamic routing algorithm is adopted in the Capsules-Unet model, so that when the data of the child capsule is routed to the data of the parent capsule of the next layer, the child capsule is routed to the parent capsule only in a defined local window. 5. A remote sensing image deep learning classification system based on the Capsules-Unet model, comprising: A data preprocessing unit is used to perform data preprocessing on remote sensing image data, and divide the preprocessed remote sensing image data into training set data and verification set data; Model building unit: Using the Unet model as the basic network architecture, integrating the capsule model, and building the Capsules-Unet model; The Capsules-Unet model includes: A feature extraction module, comprising an input convolution layer and a convolution capsule layer, wherein the input convolution layer is used to extract low-level features of an input remote sensing image; the convolution capsule layer is used to convert the low-level features extracted by the convolution layer into capsules through convolution filtering; A contraction path module, comprising a plurality of main capsule layers, for downsampling the capsules obtained by the feature extraction module; an expansion path module, comprising a plurality of main capsule layers and a plurality of deconvolution capsule layers, the plurality of main capsule layers and the plurality of deconvolution capsule layers being interlaced with each other, and configured to upsample capsules from the contraction path module; and an output main capsule layer, configured to convolute the data obtained by the upsampling process in the expansion path module and output the convolution process to the classification module; A skip connection layer, wherein the expansion path module cuts and copies low-level features in the contraction path module through the skip connection layer for upsampling processing in the expansion path module; A classification module, comprising a classification capsule layer, wherein the classification capsule layer comprises a plurality of capsules, wherein the activation vector modulus of each capsule in the plurality of capsules is used to calculate the probability of the existence of an instance of each class; Model training unit: training the Capsules-Unet model using the training set data and the validation set data to obtain the trained Capsules-Unet model; Classification unit: Use the trained Capsules-Unet model to classify the remote sensing image data to be classified. 6. The remote sensing image deep learning classification system according to claim 5 is characterized in that: the model training unit also includes a model verification subunit, which is used to verify the Capsules-Unet model using the verification set data, including setting the training iteration to stop when the error is less than a given threshold or meets the maximum number of iterations, thereby completing the training of the Capsules-Unet model. 7. The remote sensing image deep learning classification system according to claim 5 is characterized in that: in the model training unit and the classification unit, an improved local constraint dynamic routing algorithm is adopted in the Capsules-Unet model, so that when the data of the child capsule is routed to the data of the parent capsule of the next layer, the same transformation matrix is used for the child capsule and the parent capsule of the same type. 8. The remote sensing image deep learning classification system according to claim 5 or 7, characterized in that: in the model training unit and the classification unit, an improved local constrained dynamic routing algorithm is adopted in the Capsules-Unet model, so that when the data of a child capsule is routed to the data of a parent capsule of the next layer, the child capsule is routed to the parent capsule only in a defined local window.

Images