CN114332466B - Continuous learning method, system, equipment and storage medium for image semantic segmentation network - Google Patents

Abstract

The invention discloses a continuous learning method, system, equipment and storage medium for image semantic segmentation network. On the one hand, the method of extracting old knowledge representations and aligning them through nonlinear transformation in feature space effectively maintains the invariance of old knowledge and improves the Ability to learn new knowledge. On the other hand, the topology structure of the new category is optimized in the embedding space, and the invariance of its topology structure is maintained for the old category, so as to reduce forgetting and prevent confusion between classes. This makes it unnecessary to provide labels of old categories in the continuous learning of semantic segmentation, reducing the cost of labeling. In general, as a universal continuous learning method for semantic segmentation, the present invention has no restrictions on application scenarios, and has strong generalization ability and practical value.

Description

Image semantic segmentation network continuous learning method, system, device and storage medium

technical field

The present invention relates to the technical field of image semantic segmentation, in particular to a continuous learning method, system, device and storage medium for image semantic segmentation network.

Background technique

In recent years, deep neural networks have achieved great success in semantic segmentation tasks. However, the traditional semantic segmentation network training method needs to obtain all the training data at one time, and it is difficult to update after the training is completed. In practical applications, the network is often required to gradually learn and update the learned knowledge from the data stream, thereby effectively reducing the cost of data storage and training. But having deep neural networks learn directly on new data can lead to severe forgetting of what has been learned. The continuous learning technology imposes additional constraints on the learning process to achieve the purpose of learning new knowledge without forgetting the learned knowledge.

A general approach to continuous learning is to use knowledge distillation to maintain knowledge consistency between old and new networks. This operation is usually performed in the output space or feature space. Specific to the field of semantic segmentation, in addition to using the above methods to prevent the forgetting of old knowledge, there are two new challenges: First, as learning progresses, it may be necessary to learn categories that have been ignored in the past, resulting in The semantic information it contains is not static, which requires the network to have a stronger ability to learn new knowledge. Secondly, since obtaining labeled data requires a lot of manpower and material resources, it is hoped that only the categories that need to be learned will be labeled on the newly added data, which will cause the regions labeled as background classes may contain learned categories, and the introduced semantic inconsistency will give Network training brings greater challenges. Therefore, continuous learning methods in the field of image classification are not competent for the continuous learning task of semantic segmentation.

Specifically: In the Chinese patent application "Continuous Learning Framework and Continuous Learning Method for Deep Neural Networks" with the publication number of CN111191709A, a generation network is used to generate data of old categories, and it is mixed with new data to train the network, but it only solves the problem of image classification tasks. In addition, this method relies heavily on the generation quality of the generator, and is incompetent for large-scale and complex data, especially the task of image semantic segmentation. In the Chinese patent application "An Incremental Learning Method for Image Classes Based on Single Classification Technology" with the publication number of CN111368874A, two methods of knowledge distillation and preference correction of the output space are used to realize the continuous learning of image classification tasks. However, it still cannot solve the unique challenges in the aforementioned continuous learning task of semantic segmentation, so it cannot be directly applied in image semantic segmentation. The Chinese patent application "Face Detection System and Method Based on Incremental Learning" with the publication number CN103366163A, the Chinese patent application with the publication number CN106897705A "A method for the distribution of ocean observation big data based on the incremental learning", and the publication number of CN103593680A's Chinese patent application "A Dynamic Gesture Recognition Method Based on Hidden Markov Model Self-Incremental Learning" is a method dedicated to a specific field, and it cannot be proved that it has generalization and universality.

Therefore, it is of great practical value and practical significance to design a general method for the continuous learning task of semantic segmentation, which can reduce the forgetting of old knowledge as much as possible while solving the semantic inconsistency in the continuous learning task of semantic segmentation.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide an image semantic segmentation network continuous learning method, system, equipment and storage medium, which have no restrictions on application scenarios, have strong generalization ability and practical value, and fill the gap in the continuous learning task of semantic segmentation. .

The purpose of this invention is to realize through the following technical solutions:

A continuous learning method for image semantic segmentation network, including:

Obtaining the newly added semantic segmentation data set and the label corresponding to the newly added category, using the original image semantic segmentation network to extract the original feature map of the image data in the newly added semantic segmentation data set, and transforming the original feature map through a feature transformation module, and utilize the difference between the reconstructed feature map of the transformation result and the original feature map to preliminarily train the feature transformation module;

Use the original image semantic segmentation network and the initially trained feature transformation module to initialize an identical image semantic segmentation network and feature transformation module, the original image semantic segmentation network is called the old network, and the initially trained feature transformation module is called the old network. Feature transformation module, the image semantic segmentation network generated by initialization is called a new network, and the feature transformation module generated by initialization is called a new feature transformation module; the old network and the old feature transformation module are fixed, and the new network and the new feature transformation module are trained. ;

During training, the image data of the newly added semantic segmentation dataset is input into the old network and the new network at the same time, and feature map extraction, decoding and semantic segmentation are performed in the old network and the new network respectively to obtain segmentation results; The feature map extracted by the old network is transformed by the old feature transformation module, the feature map extracted by the new network is transformed by the new feature transformation module, and the alignment loss of the two transformation results is calculated; using the old network With the segmentation result of the new network and the eigenvector obtained by decoding, the corresponding inter-class relationship matrix and intra-class relationship set are independently constructed for the old class, and the inter-class structure retention loss is calculated by using the inter-class relationship matrix of the old network and the new network. , the intra-class structure preservation loss is calculated by using the intra-class relationship set of the old network and the new network, and the inter-class structure preservation loss and the intra-class structure preservation loss are used to maintain the consistency between the inter-class structure and the intra-class structure in the old class At the same time, for the newly added category, the initial structure optimization loss is calculated by using the feature vector obtained by the new network decoding, and the initial structure optimization loss is used to narrow the distribution of the feature vectors of the same newly added category, and alienate the different newly added categories. The distribution of feature vectors, and use the category-by-category dynamic threshold to optimize and denoise the segmentation results of the old network, obtain corresponding pseudo-labels, and use the pseudo-labels to calculate the classification loss of the new network; The new network and the new feature transformation module are trained with inter-class structure preservation loss, intra-class structure preservation loss, initial structure optimization loss, and classification loss.

An image semantic segmentation network continuous learning system, the system includes:

The data collection and preliminary training unit is used to obtain the newly added semantic segmentation data set and the labels corresponding to the newly added categories, use the original image semantic segmentation network to extract the original feature map of the image data in the newly added semantic segmentation data set, and pass the feature transformation module. Transforming the original feature map, and preliminarily training the feature transformation module by using the difference between the reconstructed feature map and the original feature map from the transformation result;

The learning unit is used to initialize an identical image semantic segmentation network and feature transformation module using the original image semantic segmentation network and the initially trained feature transformation module, and the original image semantic segmentation network is called the old network, and the initially trained feature The transformation module is called the old feature transformation module, the image semantic segmentation network generated by initialization is called the new network, and the feature transformation module generated by initialization is called the new feature transformation module; the old network and the old feature transformation module are fixed, and the new network is trained. and the new feature transformation module; during training, input the image data of the newly added semantic segmentation data set into the old network and the new network at the same time, and perform feature map extraction, decoding and semantic segmentation in the old network and the new network respectively, and obtain Segmentation result; wherein, the feature map extracted by the old network is transformed by the old feature transformation module, the feature map extracted by the new network is transformed by the new feature transformation module, and the alignment loss of the two transformation results is calculated. ; Use the segmentation results of the old network and the new network and the eigenvectors obtained by decoding to independently construct the corresponding inter-class relationship matrix and the intra-class relationship set for the old category, and use the old network and the new network. The relationship matrix between classes The inter-class structure preservation loss is calculated, and the intra-class structure preservation loss is calculated by using the intra-class relationship set of the old network and the new network. The inter-class structure preservation loss and the intra-class structure preservation loss are used to preserve the inter-class structure and Consistency of intra-class structure; at the same time, for newly added categories, the initial structure optimization loss is calculated by using the feature vector obtained by decoding the new network, and the initial structure optimization loss is used to narrow the distribution of feature vectors of the same newly added category, Alienate the distribution of feature vectors of different new categories, and use the category-by-category dynamic threshold to optimize and denoise the segmentation results of the old network, obtain corresponding pseudo-labels, and use the pseudo-labels to calculate the classification loss of the new network; The alignment loss, inter-class structure preservation loss, intra-class structure preservation loss, initial structure optimization loss, and classification loss train the new network and new feature transformation module.

A processing device, comprising: one or more processors; a memory for storing one or more programs;

Wherein, when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the aforementioned method.

A readable storage medium storing a computer program which, when executed by a processor, implements the aforementioned method.

It can be seen from the technical solutions provided by the present invention that, on the one hand, the method of extracting old knowledge representations and aligning them through nonlinear transformation in the feature space effectively maintains the invariance of old knowledge and improves the ability to learn new knowledge. On the other hand, the topology structure of the new category is optimized in the embedding space, and the invariance of its topology structure is maintained for the old category, so as to reduce forgetting and prevent confusion between classes. This makes it unnecessary to provide labels of old categories in the continuous learning of semantic segmentation, reducing the cost of labeling. In general, as a universal continuous learning method for semantic segmentation, the present invention has no restrictions on application scenarios, and has strong generalization ability and practical value.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

1 is a schematic diagram of a model of an image semantic segmentation network continuous learning method provided by an embodiment of the present invention;

FIG. 2 is a schematic diagram of the principle of an initial structure optimization part provided by an embodiment of the present invention;

FIG. 3 is a schematic diagram of part of the principle of maintaining inter-class and intra-class structures according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of comparison of segmentation results of different image semantic segmentation networks according to an embodiment of the present invention;

5 is a schematic diagram of an image semantic segmentation network continuous learning system provided by an embodiment of the present invention;

FIG. 6 is a schematic diagram of a processing device according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present invention.

First a description of terms that may be used in this article:

The terms "comprising", "comprising", "containing", "having" or other descriptions with similar meanings should be construed as non-exclusive inclusions. For example: including certain technical characteristic elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.), should be construed to include not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

A method, system, device and storage medium for continuous learning of image semantic segmentation network provided by the present invention will be described in detail below. Contents that are not described in detail in the embodiments of the present invention belong to the prior art known to those skilled in the art. If the specific conditions are not indicated in the examples of the present invention, it is carried out according to the conventional conditions in the art or the conditions suggested by the manufacturer.

Example 1

The embodiment of the present invention provides a continuous learning method for image semantic segmentation network, which is a continuous learning method for semantic segmentation based on category structure keeping alignment with features. The current mainstream image semantic segmentation network consists of a feature extractor, a decoder and a classifier. The main process flow is: extract the input feature map of the image to be segmented through the feature extractor, obtain the corresponding feature vector through the decoder, and finally classify the image. Semantic segmentation is performed by the processor, and the classification result of each pixel (ie, the segmentation result) is obtained. The present invention designs corresponding modules for the image semantic segmentation network to prevent old knowledge from being forgotten. Specifically, the core content of the method includes four parts: feature transformation module, category structure information retention module, pseudo-label generation module and joint loss function training semantic segmentation network. The feature transformation module performs nonlinear transformation on the output feature map of the feature extractor, and then extracts the representation of the old knowledge for alignment, so as to provide a high degree of freedom for the learning of new knowledge while still effectively maintaining the integrity of the old knowledge. The class structure information retention module uses the decoder output to establish the intra-class topology and inter-class topology. By maintaining the above structural consistency during the learning process, the phenomenon of class topology destruction in the continuous learning process is effectively reduced, thereby reducing forgetting and inter-class topology. confused. Further, for the problem of inconsistent semantics before and after, the pseudo-label generation module uses a category-by-category dynamic threshold to optimize and denoise the segmentation results output by the old network, thereby generating high-quality pseudo-labels to make up for the missing old category labels. Finally, the loss function of the above modules is combined to train the semantic segmentation network to achieve the effect of continuous learning.

Each continuous learning process can be described as:

Obtaining the newly added semantic segmentation data set and the label corresponding to the newly added category, using the original image semantic segmentation network to extract the original feature map of the image data in the newly added semantic segmentation data set, and transforming the original feature map through a feature transformation module, and utilize the difference between the reconstructed feature map of the transformation result and the original feature map to preliminarily train the feature transformation module;

Use the original image semantic segmentation network and the initially trained feature transformation module to initialize an identical image semantic segmentation network and feature transformation module, the original image semantic segmentation network is called the old network, and the initially trained feature transformation module is called the old network. Feature transformation module, the image semantic segmentation network generated by initialization is called a new network, and the feature transformation module generated by initialization is called a new feature transformation module; the old network and the old feature transformation module are fixed, and the new network and the new feature transformation module are trained. ;

During training, the image data of the newly added semantic segmentation dataset is input into the old network and the new network at the same time, and feature map extraction, decoding and semantic segmentation are performed in the old network and the new network respectively to obtain segmentation results; The feature map extracted by the old network is transformed by the old feature transformation module, the feature map extracted by the new network is transformed by the new feature transformation module, and the alignment loss of the two transformation results is calculated; using the old network With the respective segmentation results of the new network and the eigenvectors obtained by decoding, the corresponding inter-class relationship matrix and intra-class relationship set are independently constructed for the old class, and the inter-class relationship matrix of the old network and the new network is used to calculate the inter-class structure. Preservation loss, the intra-class structure preservation loss is calculated using the set of intra-class relationships between the old network and the new network, and the inter-class structure preservation loss and intra-class structure preservation loss are used to preserve the difference between the inter-class structure and the intra-class structure in the old class. Consistency; at the same time, for the newly added category, the initial structure optimization loss is calculated by using the feature vector obtained by the new network decoding, and the initial structure optimization loss is used to narrow the distribution of the feature vectors of the same newly added category, and alienate different new ones. The distribution of the feature vectors of the categories, and use the category-by-category dynamic threshold to optimize the denoising of the segmentation results of the old network, obtain the corresponding pseudo-labels, and use the pseudo-labels to calculate the classification loss of the new network; combine the alignment loss , inter-class structure preservation loss, intra-class structure preservation loss, initial structure optimization loss and classification loss to train the new network and the new feature transformation module.

In order to facilitate understanding, the following is a further introduction to the above learning process.

As shown in FIG. 1, it is a schematic diagram of a model of an image semantic segmentation network continuous learning method provided by the present invention, which shows the relevant processes and loss functions involved in the continuous learning process, and is mainly described as follows:

1. Feature transformation module and related loss function.

In the embodiment of the present invention, before learning the image semantic segmentation network by using the newly added semantic segmentation data set, it is necessary to perform preliminary training on the feature transformation module part.

。然后，利用特征变换模块（FeatureProjector）进行非线性变换，生成关于旧知识的表征，通过训练特征变换模块引导其输出的表征中包含丰富、有效的信息。As mentioned earlier, the image semantic segmentation network includes a feature extractor, and the feature extractor is used to extract the original feature map of each image data in the newly added semantic segmentation dataset, denoted as

. Then, the feature transformation module (FeatureProjector) is used for nonlinear transformation to generate representations of old knowledge, and the output representations are guided by training the feature transformation module to contain rich and effective information.

进行变换包括：先通过卷积操作（例如，1*1卷积）进行通道降维，再通过若干空洞卷积操作（例如，两个3*3空洞卷积）进行局部空间信息混合，生成关于原始特征图

的表征。初步训练时，使用重构网络R ^* （例如，可采用两个3*3卷积构成）对变换结果

进行重构，通过尝试从特征变换模块的输出变换结果中重构原始特征图，并使用原始特征图与重构特征图的差异构造重构损失函数，利用重构损失初步训练所述特征变换模块P ^* ，从而引导特征变换模块P ^* 输出的表征中包含丰富的信息。In the embodiment of the present invention, the feature transformation module is preliminarily trained using the self-encoder structure, and the feature transformation module is denoted as P ^* ; the original feature map is transformed by the feature transformation module P ^*

The transformation includes: first performing channel dimensionality reduction through convolution operations (for example, 1*1 convolution), and then mixing local spatial information through several hole convolution operations (for example, two 3*3 hole convolutions) to generate information about original feature map

characterization. During initial training, use the reconstruction network R ^* (for example, it can be composed of two 3*3 convolutions) to transform the results

Reconstruct, by trying to reconstruct the original feature map from the output transformation result of the feature transformation module, and use the difference between the original feature map and the reconstructed feature map to construct a reconstruction loss function, and use the reconstruction loss to initially train the feature transformation module P ^* , which guides the feature transformation module P ^* output representations that contain rich information.

Those skilled in the art can understand that the convolution operation and the hole convolution operation are both conventional two types of convolution operations. In comparison, the convolution operation is a standard convolution operation, and the hole convolution operation can be performed with Fewer layers mix spatial information in a larger range.

与所述原始特征图

的欧氏距离，表示为：Specifically, the reconstruction loss function can be the reconstructed feature map

with the original feature map

The Euclidean distance of , expressed as:

。

.

。When the feature transformation module P ^* can effectively generate the representation of the old knowledge, the preliminary training is completed, and the feature transformation module at this time is recorded as

.

的参数初始化用于学习新知识的新网络与相应的特征变换模块

，学习新知识的过程中保持原始图像分割网络（即旧网络，Old Model）及其特征变换模块（即特征变换模块

）不变，只更新初始化产生的图像分割网络（即新网络，New Model）及其特征变换模块

（即初始化产生的特征变换模块）。学习阶段是增量学习的关键概念，初始阶段记为1，每新增一次类别集合，即为一个新的持续学习阶段。图1中，下标t-1、t代表不同学习阶段，相对而言，t-1学习阶段的网络为旧网络，t学习阶段的网络为新网络，E代表编码器（特征提取器），D代表解码器，G代表分类器。After that, the original image segmentation network and feature transformation module can be used

The parameters initialize the new network for learning new knowledge and the corresponding feature transformation module

, keep the original image segmentation network (ie old network, Old Model) and its feature transformation module (ie feature transformation module) in the process of learning new knowledge

) remains unchanged, only the image segmentation network (ie new network, New Model) and its feature transformation module generated by initialization are updated

(i.e. initialize the resulting feature transformation module). The learning stage is the key concept of incremental learning. The initial stage is denoted as 1, and each new category set is a new continuous learning stage. In Figure 1, the subscripts t -1 and t represent different learning stages. Relatively speaking, the network in the t -1 learning stage is the old network, the network in the t learning stage is the new network, and E represents the encoder (feature extractor), D stands for decoder and G stands for classifier.

In the embodiment of the present invention, a consistency constraint is imposed on the outputs of the two network feature transformation modules to ensure that the old knowledge remains unchanged during the continuous learning process, and at the same time, the feature map is given a high degree of freedom of change with a good degree of freedom. learn new knowledge.

，旧特征变换模块

的变换结果表示为

；将所述新网络提取的特征图记为

，将所述新特征变换模块记为

，变换结果表示为

，对齐损失（Alignment Loss）为两种变换结果的L1距离，表示为：In the embodiment of the present invention, the original image segmentation network in the previous stage is not updated. Therefore, the extracted feature map is the original feature map, so it is still recorded as

, the old feature transformation module

The result of the transformation is expressed as

; mark the feature map extracted by the new network as

, denote the new feature transformation module as

, the transformation result is expressed as

, Alignment Loss (Alignment Loss) is the L1 distance of the two transformation results, expressed as:

。

.

2. The category structure information retention module and related loss function.

In the embodiment of the present invention, the class structure information retention module respectively constructs the intra-class structure relationship and the inter-class structure relationship in the embedding space based on the decoder output of the image segmentation network. By maintaining the above two relationships in the process of continuous learning, the discriminative power of the network for the old categories is effectively maintained. The class structure information preservation module mainly includes three parts: the initial structure optimization part, the inter-class structure preservation part, and the intra-class structure preservation part. Among them, the initial structure optimization part mainly calculates the initial structure optimization loss for the new category, which belongs to the Contrastive Loss; the latter two parts are mainly for the old category, calculate the structure preservation loss (Structure Preserving Loss), including the inter-class structure The preservation loss and the intra-class structure preservation loss; the old category and the new category are relative concepts, that is, the categories that the image segmentation network can recognize before continuous learning. The principles and related loss functions of the above three parts are mainly as follows:

1. The initial structure optimization part.

As shown in Figure 2, it is a schematic diagram of the initial structure optimization part. When only using cross-entropy training, the distribution of different categories (such as A and B on the left side of the figure) in the embedding space is often scattered and prone to partial overlap, and this distribution is more likely to be induced in the subsequent learning process. Category confusion, and then forgetting. By guiding the feature vector (the triangle in the figure) as close as possible to its corresponding feature prototype (the X in the figure), and at the same time making the distance between the prototypes of different categories not less than a given threshold (gray circle in the right figure), the category distribution is optimized. Reduce the effect of confusion.

The initial structure optimization module is to guide its distribution in the embedding space when learning a new category, so that the distribution of feature vectors of the same category is as compact as possible, and the distribution of feature vectors of different categories is as discrete as possible. This part makes the classification boundary of the model clearer, reduces confusion, and is more robust to forgetting. Initially, since the initial structure optimization module is only for newly added categories, only the output of the new network is used to calculate the initial structure optimization loss.

In order to achieve the above purpose, the loss function of the initial structure optimization part (initial structure optimization loss) uses two loss functions to guide the learning of the intra-class structure and the inter-class structure respectively, which are expressed as:

表示引导类内结构的损失，

表示引导类间结构的损失，

为

的权重（具体大小可根据实际情况或者经验设定）。in,

represents the loss of bootstrap intra-class structure,

represents the loss of bootstrap inter-class structure,

for

(the specific size can be set according to the actual situation or experience).

用于拉近相同新增类别的特征向量的分布，表示为：Loss of bootstrap intra-class structure

The distribution of feature vectors used to close the same newly added category is expressed as:

表示当前学习阶段t新增类别的集合，

表示当前学习阶段t新增类别的数量，所述当前学习阶段t表示训练所述新网络与新特征变换模块的阶段；

表示新增类别c对应的类别原型，

表示属于新增类别c的特征向量。in,

represents the set of newly added categories in the current learning stage t ,

Represents the number of new categories in the current learning stage t , and the current learning stage t represents the stage of training the new network and the new feature transformation module;

Indicates the category prototype corresponding to the newly added category c ,

represents the feature vector belonging to the newly added category c .

用于疏远不同新增类别的分布，表示为：loss of bootstrap inter-class structure

The distribution used to alienate different newly added categories, expressed as:

与

分别表示新增类别m与新增类别n对应的类别原型，

表示类别原型

与

的余弦相似度，

为预定义的距离（具体大小可根据实际情况或者经验设定）。in,

and

represent the category prototypes corresponding to the newly added category m and the newly added category n , respectively,

Represents a class prototype

and

The cosine similarity of ,

It is a predefined distance (the specific size can be set according to the actual situation or experience).

表示为：In the embodiment of the present invention, the category prototype is the average of all feature vectors under the corresponding category, and for the newly added category c , the category prototype

Expressed as:

为指示函数，当y=c时，输出为1，其他情况输出为0。Among them, y is the label of the new category in the current stage, |y=c| indicates the number of pixels belonging to the new category c in the label,

For the indicator function, when y=c , the output is 1, and the output is 0 in other cases.

类似方式计算。Those skilled in the art can understand that Class Prototypes is a proper term in the field of computer vision, which means that the mean value of a series of features belonging to a certain category is calculated, and the mean value is used to represent the information of the entire category. The prototypes of each category also use the aforementioned

Calculated in a similar way.

2. Inter-class structure maintenance part.

A well-trained deep neural network can map input samples into an embedding space and distribute them according to categories in different regions of the embedding space. This is an important property of deep neural networks to correctly classify each category. Based on this property, the inter-class topology is constructed in the embedding space, and this structure is maintained in the process of continuous learning to keep the classes linearly separable.

，利用所述新网络的分割结果以及解码获得的特征向量对于旧类别构建的类间关系矩阵

；其中，类间关系矩阵中的单个元素表示两个旧类别对应的类别原型之间的余弦距离；对于旧类别i与旧类别j，旧网络中对应的类别原型分别表示为

与

，新网络中对应的类别原型分别表示为

与

，则类间关系矩阵

与

中相应元素

与

的计算方式表示为：In the embodiment of the present invention, an inter-class relationship matrix is constructed for the old class by using the segmentation result of the old network and the feature vector obtained by decoding

, using the segmentation result of the new network and the eigenvectors obtained by decoding to construct the inter-class relationship matrix for the old class

; where a single element in the inter-class relationship matrix represents the cosine distance between the class prototypes corresponding to the two old classes; for the old class i and the old class j , the corresponding class prototypes in the old network are represented as

and

, the corresponding category prototypes in the new network are respectively expressed as

and

, then the inter-class relationship matrix

and

corresponding element in

and

is calculated as:

、

分别表示类别原型

与

余弦相似度、类别原型

与

的余弦相似度。in,

,

class prototype

and

Cosine similarity, class prototype

and

The cosine similarity of .

In the process of continuous learning, the inter-class structure is used to maintain the consistency of the loss function, which is expressed as:

where ||.|| _F represents the F-norm of the matrix.

3. The structure of the class is maintained.

，利用所述新网络的分割结果以及解码获得的特征向量对于旧类别构建的类内关系集合表示为

，此处的D表示某一种距离度量函数，例如，欧式距离。类内关系集合反映了嵌入空间中细粒度的拓扑结构信息。在持续学习的过程中保持类内特征向量在嵌入空间中的拓扑结构保持不变，能有效维护单类别知识的完整性。建模类内结构时所选取的距离函数为欧式距离，以利用其敏感性反应反映类内结构的微小变化。The intra-class relationship is defined as the set of relative relationship between each feature vector and its class prototype, and the intra-class relationship set constructed for the old class using the segmentation result of the old network and the feature vector obtained by decoding is expressed as

, using the segmentation result of the new network and the feature vector obtained by decoding, the intra-class relationship set constructed for the old class is expressed as

, where D represents a certain distance metric function, such as Euclidean distance. The set of intra-class relations reflects fine-grained topology information in the embedding space. In the process of continuous learning, the topology structure of intra-class feature vectors in the embedding space remains unchanged, which can effectively maintain the integrity of single-class knowledge. The distance function selected when modeling intra-class structure is Euclidean distance, in order to use its sensitivity to reflect small changes in intra-class structure.

与

在位置信息与距离信息）的一致性，表示为：In the process of continuous learning, the intra-class structure preservation loss is used to preserve the intra-class structure in the old class (i.e. the set of intra-class relations)

and

In the consistency of position information and distance information), it is expressed as:

与

分别表示旧网络获得的属于旧类别i的特征向量与相应的类别原型；

与

表示新网络获得的属于旧类别i的特征向量与相应的类别原型；

表示旧类别集合（即已学习过的所有旧类别），

表示旧类别的数量。in,

and

respectively represent the feature vector belonging to the old category i obtained by the old network and the corresponding category prototype;

and

Represents the feature vector obtained by the new network belonging to the old category i and the corresponding category prototype;

represents the set of old categories (i.e. all old categories that have been learned),

Indicates the number of old categories.

的公式，区别主要在于，由于此部分是针对旧类别，因此，需要将分割结果带入

公式。The category prototypes involved in the inter-class structure retention part and the intra-class structure retention part are calculated using the segmentation results and feature vectors output by the corresponding network. The calculation formula can be found in the previous section.

The main difference is that since this part is for the old category, the segmentation result needs to be brought into

formula.

为例，其计算公式为：prototype by category

For example, the calculation formula is:

表示新网络输出的分割结果，

表示新网络输出的分割结果中预测类别为旧类别i的像素的数量。in,

represents the segmentation result of the new network output,

Represents the number of pixels whose predicted class is the old class i in the segmentation result output by the new network.

The same is true for the old network, combining its segmentation results into the above formula to calculate the corresponding category prototype.

As shown in Figure 3, it is a schematic diagram of the principle of maintaining the structure between classes and within classes. When a new category needs to be learned, the inter-class topology maintenance only ensures that the adjacent and related relationships between categories remain unchanged during the update process, but allows them to change in the embedding space such as rotation, translation, etc., so the old category is effectively maintained. At the same time, it is more conducive to the learning of new categories. When the intra-class structure is not constrained, the update of the network often results in a large change in the relative relationship of the same input to the feature prototype (lower left side of Figure 3). The intra-class structure preservation loss can reduce such changes, thereby maintaining the integrity of old knowledge at a finer granularity.

3. Pseudo-label generation module and related loss function.

In the embodiment of the present invention, the segmentation result of the old network is optimized and denoised by the category-by-category dynamic threshold, so as to generate high-quality pseudo-labels to make up for the missing old category labels. This process is called pseudo-label refinement (Pseudo Label Refinement). The principle is as follows:

使得相应旧类别i保留固定比例的伪标签，融合新增类别的真实标签（在前述阶段获取得到）后生成最终的监督标签，监督标签（伪标签）生成的方法表示为：In the process of continuous learning, the labels of the old classes are not given in the current learning stage, that is, the learned classes are labeled as background classes in the given labels. Therefore, when the network is directly trained with a given label as a supervision signal, the forgetting effect of the learned class will be exacerbated. To this end, the background classes for a given label are labeled with the semantic segmentation results of the old network, thus providing pseudo-labels for the learned classes. Further, the segmentation results output by the old network inevitably contain erroneous results. To solve this problem, the entropy of the output category probability is used as the confidence evaluation index, and only the results with higher confidence are used as pseudo labels. Since the network learns differently for different categories, the present invention calculates the distribution of the entropy output for each category, and selects the threshold accordingly

Make the corresponding old category i retain a fixed proportion of pseudo-labels, and fuse the real labels of the new categories (obtained in the previous stage) to generate the final supervised labels. The method of generating supervised labels (pseudo-labels) is expressed as:

表示当前学习阶段t输入图像

中像素k对应的新增类别的真实标签，

表示旧网络对像素k的分类置信度，

表示旧类别i对应的动态阈值，

表示旧类别集合，

表示旧网络

对输入图像

输出的分割结果，即每一像素的分类结果，

为最终生成的像素k的伪标签。in,

represents the current learning stage t input image

The ground truth label of the new category corresponding to the pixel k in the middle,

represents the classification confidence of the old network for pixel k ,

represents the dynamic threshold corresponding to the old category i ,

represents a collection of old categories,

Indicates the old network

for the input image

The output segmentation result, that is, the classification result of each pixel,

is the pseudo-label for the final generated pixel k .

After that, the classification loss of the new network is calculated using the finally generated pseudo-label, specifically the cross entropy loss (Cross Entropy Loss), which is expressed as:

表示所述新网络对于输入图像

输出的分割结果。in,

represents the new network for the input image

The output segmentation result.

Fourth, the joint loss function trains the semantic segmentation network.

In the embodiment of the present invention, the new network and the new feature transformation module are trained in combination with the alignment loss, inter-class structure preservation loss, intra-class structure preservation loss, initial structure optimization loss, and classification loss in the above-mentioned one to three, and finally achieve a semantic The purpose of continuous learning is achieved on segmentation tasks. The target loss function for training is the weighted sum of the above loss functions:

及

分别为相应损失的权重。in,

and

are the weights of the corresponding losses, respectively.

The embodiment of the present invention provides a continuous learning method for image semantic segmentation network, which mainly obtains the following beneficial effects:

1) The method of extracting old knowledge representations and aligning them through nonlinear transformation in the feature space effectively maintains the invariance of old knowledge and improves the ability to learn new knowledge.

2) Optimize the topology of the new category in the embedding space, and maintain the invariance of the topology of the old category, so as to reduce forgetting and prevent confusion between classes.

3) The combination of pseudo-label and pseudo-label noise reduction technology makes it unnecessary to provide labels of old categories in the continuous learning of semantic segmentation, reducing the cost of labeling.

In general, as a universal continuous learning method for semantic segmentation, the present invention has no restrictions on application scenarios, and has strong generalization ability and practical value.

Based on the above introduction, a complete implementation process is provided below, including the initial stage learning of the image semantic segmentation network, the continuous learning of the image semantic segmentation network, and the testing of the image semantic segmentation network.

First, the initial stage of image semantic segmentation network learning.

1. Prepare the initial semantic segmentation data set and the corresponding category labels to form the training data, change the spatial resolution of the image by random cropping, so that the width and height of the image are both 512, and normalize it.

2. Use the deep learning framework to establish an image semantic segmentation model based on category structure keeping and feature alignment, including fully convolutional semantic segmentation network, feature transformation module, category structure information preservation module and pseudo-label generation module, etc. Among them, the fully convolutional semantic segmentation network is DeeplabV3, and its feature extractor can choose ResNet, MobileNet, etc. ResNet-101 is used here as the feature extractor. Its decoder part is ASPP module. A feature transformation module is set in the output part of the feature extractor to perform nonlinear transformation and alignment operations on the features. The category structure information preservation module is set in the decoder part of the semantic segmentation network. A pseudo-label generation module is set in the output part of the semantic segmentation network.

3. In the initial learning process, randomly select a set of data from the training data to input into the network, give the semantic segmentation results through the model, and use the cross-entropy loss and the initial structure optimization loss to train the network.

The training procedures involved in this part are all conventional techniques, so they will not be described again; in addition, the specific image size, network structure and type involved in the above procedures are examples and do not constitute limitations.

2. Continuous learning of image semantic segmentation network.

1. After the initial stage of training is completed, prepare to add new semantic segmentation datasets and labels corresponding to the new categories. The spatial resolution of the image is changed by random cropping, so that the width and height of the image are both 512, and normalized.

Likewise, the specific image size involved here is only an example, not a limitation.

Those skilled in the art can understand that the newly added semantic segmentation dataset includes new categories and old categories. Of course, it is possible that a few images do not contain old categories, but the impact on the learning effect is small. In addition, new categories will be marked, and old categories do not need to be marked.

训练特征变换模块，使其在新增的数据上能完成特征变换操作。2. Preliminary training feature transformation module. Each iteration randomly selects a set of data from the training data to input the image semantic segmentation network, obtains the feature map output by the feature extractor, and uses the loss function

Train the feature transformation module so that it can complete the feature transformation operation on the newly added data.

。使用新、旧网络解码器输出对旧类别构建类间关系矩阵

、

及类内关系集合

、

。并计算类间结构保持损失

与类内结构保持损失

。同时对新类别计算初始结构优化损失

。最后，使用旧网络的输出经由伪标签生成模块生成完整的语义标签，在新网络的分割结果上计算交叉熵损失。3. Use the weights of the image semantic segmentation network and feature transformation module to initialize—the same network and feature transformation module (ie, new network and new feature transformation module) are used to learn new categories, and the old network and its feature transformation module are no longer updated. . Each iteration randomly selects a set of data from the training data and feeds both the new and old networks. The output feature maps of the two feature extractors respectively obtain the old knowledge representation through the new and old feature transformation modules, and calculate the alignment loss

. Build an inter-class relationship matrix for the old classes using the new and old network decoder outputs

,

and a collection of intra-class relationships

,

. and compute the inter-class structure preservation loss

maintain loss with intra-class structure

. Simultaneously compute the initial structural optimization loss for the new class

. Finally, complete semantic labels are generated via the pseudo-label generation module using the output of the old network, and the cross-entropy loss is calculated on the segmentation results of the new network.

4. Calculate the total loss function L according to the loss function in the above steps, minimize the loss function through the back-propagation algorithm and the gradient descent strategy, and update the parameter weights of the semantic segmentation network and the feature transformation module.

The back-propagation algorithm and gradient descent strategy involved in this stage can refer to conventional techniques, so they will not be described in detail.

When it is necessary to continue to learn new categories, repeat steps 1 to 4 of the continuous learning part of the image semantic segmentation network until all interested categories have been learned.

3. Image Semantic Segmentation Network Test.

The images in the test dataset are input to the image semantic segmentation network after continuous learning, and the segmentation results are obtained through its internal feature extractor and decoder in turn. The segmentation results can be evaluated by setting indicators to judge the semantic segmentation performance of the image semantic segmentation network after continuous learning.

As shown in Figure 4, a schematic diagram of the comparison of the segmentation results of different image semantic segmentation networks is shown; from left to right, the four columns of images represent: the input image, the segmentation result of the existing scheme, the segmentation result of the present invention, and the real segmentation result, It can be found from FIG. 4 that the segmentation result of the present invention is close to the real segmentation result, and far superior to the segmentation result of the existing scheme.

Embodiment 2

The present invention also provides an image semantic segmentation network continuous learning system, which is mainly implemented based on the method provided in the first embodiment. As shown in FIG. 5 , the system mainly includes:

The data collection and preliminary training unit is used to obtain the newly added semantic segmentation data set and the labels corresponding to the newly added categories, use the original image semantic segmentation network to extract the original feature map of the image data in the newly added semantic segmentation data set, and pass the feature transformation module. Transforming the original feature map, and preliminarily training the feature transformation module by using the difference between the reconstructed feature map and the original feature map from the transformation result;

The learning unit is used to initialize an identical image semantic segmentation network and feature transformation module using the original image semantic segmentation network and the initially trained feature transformation module, and the original image semantic segmentation network is called the old network, and the initially trained feature The transformation module is called the old feature transformation module, the image semantic segmentation network generated by initialization is called the new network, and the feature transformation module generated by initialization is called the new feature transformation module; the old network and the old feature transformation module are fixed, and the new network is trained. and the new feature transformation module; during training, input the image data of the newly added semantic segmentation data set into the old network and the new network at the same time, and perform feature map extraction, decoding and semantic segmentation in the old network and the new network respectively, and obtain Segmentation result; wherein, the feature map extracted by the old network is transformed by the old feature transformation module, the feature map extracted by the new network is transformed by the new feature transformation module, and the alignment loss of the two transformation results is calculated. ; Use the respective segmentation results of the old network and the new network and the eigenvectors obtained by decoding, independently construct the corresponding inter-class relationship matrix and the intra-class relationship set for the old category, and use the old network and the new network. The relationship matrix calculates the inter-class structure preservation loss, and uses the intra-class relationship set of the old network and the new network to calculate the intra-class structure preservation loss. The inter-class structure preservation loss and the intra-class structure preservation loss are used to preserve the inter-class in the old category The consistency between the structure and the intra-class structure; at the same time, for the newly added category, the eigenvector obtained by the new network decoding is used to calculate the initial structural optimization loss, and the initial structural optimization loss is used to shorten the eigenvectors of the same newly added category. distribution, alienate the distribution of feature vectors of different new categories, and use the category-by-category dynamic threshold to optimize and denoise the segmentation results of the old network to obtain corresponding pseudo-labels, and use the pseudo-labels to calculate the classification loss of the new network ; train the new network and the new feature transformation module in combination with the alignment loss, the inter-class structure preservation loss, the intra-class structure preservation loss, the initial structure optimization loss and the classification loss.

Those skilled in the art can clearly understand that, for the convenience and conciseness of the description, only the division of the above-mentioned functional modules is used for illustration. In practical applications, the above-mentioned functions can be allocated to different functional modules as required. The internal structure of the system is divided into different functional modules to complete all or part of the functions described above.

It should be noted that the main principles involved in the above-mentioned units have been described in detail in the previous Embodiment 1, so they are not repeated here.

Embodiment 3

The present invention also provides a processing device, as shown in FIG. 6 , which mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are described When executed by one or more processors, the one or more processors are caused to implement the methods provided by the foregoing embodiments.

Further, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, the memory, the input device, and the output device are connected through a bus.

In this embodiment of the present invention, the specific types of the memory, the input device, and the output device are not limited; for example:

The input device can be a touch screen, an image capture device, a physical button or a mouse, etc.;

The output device can be a display terminal;

The memory may be random access memory (Random Access Memory, RAM), or may be non-volatile memory (non-volatile memory), such as disk memory.

Embodiment 4

The present invention also provides a readable storage medium storing a computer program, and when the computer program is executed by a processor, the methods provided by the foregoing embodiments are implemented.

In this embodiment of the present invention, the readable storage medium, as a computer-readable storage medium, may be provided in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium may also be a U disk, a removable hard disk, a read-only memory (Read-Only Memory, ROM), a magnetic disk, or an optical disk, and other mediums that can store program codes.

The above description is only a preferred embodiment of the present invention, but the protection scope of the present invention is not limited to this. Substitutions should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (10)

1. an image semantic segmentation network continuous learning method, is characterized in that, comprises: Obtaining the newly added semantic segmentation data set and the label corresponding to the newly added category, using the original image semantic segmentation network to extract the original feature map of the image data in the newly added semantic segmentation data set, and transforming the original feature map through a feature transformation module, and utilize the difference between the reconstructed feature map of the transformation result and the original feature map to preliminarily train the feature transformation module; Use the original image semantic segmentation network and the initially trained feature transformation module to initialize an identical image semantic segmentation network and feature transformation module, the original image semantic segmentation network is called the old network, and the initially trained feature transformation module is called the old network. Feature transformation module, the image semantic segmentation network generated by initialization is called a new network, and the feature transformation module generated by initialization is called a new feature transformation module; the old network and the old feature transformation module are fixed, and the new network and the new feature transformation module are trained. ; During training, the image data of the newly added semantic segmentation dataset is input into the old network and the new network at the same time, and feature map extraction, decoding and semantic segmentation are performed in the old network and the new network respectively to obtain segmentation results; The feature map extracted by the old network is transformed by the old feature transformation module, the feature map extracted by the new network is transformed by the new feature transformation module, and the alignment loss of the two transformation results is calculated; using the old network With the segmentation result of the new network and the eigenvector obtained by decoding, the corresponding inter-class relationship matrix and intra-class relationship set are independently constructed for the old class, and the inter-class structure retention loss is calculated by using the inter-class relationship matrix of the old network and the new network. , the intra-class structure preservation loss is calculated by using the intra-class relationship set of the old network and the new network, and the inter-class structure preservation loss and the intra-class structure preservation loss are used to maintain the consistency between the inter-class structure and the intra-class structure in the old class At the same time, for the newly added category, the initial structure optimization loss is calculated by using the feature vector obtained by the new network decoding, and the initial structure optimization loss is used to narrow the distribution of the feature vectors of the same newly added category, and alienate the different newly added categories. The distribution of feature vectors, and the segmentation results of the old network are optimized by the category-by-category dynamic threshold to obtain the corresponding pseudo-labels, and the pseudo-labels are used to calculate the classification loss of the new network; combined with the alignment loss, the inter-class structure The preservation loss, the intra-class structure preservation loss, the initial structure optimization loss, and the classification loss train the new network and the new feature transformation module. 2. The method for continuous learning of an image semantic segmentation network according to claim 1, wherein the original image semantic segmentation network is used to extract the original feature map of the image data in the newly added semantic segmentation data set, and the feature transforms The module transforms the original feature map, and preliminarily trains the feature transformation module using the difference between the reconstructed feature map and the original feature map from the transformation result, including: The feature transformation module is initially trained using the autoencoder structure, and the original feature map is denoted as

, denote the feature transformation module as P ^* ; the original feature map is transformed by the feature transformation module P ^*

The transformation includes: first reducing the channel dimension through convolution operations, and then mixing local spatial information through several hole convolution operations to generate the original feature map.

characterization; Use the reconstruction network R ^* to transform the result

Perform reconstruction, reconstructed feature map

with the original feature map

The difference is the Euclidean distance between the two, expressed as:

The feature transformation module P ^* is initially trained with a reconstruction loss. 3 . The continuous learning method for image semantic segmentation network according to claim 1 , wherein the feature map extracted by the old network is transformed by the old feature transformation module, and the feature map extracted by the new network is transformed by 3 . The new feature transformation module performs transformation, and calculates the alignment loss of the two transformation results including: The feature map extracted by the old network is the original feature map, denoted as

, denote the old feature transformation module as

, the transformation result is expressed as

; mark the feature map extracted by the new network as

, denote the new feature transformation module as

, the transformation result is expressed as

, the alignment loss is the L1 distance of the two transformation results, expressed as:

. 4. A kind of image semantic segmentation network continuous learning method according to claim 1, it is characterized in that, described utilizing the inter-class relationship matrix of described old network and new network to calculate the loss of inter-class structure retention is expressed as:

in,

Represents the inter-class relationship matrix constructed for the old class using the segmentation result of the old network and the feature vector obtained by decoding;

Represents the inter-class relationship matrix constructed by using the segmentation result of the new network and the feature vector obtained by decoding for the old class; ||.|| _F represents the F norm of the matrix; A single element in the inter-class relationship matrix represents the cosine distance between the class prototypes corresponding to the two old classes; for the old class i and the old class j, the corresponding class prototypes in the old network are represented as

and

, the corresponding category prototypes in the new network are respectively expressed as

and

, then the inter-class relationship matrix

and

corresponding element in

and

is calculated as:

Among them, the category prototype is the average of all feature vectors under the corresponding category,

,

class prototype

and

Cosine similarity, class prototype

and

The cosine similarity of . 5. A kind of image semantic segmentation network continuous learning method according to claim 1, is characterized in that, described using the intra-class relationship set of the old network and the new network to calculate the intra-class structure retention loss is expressed as:

Among them, the intra-class relationship set constructed for the old class using the segmentation result of the old network and the feature vector obtained by decoding is expressed as

,

and

respectively represent the feature vector and the corresponding class prototype that belong to the old category i obtained by decoding the old network; use the segmentation result of the new network and the feature vector obtained by decoding to represent the intra-class relationship set constructed for the old category

,

and

respectively represent the feature vector of the old category i obtained by the new network decoding and the corresponding category prototype; the category prototype is the average of all feature vectors under the corresponding category, D represents the distance metric function,

represents a collection of old categories,

Indicates the number of old categories. 6. A kind of image semantic segmentation network continuous learning method according to claim 1, is characterized in that, described for the newly added category, utilizes the feature vector that described new network decoding obtains to calculate the initial structure optimization loss and expresses as:

in,

represents the loss of bootstrap intra-class structure,

represents the loss of bootstrap inter-class structure,

for

the weight of; Loss of bootstrap intra-class structure

The distribution of feature vectors used to close the same newly added category is expressed as:

in,

represents the set of newly added categories in the current learning stage t,

Represents the number of new categories in the current learning stage t, and the current learning stage t represents the stage of training the new network and the new feature transformation module;

Indicates the category prototype corresponding to the newly added category c,

Represents the feature vector belonging to the newly added category c; loss of bootstrap inter-class structure

The distribution of feature vectors used to alienate different newly added categories, expressed as:

in,

and

represent the category prototypes corresponding to the newly added category m and the newly added category n, respectively,

Represents a class prototype

and

The cosine similarity of ,

is a predefined distance; The category prototype is the average of all feature vectors under the corresponding category. For the new category c, the category prototype

Expressed as:

Among them, y is the label of the new category in the current stage, |y=c| indicates the number of pixels belonging to the new category c in the label,

For the indicator function, when y=c, the output is 1, and the output is 0 in other cases. 7. A method for continuous learning of image semantic segmentation network according to claim 1, characterized in that, the segmentation result of the old network is optimized by using a category-by-category dynamic threshold to obtain a corresponding pseudo-label, and the pseudo-label is used Computing the classification loss for the new network consists of: Use the category-by-category dynamic threshold to optimize the segmentation results of the old network, and fuse the acquired labels of the new categories to obtain the corresponding pseudo-labels, which are expressed as:

in,

Represents the input image obtained at the current learning stage t

The label of the newly added category corresponding to the pixel k, and the current learning stage t represents the stage of training the new network and the new feature transformation module;

represents the classification confidence of the old network for pixel k,

represents the dynamic threshold corresponding to the old category i,

represents a collection of old categories,

Indicates the old network

for the input image

The output segmentation result, that is, the classification result of each pixel,

is the pseudo-label of the generated pixel k; The classification loss of the new network is calculated using pseudo-labels, expressed as:

in,

represents the new network for the input image

The output segmentation result. 8. An image semantic segmentation network continuous learning system, characterized in that, based on the method implementation of any one of claims 1 to 7, the system comprises: The data collection and preliminary training unit is used to obtain the newly added semantic segmentation data set and the labels corresponding to the newly added categories, use the original image semantic segmentation network to extract the original feature map of the image data in the newly added semantic segmentation data set, and pass the feature transformation module. Transforming the original feature map, and preliminarily training the feature transformation module by using the difference between the reconstructed feature map and the original feature map from the transformation result; The learning unit is used to initialize an identical image semantic segmentation network and feature transformation module using the original image semantic segmentation network and the initially trained feature transformation module, and the original image semantic segmentation network is called the old network, and the initially trained feature The transformation module is called the old feature transformation module, the image semantic segmentation network generated by initialization is called the new network, and the feature transformation module generated by initialization is called the new feature transformation module; the old network and the old feature transformation module are fixed, and the new network is trained. and the new feature transformation module; during training, input the image data of the newly added semantic segmentation data set into the old network and the new network at the same time, and perform feature map extraction, decoding and semantic segmentation in the old network and the new network respectively, and obtain Segmentation result; wherein, the feature map extracted by the old network is transformed by the old feature transformation module, the feature map extracted by the new network is transformed by the new feature transformation module, and the alignment loss of the two transformation results is calculated. ; Use the segmentation results of the old network and the new network and the eigenvectors obtained by decoding to independently construct the corresponding inter-class relationship matrix and the intra-class relationship set for the old category, and use the old network and the new network. The relationship matrix between classes The inter-class structure preservation loss is calculated, and the intra-class structure preservation loss is calculated by using the intra-class relationship set of the old network and the new network. The inter-class structure preservation loss and the intra-class structure preservation loss are used to preserve the inter-class structure and Consistency of intra-class structure; at the same time, for newly added categories, the initial structure optimization loss is calculated by using the feature vector obtained by decoding the new network, and the initial structure optimization loss is used to narrow the distribution of feature vectors of the same newly added category, Alienate the distribution of feature vectors of different new categories, and optimize the segmentation results of the old network by using a category-by-category dynamic threshold to obtain corresponding pseudo-labels, and use the pseudo-labels to calculate the classification loss of the new network; Alignment loss, inter-class structure preservation loss, intra-class structure preservation loss, initial structure optimization loss, and classification loss train the new network and new feature transformation module. 9. A processing device, comprising: one or more processors; a memory for storing one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method according to any one of claims 1-7. 10. A readable storage medium storing a computer program, wherein the method according to any one of claims 1 to 7 is implemented when the computer program is executed by a processor.

Images