WO2023108317A1 - 基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法 - Google Patents

基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法 Download PDF

Info

Publication number
WO2023108317A1
WO2023108317A1 PCT/CN2021/137348 CN2021137348W WO2023108317A1 WO 2023108317 A1 WO2023108317 A1 WO 2023108317A1 CN 2021137348 W CN2021137348 W CN 2021137348W WO 2023108317 A1 WO2023108317 A1 WO 2023108317A1
Authority
WO
WIPO (PCT)
Prior art keywords
label
classification network
information entropy
deep multi
unknown
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2021/137348
Other languages
English (en)
French (fr)
Inventor
周冬豪
陈广勇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Institute of Advanced Technology of CAS
Original Assignee
Shenzhen Institute of Advanced Technology of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Institute of Advanced Technology of CAS filed Critical Shenzhen Institute of Advanced Technology of CAS
Priority to PCT/CN2021/137348 priority Critical patent/WO2023108317A1/zh
Publication of WO2023108317A1 publication Critical patent/WO2023108317A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Definitions

  • the invention relates to the technical field of machine learning, in particular to a robust training method for a deep multi-label classification network based on a regular mechanism for maximizing information entropy.
  • multi-label image classification Unlike single-label image classification, the purpose of multi-label image classification is to predict all corresponding labels for an image. Because a natural scene naturally contains multiple objects and concepts, multi-label image classification is more in line with real-world settings. However, it is very labor-intensive to perform detailed labeling on large-scale multi-label image datasets, and the cost of manpower and material resources consumed by labeling is also much greater than that of labeling single-label image datasets, because an image corresponds to the potential number of labels There can be many, and smaller and rarer objects are often overlooked by annotators. Therefore, the existing open source large-scale multi-label image datasets are widely considered to be missing some labels, which promotes the research on multi-label learning problems in the case of missing labels.
  • SPML Single Positive Multi-label Learning
  • the setting of only a single positive label for each sample is in line with most practical application scenarios, such as collecting image datasets from the Internet through queries, etc., which shows that the SPML problem has a wide range of practical application values.
  • deeply studying the SPML problem and proposing a proper solution can relax the labeling requirements of large-scale multi-label image datasets, significantly reduce its labeling costs, and have significant socio-economic benefits. Therefore, SPML is a variant of multi-label learning worthy of further exploration, and has become one of the challenging emerging research directions in the field of deep learning in recent years.
  • the learning paradigm of the deep multi-label classification network widely used in related technologies is to transform the prediction problem of multiple labels into multiple separate binary classification problems. Due to the particularity of the multi-label setting, the number of negative labels is far greater than the number of positive labels in large-scale multi-label image datasets. Therefore, an intuitive method is to treat all unknown labels as negative labels, and then use the standard binary cross-entropy loss function for training, which is called Assuming Negatives (AN), which is the baseline method for solving SPML problems . Assume that a sample pair (x (n) , z (n) ) with only a single positive label is given, where x (n) is the nth input image and z (n) is the label vector corresponding to the nth image. Then the loss function of the AN method is:
  • f (n) is the output of the model with respect to x (n)
  • C is the number of categories, and Indicates that there are positive and negative labels for the cth class, respectively.
  • the loss function of the AN method follows the standard binary cross-entropy loss function, and the deep multi-label classification network is trained based on the label assumption of "treating all unknown labels as negative labels".
  • the regularized online label estimation (Regularized Online Label Estimation, ROLE) method uses the average number of true positive labels for each sample in the data set to calculate the depth
  • the output of the multi-label classification network is constrained so that the sum of the output f (n) of the model on x (n) is not too large, which is equivalent to introducing a regularization term to the loss function, called the expected positive label regularization (Expected Positive Regularization, EPR).
  • the ROLE method adds an additional label estimation module to the original deep multi-label classification network, and uses mutual supervision to jointly train the two, using the label estimation module
  • the estimated label is used as the label when the deep multi-label classification network calculates the loss.
  • the prediction of the deep multi-label classification network is used as the supervision information when the label estimation module estimates the label.
  • the ROLE method also introduces EPR to the label estimation module to constrain the estimated labels.
  • the ROLE method can discard the previously trained label estimation module, and only use the trained deep multi-label classification network for label prediction.
  • the EPR regularization method introduced by the ROLE method needs to count the average number of true positive labels for each sample in the data set in advance, which is unrealistic in practical application scenarios, because the number of true positive labels in the data set is inherently unknown.
  • the label estimation module adopted by the ROLE method is essentially a huge estimated label matrix, which will introduce a large number of additional learnable parameters to the original network model, greatly increasing the time and space requirements for network training, and the introduced The number of learning parameters is positively correlated with the number of categories and samples in the dataset, which shows that the ROLE method is difficult to handle very large-scale multi-label image datasets.
  • the present invention proposes a robust training method for deep multi-label classification networks based on information entropy maximization regularization mechanism, and uses a binary cross-entropy loss function regularization mechanism based on information entropy maximization to solve the SPML problem,
  • the purpose is to maximize the information entropy of the deep multi-label classification network for the prediction probability of unknown labels, so that the model can avoid the influence of false negative label noise, use the supervision information provided by the real positive labels to learn, and distinguish unknown labels.
  • Strong prediction the method is easy to implement, does not introduce additional learnable parameters, and can be combined with any deep multi-label classification network, suitable for processing very large-scale multi-label image data sets.
  • the present invention provides a robust training method for deep multi-label classification networks based on information entropy maximization regular mechanism, including: the deep multi-label classification network regards the prediction of each unknown label of each sample as a discrete Random variables, including two types of predicted positive labels and predicted negative labels, the probability of predicted positive labels is the unknown label prediction probability output by the deep multi-label classification network, and the information controlling the unknown label prediction probability output by the deep multi-label classification network entropy maximization.
  • the method assumes that a sample pair (x (n) , z (n) ) with only a single positive label is given, where x (n) is the nth input image, z (n) is the nth label vectors corresponding to images, the loss function is:
  • f (n) is the output of the model with respect to x (n)
  • x (n) is the predicted probability of the model on the c-th class
  • C is the number of categories
  • Indicates that there is a negative label for the cth class is a regularization item.
  • ⁇ is a hyperparameter that controls the regularization strength.
  • the value of the hyperparameter ⁇ is adjusted between 0 and 1.
  • the probability of predicting a positive label in the method is the predicted probability of an unknown label output by the network
  • the probability of predicting a negative label is then
  • the method will control the unknown label prediction probability to tend to 0.5.
  • P(X) is the probability mass function of X
  • E[ ⁇ ] is the expectation function
  • I(X) is the amount of information of X.
  • the present invention aims to solve the problem of robust learning of deep multi-label classification networks when each sample has only a single positive label and the rest of the labels are missing.
  • the loss function regularization mechanism that is, the EMR method, looks at the unknown labels in the data set from a unique perspective, that is, does not make any assumptions about these unknown labels and admits that they are unknown, and maximizes the depth in the loss function during model training.
  • the information entropy of the multi-label classification network for the prediction probability of unknown labels is used to solve the SPML problem, so that the model can avoid the influence of false negative label noise, use the supervision information provided by the real positive label to learn, and distinguish unknown labels. Strong predictions.
  • the EMR method is easy to implement, does not introduce additional learnable parameters, and can be combined with any deep multi-label classification network. It has excellent performance and is suitable for processing very large-scale multi-label image datasets.
  • the generalization performance of the deep multi-label classification network can be greatly improved, and the performance is good in the case of extreme label loss.
  • the present invention provides a robust training method for a deep multi-label classification network based on the information entropy maximization regular mechanism, including: the prediction of each unknown label of each sample by the deep multi-label classification network is regarded as a discrete random variable, including the prediction is a positive label and predicted as a negative label, the probability of predicting a positive label is the unknown label prediction probability output by the deep multi-label classification network, and the information entropy of the unknown label prediction probability output by the deep multi-label classification network is controlled to maximize.
  • information entropy refers to the average amount of information contained in each received message, also known as source entropy, average self-information, etc., which can be used to measure the uncertainty of a message.
  • the "message” here represents an event, sample or feature from a distribution or data stream. In the method proposed by the present invention, it refers to the unknown label prediction probability output by the deep multi-label classification network.
  • the unit of entropy is usually bit, but it is also measured in units such as Sh, nat, Hart, etc., depending on the definition of entropy using the base of the logarithm.
  • Sh Sh
  • nat Hart, etc.
  • P(X) is the probability mass function of X
  • E[ ⁇ ] is the expectation function
  • I(X) is the amount of information of X.
  • the present invention can further complete the design of the regularization mechanism of binary cross-entropy loss function based on information entropy maximization.
  • the present invention adopts a simple but effective idea to treat unknown labels in the data set , that is, make no assumptions about these unknown labels and admit that they are unknown. Based on this idea, the prediction probability of the deep multi-label classification network for the unknown label should be as close to 0.5 as possible, that is, tend to 0.5, that is, the information entropy of the prediction probability should be as large as possible.
  • the binary cross-entropy loss function regularization mechanism based on information entropy maximization in the present invention is used to solve the SPML problem, and the purpose is to maximize the information entropy of the depth multi-label classification network for the unknown label prediction probability, called (Entropy Maximization Regularizer, EMR) method.
  • EMR Entropy Maximization Regularizer
  • f (n) is the output of the model with respect to x (n)
  • x (n) is the predicted probability of the model on the c-th class.
  • C is the number of categories, Indicates that there is a positive label in the cth class, Indicates that there is a negative label for the cth class,
  • the regularization term introduced for the EMR method is defined as follows:
  • ⁇ is a hyperparameter that controls the regularization strength, and its value is generally adjusted between 0 and 1.
  • the prediction of each unknown label of each sample by the deep multi-label classification network can be regarded as a discrete random variable, and there are two possibilities of predicting a positive label and a negative label respectively.
  • the probability of predicting a positive label is the predicted probability of an unknown label output by the network
  • the probability of predicting a negative label is In this way, the EMR method can give unknown labels the supervision of maximum information entropy instead of treating them all as negative labels, which is in line with the idea adopted in the present invention.
  • the EMR method enables the deep multi-label classification network to use the supervised information provided by the real positive labels to learn, and make highly distinguishable predictions for unknown labels, instead of making overly confident or even wrong judgments on them. This greatly improves the generalization ability of the network.
  • the binary cross-entropy loss function regularization mechanism based on information entropy maximization proposed by the present invention looks at the unknown labels in the data set from a unique perspective, that is, does not make any assumptions about these unknown labels and recognizes them is unknown, maximize the information entropy of the deep multi-label classification network for the prediction probability of the unknown label in the loss function of the model training, and use it to solve the SPML problem, so that the model can avoid the influence of false negative label noise, and use the real positive label
  • the provided supervision information is used to learn and make highly distinguishable predictions for unknown labels.
  • the EMR method is easy to implement and does not introduce additional learnable parameters. At the same time, it can be combined with any deep multi-label classification network and has excellent performance. , which is suitable for processing very large-scale multi-label image datasets.
  • the EPR regularization method introduced by the ROLE method needs to count the average number of real positive labels of each sample in the data set in advance, but this is difficult to achieve in practical application scenarios, because the number of real positive labels in the data set is Unknown.
  • the label estimation module adopted by the ROLE method introduces a large number of additional learnable parameters to the original network model, which greatly increases the time and space requirements for network training, and the number of learnable parameters introduced is not consistent with the number of categories and samples in the data set. Positively correlated, which indicates that the ROLE method is not suitable for very large-scale multi-label image datasets.
  • the method proposed in the present invention does not introduce additional learnable parameters, and is applicable to a wider range of scenarios. It can be used in combination with any deep multi-label classification network and multi-label data sets of any scale, and its performance is far superior to the ROLE method.
  • the present invention aims to solve the problem of robust learning of deep multi-label classification networks when each sample has only a single positive label and the rest of the labels are missing.
  • the present invention proposes a binary cross-entropy loss function regularization mechanism based on information entropy maximization. By using the method proposed in the present invention for training, the generalization performance of the deep multi-label classification network can be greatly improved, and the performance is good in the case of extreme label loss.

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本发明公开了基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法，旨在解决每个样本仅有单个正标签且其余标签均缺失的情况下，深度多标签分类网络的鲁棒学习问题，利用基于信息熵最大化的二元交叉熵损失函数正则化机制解决SPML问题，目的是最大化深度多标签分类网络对于未知标签预测概率的信息熵，使得模型可以免受假阴性标签噪声的影响，利用真实正标签提供的监督信息进行学习，并对未知标签做出可区分性较强的预测，方法易于实现，不会引入额外的可学习参数，同时能够与任意的深度多标签分类网络相结合，适合处理超大规模多标签图像数据集。

Description

基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法 技术领域
本发明涉及机器学习技术领域，具体涉及基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法。
背景技术
不同于单标签图像分类，多标签图像分类的目的在于为一个图像预测所有对应的标签。因为一个自然场景中天然地包含多个物体和概念，所以多标签图像分类更加符合真实世界中的设定。然而，对大规模多标签图像数据集进行详尽的标签标注十分费事费力，标注所消耗的人力物力成本也远远大于对单标签图像数据集进行标注的情况，因为一个图像对应的潜在的标签数目可能很多，而且较小物体和较稀有物体往往为标注人员所忽略。因此，现存的开源的大规模多标签图像数据集被广泛认为是缺失部分标签的，这促进了学界对于缺失标签情况下的多标签学习问题的研究。考虑到一种极端缺失标签的情况，即每个样本仅有单个正标签被标注，且其余标签均未进行标注，这被称为单正标签情况下的多标签学习(Single Positive Multi-label Learning，SPML)问题。每个样本仅有单个正标签的设定符合多数实际应用场景，例如从互联网通过查询的方式进行图像数据集的收集等，这表明SPML问题具有广泛的实际应用价值。此外，深入研究SPML问题并提出妥善的解决方法，可以放宽大规模多标签图像数据集的标注要求，显著降低其标注成本，具有重大的社会经济效益。因此，SPML是一种值得进一步探索的多标签学习变种，并成为了近年来深度学习领域颇具挑战性的新兴研究方向之一。
相关技术广泛采用的深度多标签分类网络的学习范式是将多个标签的预测问题转化为多个单独的二分类问题。由于多标签设定的特殊性，在大规模多标签图像数据集中，负标签的数量远远大于正标签的数量。因此，一种直观的方法是将所有未知的标签均视作负标签，再利用标准的二元交叉熵损失函数进行训练，这被称为Assuming Negatives(AN)，它是解决SPML问题的基线方法。假设给定一个仅有单个正标签的样本对(x (n),z (n))，其中x (n)为第n个输入图像，z (n)为第n个图像对应的标签向量。则AN方法的损失函数为：
Figure PCTCN2021137348-appb-000001
其中，f (n)是模型关于x (n)的输出，
Figure PCTCN2021137348-appb-000002
是模型在第c个类上的标签预测概率。C是类别数目，
Figure PCTCN2021137348-appb-000003
和
Figure PCTCN2021137348-appb-000004
分别表示第c个类存在正标签和负标签。AN方法的损失函数沿袭了标 准的二元交叉熵损失函数，基于“将所有未知的标签均视作负标签”的标签假设条件进行深度多标签分类网络的训练。
为了解决深度多标签分类网络的训练过程中缺少负标签会陷入退化解的问题，正则化在线标签估计(Regularized Online Label Estimation，ROLE)方法利用数据集中每个样本的平均真实正标签数量来对深度多标签分类网络的输出进行约束，使其模型关于x (n)的输出f (n)的总和不至于过大，这相当于给损失函数引入了一个正则化项，称为期望正标签正则化(Expected Positive Regularization，EPR)。为了对未知标签做出预测以提供更多的监督信息，ROLE方法在原始的深度多标签分类网络上额外添加了一个标签估计模块，采用互相监督的方式将两者进行联合训练，利用标签估计模块估计出来的标签作为深度多标签分类网络计算损失时的标签，反之，利用深度多标签分类网络的预测作为标签估计模块估计标签时的监督信息。同样地，ROLE方法也给标签估计模块引入了EPR用于约束其估计出来的标签。在进行预测时，ROLE方法可以丢弃之前训练的标签估计模块，仅用训练完毕的深度多标签分类网络进行标签预测。
然而相关技术具有以下缺点：1)虽然负标签在大规模多标签图像数据集的标签注释中占绝大多数，但是AN方法做出的假设(即把所有未知的标签均视作负标签)会引入大量的假阴性标签噪声，这给深度多标签分类网络的训练提供了大量的错误的监督信息，使其难以做出较为正确的正负标签预测。此外，该假设还会加强多标签学习中的正负标签不均衡的现象，使得网络更加倾向于预测出负标签，这进一步削弱了深度多标签分类网络的分类性能。2)ROLE方法引入的EPR正则化方法需要提前统计数据集中每个样本的平均真实正标签数量，这在实际应用场景中是不现实的，因为数据集中真实正标签的数目本来就是未知的。此外，ROLE方法采用的标签估计模块本质上是一个巨大的估计标签矩阵，这会给原本的网络模型引入额外的大量的可学习参数，大大增加网络训练时的时间和空间需求，且引入的可学习参数数目与数据集中的类别数目和样本数目正相关，这表明ROLE方法难以处理超大规模多标签图像数据集。
发明内容
为了解决现有技术中的问题，本发明提出基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法，利用基于信息熵最大化的二元交叉熵损失函数正则化机制解决SPML问题，目的是最大化深度多标签分类网络对于未知标签预测概率的信息熵，使得模型可以免受假阴性标签噪声的影响，利用真实正标签提供的监督信息进行学习，并对未知标签做出可区分性较强的预测，方法易于实现，不会引入额外的可学习参数，同时能够与任意的深度多标 签分类网络相结合，适宜处理超大规模多标签图像数据集。
为了实现以上目的，本发明提供了基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法，包括：深度多标签分类网络对于每一个样本的每一个未知标签的预测均视为一个离散随机变量，包括预测为正标签和预测为负标签两种，预测为正标签的概率即为深度多标签分类网络输出的未知标签预测概率，控制深度多标签分类网络输出的未知标签预测概率的信息熵最大化。
进一步地，所述方法中假设给定一个仅有单个正标签的样本对(x (n),z (n))，其中x (n)为第n个输入图像，z (n)为第n个图像对应的标签向量，则损失函数为：
Figure PCTCN2021137348-appb-000005
其中，f (n)是模型关于x (n)的输出，
Figure PCTCN2021137348-appb-000006
是模型在第c个类上的预测概率，C是类别数目，
Figure PCTCN2021137348-appb-000007
表示第c个类存在正标签，
Figure PCTCN2021137348-appb-000008
表示第c个类存在负标签，
Figure PCTCN2021137348-appb-000009
为正则化项。
进一步地，所述正则化项
Figure PCTCN2021137348-appb-000010
定义为：
Figure PCTCN2021137348-appb-000011
其中，α为控制正则化强度大小的超参数。
进一步地，所述超参数α的值在0到1之间进行调节。
进一步地，所述方法中预测为正标签的概率即为网络输出的未知标签预测概率
Figure PCTCN2021137348-appb-000012
预测为负标签的概率即为
Figure PCTCN2021137348-appb-000013
进一步地，所述方法会控制未知标签预测概率趋于0.5。
进一步地，所述方法中假设有一个离散随机变量X∈{x 1,x 2,...,x n}，则其信息熵的定义为：
H(X)＝Ε[I(X)]＝Ε[-ln(P(X))]
其中，P(X)为X的概率质量函数，E[·]为期望函数，I(X)即为X的信息量。
进一步地，所述方法中若已知或已对P(X)做出估计后，则信息熵的公式表示为：
Figure PCTCN2021137348-appb-000014
其中，b是对数所使用的底。
进一步地，所述方法中当b＝2，信息熵的单位是bit；当b＝e，信息熵的单位是nat；当b＝10，信息熵的单位是Hart。
进一步地，所述方法中深度多标签分类网络对于每一个样本的每一个未知标签的预测均视为一个离散随机变量X＝{x 1,x 2}，事件x 1即为预测为正标签，事件x 2即为预测为负标签， 则P(x 1)即为网络输出的未知标签预测概率，P(x 2)＝1-P(x 1)即为预测为负标签的概率。
与现有技术相比，本发明旨在解决每个样本仅有单个正标签且其余标签均缺失的情况下，深度多标签分类网络的鲁棒学习问题，基于信息熵最大化的二元交叉熵损失函数正则化机制，即EMR方法，从一个独特的角度来看待数据集中的未知标签，即不对这些未知标签做出任何假设并且承认它们是未知的，在模型训练时的损失函数中最大化深度多标签分类网络对于未知标签预测概率的信息熵，用于解决SPML问题，使得模型可以免受假阴性标签噪声的影响，利用真实正标签提供的监督信息进行学习，并对未知标签做出可区分性较强的预测。EMR方法易于实现，不会引入额外的可学习参数，同时能够与任意的深度多标签分类网络相结合，性能优异，适宜处理超大规模多标签图像数据集。利用本发明所提出的方法进行训练，深度多标签分类网络的泛化性能可以大幅提高，在极度标签缺失的情况下表现良好。
具体实施方式
为使本发明实施例的目的、技术方案和优点更加清楚，下面将对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本发明一部分实施例，而不是全部的实施例。因此，本发明的实施例的详细描述并非旨在限制要求保护的本发明的范围，而是仅仅表示本发明的选定实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
本发明提供了基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法，包括：深度多标签分类网络对于每一个样本的每一个未知标签的预测均视为一个离散随机变量，包括预测为正标签和预测为负标签，预测为正标签的概率即为深度多标签分类网络输出的未知标签预测概率，控制深度多标签分类网络输出的未知标签预测概率的信息熵最大化。
在信息论中，信息熵指的是接收的每条消息中包含的信息的平均量，又被称为信源熵、平均自信息量等，它可以用于度量消息的不确定性。这里的“消息”代表来自分布或数据流中的事件、样本或特征，在本发明所提出的方法中，其代指深度多标签分类网络输出的未知标签预测概率。熵的单位通常为bit，但也用Sh、nat、Hart等单位进行计量，这取决于熵的定义用到对数的底数。假设有一离散随机变量X∈{x 1,x 2,...,x n}，则其信息熵的定义为：
H(X)＝Ε[I(X)]＝Ε[-ln(P(X))]
其中，P(X)为X的概率质量函数，E[·]为期望函数，I(X)即为X的信息量。
若已知或已对P(X)做出估计后，则信息熵的公式可以表示为：
Figure PCTCN2021137348-appb-000015
其中，b是对数所使用的底。当b＝2，熵的单位是bit；当b＝e，熵的单位是nat；而当b＝10，熵的单位是Hart。在本发明的方法中，深度多标签分类网络对于每一个样本的每一个未知标签的预测均可以视为一个离散随机变量X＝{x 1,x 2}，事件x 1即为将其预测为正标签，事件x 2即为将其预测为负标签。则P(x 1)即为网络输出的未知标签预测概率，相对应地，预测为负标签的概率即为P(x 2)＝1-P(x 1)。基于此设定，本发明可以进一步完成基于信息熵最大化的二元交叉熵损失函数正则化机制设计。
为了避免错误的假设带来的假阴性标签噪声的影响，同时为了防止深度多标签分类网络过拟合到大量的负标签，本发明采用了一种朴素但有效的思想来看待数据集中的未知标签，即不对这些未知标签做出任何假设并且承认它们是未知的。基于这种思想，深度多标签分类网络对于未知标签的预测概率应该尽量靠近0.5，即趋于0.5，即预测概率的信息熵应该尽量大。因此，本发明基于信息熵最大化的二元交叉熵损失函数正则化机制用于解决SPML问题，目的是最大化深度多标签分类网络对于未知标签预测概率的信息熵，称为(Entropy Maximization Regularizer，EMR)方法。假设给定一个仅有单个正标签的样本对(x (n),z (n))，其中x (n)为第n个输入图像，z (n)为第n个图像对应的标签向量。则EMR方法的损失函数为：
Figure PCTCN2021137348-appb-000016
其中，f (n)是模型关于x (n)的输出，
Figure PCTCN2021137348-appb-000017
是模型在第c个类上的预测概率。C是类别数目，
Figure PCTCN2021137348-appb-000018
表示第c个类存在正标签，
Figure PCTCN2021137348-appb-000019
表示第c个类存在负标签，
Figure PCTCN2021137348-appb-000020
为EMR方法引入的正则化项，其定义如下所示：
Figure PCTCN2021137348-appb-000021
其中，α为控制正则化强度大小的超参数，其值一般在0到1之间进行调节。在EMR方法中，深度多标签分类网络对于每一个样本的每一个未知标签的预测均可以视为一个离散随机变量，分别有预测为正标签和预测为负标签两种可能性。预测为正标签的概率即为网络输出的未知标签预测概率
Figure PCTCN2021137348-appb-000022
相对应的，预测为负标签的概率即为
Figure PCTCN2021137348-appb-000023
这样一来，EMR方法可以给未知标签施予最大信息熵的监督，而不是把它们全部当作负标签来处理，这契合了本发明采用的思想。EMR方法使得深度多标签分类网络可以利用真实正标签提供的监督信息进行学习，并对未知标签做出可区分性较强的预测，而不是对它们做出置信度过高甚至是错误的判断，这大幅提升了网络的泛化能力。
本发明所提出的基于信息熵最大化的二元交叉熵损失函数正则化机制，即EMR方法， 从一个独特的角度来看待数据集中的未知标签，即不对这些未知标签做出任何假设并且承认它们是未知的，在模型训练时的损失函数中最大化深度多标签分类网络对于未知标签预测概率的信息熵，用于解决SPML问题，使得模型可以免受假阴性标签噪声的影响，利用真实正标签提供的监督信息进行学习，并对未知标签做出可区分性较强的预测，EMR方法易于实现，不会引入额外的可学习参数，同时能够与任意的深度多标签分类网络相结合，性能优异，适宜处理超大规模多标签图像数据集。
本发明相比于ROLE方法，ROLE方法引入的EPR正则化方法需要提前统计数据集中每个样本的平均真实正标签数量，然而这在实际应用场景中难以实现，因为数据集中真实正标签的数目是未知的。此外，ROLE方法采用的标签估计模块给原本的网络模型引入额外的大量的可学习参数，大大增加网络训练时的时间和空间需求，且引入的可学习参数数目与数据集中的类别数目和样本数目正相关，这表明ROLE方法不适用于超大规模多标签图像数据集。而本发明提出的方法不会引入额外的可学习参数，适用场景更加广泛，可以与任意的深度多标签分类网络和任意规模的多标签数据集结合使用，同时性能远远优于ROLE方法。
为了验证本发明的优越性，采用本发明EMR方法和现有技术的AN方法和ROLE方法分别在PASCAL VOC 2012(VOC)、MS-COCO 2014(COCO)、NUS-WIDE(NUS)和CUB-200-2011(CUB)这四个大规模多标签图像数据集上进行实验验证，实验结果表明本发明所提出的方法性能优异，超越了所有的现存技术方案。实验结果为每个数据集中的最优性能，如下表所示：
Figure PCTCN2021137348-appb-000024
本发明旨在解决每个样本仅有单个正标签且其余标签均缺失的情况下，深度多标签分类网络的鲁棒学习问题。本发明提出了一种基于信息熵最大化的二元交叉熵损失函数正则化机制。利用本发明所提出的方法进行训练，深度多标签分类网络的泛化性能可以大幅提高，在极度标签缺失的情况下表现良好。
以上所述仅为本发明的优选实施例而已，并不用于限制本发明，对于本领域的技术人员来说，本发明可以有各种更改和变化。凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。

Claims (10)

  1. 基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法，其特征在于，包括：深度多标签分类网络对于每一个样本的每一个未知标签的预测均视为一个离散随机变量，包括预测为正标签和预测为负标签两种事件，预测为正标签的概率即为深度多标签分类网络输出的未知标签预测概率，控制深度多标签分类网络输出的未知标签预测概率的信息熵最大化。
  2. 根据权利要求1所述的基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法，其特征在于，所述方法中假设给定一个仅有单个正标签的样本对(x (n),z (n))，其中x (n)为第n个输入图像，z (n)为第n个图像对应的标签向量，则损失函数为：
    Figure PCTCN2021137348-appb-100001
    其中，f (n)是模型关于x (n)的输出，
    Figure PCTCN2021137348-appb-100002
    是模型在第c个类上的预测概率，C是类别数目，
    Figure PCTCN2021137348-appb-100003
    表示第c个类存在正标签，
    Figure PCTCN2021137348-appb-100004
    表示第c个类存在负标签，
    Figure PCTCN2021137348-appb-100005
    为正则化项。
  3. 根据权利要求2所述的基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法，其特征在于，所述正则化项
    Figure PCTCN2021137348-appb-100006
    定义为：
    Figure PCTCN2021137348-appb-100007
    其中，α为控制正则化强度大小的超参数。
  4. 根据权利要求3所述的基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法，其特征在于，所述超参数α的值在0到1之间进行调节。
  5. 根据权利要求4所述的基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法，其特征在于，所述方法中预测为正标签的概率即为网络输出的未知标签预测概率
    Figure PCTCN2021137348-appb-100008
    预测为负标签的概率即为
    Figure PCTCN2021137348-appb-100009
  6. 根据权利要求1所述的基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法，其特征在于，所述方法控制未知标签预测概率趋于0.5。
  7. 根据权利要求1所述的基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法，其特征在于，所述方法中假设有一个离散随机变量X∈{x 1,x 2,...,x n}，则其信息熵的定义为：
    H(X)＝E[I(X)]＝E[-ln(P(X))]
    其中，P(X)为X的概率质量函数，E[·]为期望函数，I(X)即为X的信息量。
  8. 根据权利要求7所述的基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法，其特征在于，所述方法中若已知或已对P(X)做出估计后，则信息熵的公式表示为：
    Figure PCTCN2021137348-appb-100010
    其中，b是对数所使用的底。
  9. 根据权利要求8所述的基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法，其特征在于，所述方法中当b＝2，信息熵的单位是bit；当b＝e，信息熵的单位是nat；当b＝10，信息熵的单位是Hart。
  10. 根据权利要求9所述的基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法，其特征在于，所述方法中深度多标签分类网络对于每一个样本的每一个未知标签的预测均视为一个离散随机变量X＝{x 1,x 2}，事件x 1即为预测为正标签，事件x 2即为预测为负标签，则P(x 1)即为网络输出的未知标签预测概率，P(x 2)＝1-P(x 1)即为预测为负标签的概率。
PCT/CN2021/137348 2021-12-13 2021-12-13 基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法 Ceased WO2023108317A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/137348 WO2023108317A1 (zh) 2021-12-13 2021-12-13 基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/137348 WO2023108317A1 (zh) 2021-12-13 2021-12-13 基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法

Publications (1)

Publication Number Publication Date
WO2023108317A1 true WO2023108317A1 (zh) 2023-06-22

Family

ID=86775227

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/137348 Ceased WO2023108317A1 (zh) 2021-12-13 2021-12-13 基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法

Country Status (1)

Country Link
WO (1) WO2023108317A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117906960A (zh) * 2023-12-14 2024-04-19 中国人民解放军海军航空大学 航空发动机的状态检测方法、系统、电子设备及存储介质
CN119669539A (zh) * 2024-12-02 2025-03-21 天津大学 一种基于样本和标签联合嵌入的多维数据探索方法
WO2025152448A1 (zh) * 2024-01-16 2025-07-24 华南理工大学 一种自适应深度迁移故障诊断方法、系统、装置及介质

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110163234A (zh) * 2018-10-10 2019-08-23 腾讯科技（深圳）有限公司 一种模型训练方法、装置和存储介质
US20200160177A1 (en) * 2018-11-16 2020-05-21 Royal Bank Of Canada System and method for a convolutional neural network for multi-label classification with partial annotations
US20200334565A1 (en) * 2019-04-16 2020-10-22 Siemens Aktiengesellschaft Maximum entropy regularised multi-goal reinforcement learning

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110163234A (zh) * 2018-10-10 2019-08-23 腾讯科技（深圳）有限公司 一种模型训练方法、装置和存储介质
US20200160177A1 (en) * 2018-11-16 2020-05-21 Royal Bank Of Canada System and method for a convolutional neural network for multi-label classification with partial annotations
US20200334565A1 (en) * 2019-04-16 2020-10-22 Siemens Aktiengesellschaft Maximum entropy regularised multi-goal reinforcement learning

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117906960A (zh) * 2023-12-14 2024-04-19 中国人民解放军海军航空大学 航空发动机的状态检测方法、系统、电子设备及存储介质
WO2025152448A1 (zh) * 2024-01-16 2025-07-24 华南理工大学 一种自适应深度迁移故障诊断方法、系统、装置及介质
CN119669539A (zh) * 2024-12-02 2025-03-21 天津大学 一种基于样本和标签联合嵌入的多维数据探索方法

Similar Documents

Publication Publication Date Title
CN111476168B (zh) 一种基于三阶段的跨域行人重识别方法和系统
Wang et al. Information theoretic counterfactual learning from missing-not-at-random feedback
Mu et al. Streaming classification with emerging new class by class matrix sketching
WO2023108317A1 (zh) 基于信息熵最大化正则机制的深度多标签分类网络鲁棒训练方法
Sánchez-Martin et al. Vaca: Designing variational graph autoencoders for causal queries
CN113704758B (zh) 一种黑盒攻击对抗样本生成方法及系统
CN112734032A (zh) 一种用于横向联邦学习的优化方法
CN113963165B (zh) 一种基于自监督学习的小样本图像分类方法及系统
Yu et al. Markov blanket feature selection using representative sets
CN116883751B (zh) 基于原型网络对比学习的无监督领域自适应图像识别方法
CN114547102A (zh) 基于梯度驱动数据生成的模型窃取攻击方法
CN113971733A (zh) 一种基于超图结构的模型训练方法、分类方法及装置
WO2023040195A1 (zh) 对象识别方法、网络的训练方法及装置、设备、介质及产品
CN117371541B (zh) 一种零知识、免真实数据的模型推理方法
Tong et al. Federated nonconvex sparse learning
Zhu et al. Improved self-paced learning framework for nonnegative matrix factorization
CN117391816A (zh) 一种异质图神经网络推荐方法、装置及设备
Jiang et al. Feed: Fairness-enhanced meta-learning for domain generalization
CN114792114B (zh) 一种基于黑盒多源域通用场景下的无监督域适应方法
Hou et al. Learning how to learn domain-invariant parameters for domain generalization
Thirumalairaj et al. Hybrid cuckoo search optimization based tuning scheme for deep neural network for intrusion detection systems in cloud environment
Zhao et al. Dual-level contrastive learning for unsupervised person re-identification
CN115690428A (zh) 一种用于语义分割的无源数据无监督领域自适应方法
Yao et al. Continual federated learning for dynamic data environments
CN118540117B (zh) 一种基于迁移学习和知识蒸馏的恶意流量检测方法及系统

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21967462

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 21967462

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 14.04.2025)