WO2022007349A1 - 一种神经网络调优方法、系统、终端以及存储介质 - Google Patents

一种神经网络调优方法、系统、终端以及存储介质 Download PDF

Info

Publication number
WO2022007349A1
WO2022007349A1 PCT/CN2020/139297 CN2020139297W WO2022007349A1 WO 2022007349 A1 WO2022007349 A1 WO 2022007349A1 CN 2020139297 W CN2020139297 W CN 2020139297W WO 2022007349 A1 WO2022007349 A1 WO 2022007349A1
Authority
WO
WIPO (PCT)
Prior art keywords
neural network
intra
regularization
class spacing
loss function
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/139297
Other languages
English (en)
French (fr)
Inventor
赵宝新
须成忠
赵娟娟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Institute of Advanced Technology of CAS
Original Assignee
Shenzhen Institute of Advanced Technology of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Institute of Advanced Technology of CAS filed Critical Shenzhen Institute of Advanced Technology of CAS
Publication of WO2022007349A1 publication Critical patent/WO2022007349A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Definitions

  • the present application belongs to the technical field of artificial intelligence, and particularly relates to a neural network tuning method, system, terminal and storage medium.
  • neural network With the development of artificial intelligence, deep neural network has achieved amazing results in many fields with its powerful fitting ability. However, there is still no strong theoretical support for the powerful fitting ability of neural network, and the ability of deep neural network still has a strong development space.
  • the existing network structure becomes more and more complex, and the number of network parameters also shows an explosive growth trend.
  • the training process of complex neural network becomes very slow. At the same time, training the network The resulting energy consumption also increases exponentially.
  • neural networks also have problems such as poor anti-interference and sensitivity to input data. In order to improve the above problems, it is necessary to improve the generalization ability of neural networks.
  • the present application provides a neural network tuning method, system, terminal, and storage medium, aiming to solve one of the above-mentioned technical problems in the prior art at least to a certain extent.
  • a neural network tuning method comprising the following steps:
  • the intra-class spacing regularization loss is added to the loss function of the neural network to obtain a loss function based on the intra-class spacing regularization; wherein, the intra-class spacing regularization loss represents the feature map of the same category of data output by the neural network
  • the intra-class spacing between is close; given a neural network structure z( xi , ⁇ ), ⁇ is the parameter of the neural network, x i is the input data, the loss function based on the regularization of the intra-class spacing is:
  • y i is the label corresponding to the input data x i
  • L(z(x i , ⁇ ),y i ) is the empirical loss function
  • ⁇ is a hyperparameter
  • the regularization loss for the intra-class spacing is the set of feature maps for all categories
  • the loss function based on the regularization of the intra-class spacing is inserted into a set proportion of the number of iterations, and the neural network optimization algorithm and the intra-class spacing regularization algorithm are overlapped to train the neural network.
  • the network is iteratively trained to obtain the optimal neural network.
  • the technical solution adopted in the embodiment of the present application further includes: the loss function based on the regularization of the intra-class distance is used to extract the category center of the feature map of the data of each category, and the calculation method of the category center is:
  • the formula for calculating the feature map of the category center is:
  • the technical solution adopted in the embodiment of the present application further includes: the network optimization algorithm adopts the overlapping training method of the network optimization algorithm and the intra-class spacing regularization algorithm to perform iterative training on the neural network, including:
  • a complete training of the neural network is recorded as an epoch. It is assumed that there are T epochs in the entire training process.
  • a network optimization algorithm is used for iterative training. After the epoch of the last network optimization algorithm training, insert 10% of the set number of iterations of the threshold for the iterative training of the intra-class spacing regularization algorithm, and use network optimization again after the epoch of the last intra-class spacing regularization algorithm training The algorithm is trained iteratively; so on until T epochs are completed.
  • the technical solution adopted in the embodiment of the present application further includes: the set threshold of the number of iterations of the network optimization algorithm is 20 times.
  • the loss function of the intra-class spacing regularization algorithm is:
  • the technical solutions adopted in the embodiments of the present application further include: the network optimization algorithm is a stochastic gradient descent method.
  • the loss function of the stochastic gradient descent method is:
  • a neural network tuning system comprising:
  • Loss function optimization module used to add an intra-class spacing regularization loss to the loss function of the neural network to obtain a loss function based on intra-class spacing regularization; wherein, the intra-class spacing regularization loss represents the output of the neural network.
  • the intra-class spacing between feature maps of the same category of data is close; given a neural network structure z( xi , ⁇ ), ⁇ is the parameter of the neural network, x i is the input data, the regularization based on the intra-class spacing
  • the loss function is:
  • y i is the label corresponding to the input data x i
  • L(z(x i , ⁇ ),y i ) is the empirical loss function
  • ⁇ is a hyperparameter
  • the regularization loss for the intra-class spacing is the set of feature maps for all categories
  • Model training module used to insert the loss function based on the regularization of the intra-class spacing in a set proportion of the number of iterations during the neural network training process, and use the network optimization algorithm and the intra-class spacing regularization algorithm for overlapping training
  • the neural network is iteratively trained in the manner to obtain the optimal neural network.
  • a terminal includes a processor and a memory coupled to the processor, wherein,
  • the memory stores program instructions for implementing the neural network tuning method
  • the processor is configured to execute the program instructions stored in the memory to control neural network tuning.
  • a storage medium storing program instructions executable by a processor, where the program instructions are used to execute the neural network tuning method.
  • the beneficial effects of the embodiments of the present application are: the neural network tuning method, system, terminal, and storage medium of the embodiments of the present application add intra-class spacing regularization loss to the loss function of the neural network, and In the process of network training, the intra-class spacing regularization algorithm is inserted into a certain proportion of the number of iterations to iteratively train the neural network.
  • the intra-class spacing regularization algorithm only needs to extract the category center of the feature map of the sample data of each category, and the rest The iterative process is consistent with the training method of the original network, so that the generalization ability and anti-interference of the network structure can be improved without changing the structure of the neural network, and the additional time overhead caused by this is small.
  • FIG. 1 is a flowchart of a neural network tuning method according to an embodiment of the present application.
  • Figure 2 is a schematic diagram of a neural network structure
  • FIG. 3 is a schematic diagram of neural network training according to an embodiment of the application.
  • FIG. 4 is a schematic structural diagram of a neural network tuning system according to an embodiment of the present application.
  • FIG. 5 is a schematic structural diagram of a terminal according to an embodiment of the present application.
  • FIG. 6 is a schematic structural diagram of a storage medium according to an embodiment of the present application.
  • the neural network tuning method of the embodiment of the present application adds an intra-class spacing regularization loss to the loss function of the neural network.
  • the inner distance regularization algorithm performs iterative training on the neural network, and the rest of the iterative process is consistent with the training method of the original network. Without changing the neural network structure, the generalization ability of the network structure is improved, and the additional time brought by it. Less overhead.
  • FIG. 1 is a flowchart of a neural network tuning method according to an embodiment of the present application.
  • the neural network tuning method of the embodiment of the present application includes the following steps:
  • Step 100 adding an intra-class spacing regularization loss to the loss function of the neural network to obtain a loss function based on the intra-class spacing regularization;
  • the neural network usually includes a convolution layer and a fully connected layer, as shown in FIG. 2, which is a schematic diagram of the structure of the neural network.
  • the trapezoid is the convolutional layer
  • the triangle is the fully connected layer.
  • the output feature map also has a very good clustering effect, that is, the intra-class spacing between the feature maps of the same category of data is as small as possible, based on This assumption is that the embodiments of the present application propose an intra-class spacing regularization loss.
  • ⁇ is the parameter of the neural network
  • x i is the input data
  • its loss function includes an empirical loss function and a regularization loss function
  • y i is the label corresponding to the input data x i
  • L(z(x i , ⁇ ),y i ) is the empirical loss function
  • ⁇ ( ⁇ ) is the regularization loss function
  • ⁇ is a hyperparameter , to balance the empirical loss function with the regularization loss function.
  • the intra-class spacing regularization loss is added to the loss function, and the loss function based on intra-class spacing regularization is obtained:
  • FIG. 3 wherein x i represents the input data obtained through the neural network, the data are tagged c x i, denoted l (x_i) ⁇ c, fm c is tagged as cluster center c of FIG feature of all samples, i.e. category center, Category between the representative sample from the center of the feature x i FIG.
  • the feature map of the category center is obtained by self-learning:
  • Step 200 During the training process of the neural network, insert a loss function (ICR) based on the regularization of the intra-class spacing in a set proportion of the number of iterations.
  • the neural network is iteratively trained to obtain the optimal neural network;
  • step 200 during the training process of the neural network, a complete training of the model using all the data samples of the training set is recorded as an epoch (epoch). 1,2,...,T].
  • Traditional network optimization algorithms include but are not limited to stochastic gradient descent (SGD), mini-batch gradient descent (MBGD), and the like. Taking SGD as an example, the loss function of the stochastic gradient descent method is:
  • the loss function of the intra-class spacing regularization algorithm is:
  • the overlapping method of the stochastic gradient descent method and the intra-class spacing regularization method is as follows: in the neural network training process, SGD is used for iterative training first, and whenever the number of SGD iterations reaches the set threshold, the After the epoch of the last SGD training, an iteration number of 10% of the set number of thresholds is inserted for ICR iterative training, and then SGD is used again for iterative training after the epoch of the last ICR training. This cycle is repeated until T epochs are completed.
  • FIG. 3 it is a schematic diagram of training a neural network according to an embodiment of the present application.
  • the gray represents the training method of SGD
  • the black represents the training method of ICR
  • the threshold for the number of times of SGD iterative training is 20
  • the performance improvement effect of the neural network is the best. That is, two ICR trainings are inserted after every 20 epochs of SGD training. It can be understood that the threshold of the number of times and the insertion ratio of the ICR method can be adjusted according to the actual operation.
  • the intra-class spacing regularization algorithm only needs to extract the category center of the feature map of the sample data of each category, so as not to change the neural network
  • the additional burden on the training time of the neural network can be ignored.
  • FIG. 4 is a schematic structural diagram of a neural network tuning system according to an embodiment of the present application.
  • the neural network tuning system of the embodiment of the present application includes:
  • Loss function optimization module It is used to add the regularization loss of the intra-class spacing to the loss function of the neural network to obtain a loss function based on the regularization of the intra-class spacing; where, given a neural network structure z(x i , ⁇ ), ⁇ is the parameter of the neural network, x i is the input data, and its loss function includes an empirical loss function and a regularization loss function.
  • the loss function can be summarized as:
  • y i is the label corresponding to the input data x i
  • L(z(x i , ⁇ ),y i ) is the empirical loss function
  • ⁇ ( ⁇ ) is the regularization loss function
  • ⁇ is a hyperparameter , to balance the empirical loss function with the regularization loss function.
  • the intra-class spacing regularization loss is added to the loss function, and the loss function based on intra-class spacing regularization is obtained:
  • Equation (3) Represents the feature map of the input data x i after passing through the neural network, the label of the data x i is c, denoted as l( xi ) ⁇ c, and fm c is the cluster center of the feature map of all samples with the label c, that is category Center, d (fm xi, fm c ) represents the distance between sample x i of the feature categories FIG center belongs.
  • the feature map of the category center is obtained by self-learning:
  • Model training module It is used to insert a loss function (ICR) based on the regularization of intra-class spacing in a set proportion of iterations during the neural network training process, using the traditional network optimization algorithm and the overlapping training of the intra-class spacing regularization algorithm Iteratively trains the neural network in a way to obtain the optimal neural network;
  • ICR loss function
  • epoch epoch
  • MBGD mini-batch gradient descent
  • the loss function of the intra-class spacing regularization algorithm is:
  • the overlapping method of the stochastic gradient descent method and the intra-class spacing regularization method is as follows: in the neural network training process, SGD is used for iterative training first, and whenever the number of SGD iterations reaches the set threshold, the After the epoch of the last SGD training, an iteration number of 10% of the set number of thresholds is inserted for ICR iterative training, and then SGD is used again for iterative training after the epoch of the last ICR training. This cycle is repeated until T epochs are completed.
  • the threshold for the number of times of SGD iterative training is 20
  • the performance improvement effect of the neural network is the best. That is, two ICR trainings are inserted after every 20 epochs of SGD training. It can be understood that the threshold of the number of times and the insertion ratio of the ICR method can be adjusted according to the actual operation.
  • FIG. 5 is a schematic structural diagram of a terminal according to an embodiment of the present application.
  • the terminal 50 includes a processor 51 and a memory 52 coupled to the processor 51 .
  • the memory 52 stores program instructions for implementing the above-mentioned neural network tuning method.
  • the processor 51 is configured to execute program instructions stored in the memory 52 to control neural network tuning.
  • the processor 51 may also be referred to as a CPU (Central Processing Unit, central processing unit).
  • the processor 51 may be an integrated circuit chip with signal processing capability.
  • the processor 51 may also be a general purpose processor, digital signal processor (DSP), application specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component .
  • DSP digital signal processor
  • ASIC application specific integrated circuit
  • FPGA off-the-shelf programmable gate array
  • a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • FIG. 6 is a schematic structural diagram of a storage medium according to an embodiment of the present application.
  • the storage medium of this embodiment of the present application stores a program file 61 capable of implementing all the above methods, wherein the program file 61 may be stored in the above-mentioned storage medium in the form of a software product, and includes several instructions to enable a computer device (which may It is a personal computer, a server, or a network device, etc.) or a processor that executes all or part of the steps of the methods of the various embodiments of the present invention.
  • a computer device which may It is a personal computer, a server, or a network device, etc.
  • a processor that executes all or part of the steps of the methods of the various embodiments of the present invention.
  • the aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes , or terminal devices such as computers, servers, mobile phones, and tablets.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)

Abstract

本申请涉及一种神经网络调优方法、系统、终端以及存储介质。所述方法包括：在神经网络的损失函数中加入类内间距正则化损失，得到基于类内间距正则化的损失函数；所述类内间距正则化损失表示所述神经网络输出的同一类别的数据的特征图之间的类内间距接近；在所述神经网络训练过程中，在设定比例的迭代次数中插入所述基于类内间距正则化的损失函数，采用网络优化算法与类内间距正则化算法交叠训练的方式对所述神经网络进行迭代训练，得到最优神经网络。本申请实施例通过在一定比例的迭代次数中插入类内间距正则化算法对神经网络进行迭代训练，在不改变神经网络结构的情况下，提升网络结构的泛化能力和抗干扰性，且由此带来的额外时间开销较小。

Description

一种神经网络调优方法、系统、终端以及存储介质 技术领域
本申请属于人工智能技术领域，特别涉及一种神经网络调优方法、系统、终端以及存储介质。
背景技术
随着人工智能的发展，深度神经网络以其强大的拟合能力在众多领域取得了惊人的成绩。然而神经神经网络的强大拟合能力依旧没有一个强有力的理论支持，深度神经网络的能力依旧具有很强的开发空间。为了得到更好的神经网络泛化能力，现有的网络结构变得越来越复杂，网络的参数量也呈现爆炸式增长的趋势，复杂的神经网络的训练过程变得非常缓慢，同时训练网络所带来的能量消耗也成倍增长。另外，神经网络也存在抗干扰性差、对输入数据敏感等问题。为了改善上述问题，需要对神经网络泛化的能力进行提升。
目前，提升神经网络泛化能力的主要方法分为三大类，具体为：
一、从网络结构出发，设计更好的网络结构。这种方法主要从深度神经网络的深度、宽度和广度三个角度进行网络结构的修正，复杂的网络结构可以带来更好的性能收益，但是也会导致网络的参数量巨大，训练时间较长，能源消耗较大。
二、提供更多的训练数据，使得神经网络能够识别更多的数据类型。该方法需要更多优质的数据，对数据进行处理需要消耗非常多的人力和算力。
三、从训练的过程出发，寻找更好的超参数或者优化方法。该方法可以在不改变网络结构的条件下，提高网络性能。然而超参搜索也需要迭代非常多次的网络训练过程，对算力和能源的消耗非常大。
发明内容
本申请提供了一种神经网络调优方法、系统、终端以及存储介质，旨在至少在一定程度上解决现有技术中的上述技术问题之一。
为了解决上述问题，本申请提供了如下技术方案：
一种神经网络调优方法，包括以下步骤：
在神经网络的损失函数中加入类内间距正则化损失，得到基于类内间距正则化的损失函数；其中，所述类内间距正则化损失表示所述神经网络输出的同一类别的数据的特征图之间的类内间距接近；给定一个神经网络结构z(x i,ω)，ω是神经网络的参数，x i是输入数据，所述基于类内间距正则化的损失函数为：
Figure PCTCN2020139297-appb-000001
上式中，y i是输入数据x i对应的标签，L(z(x i,ω),y i)是经验损失函数，λ是一个超参数，
Figure PCTCN2020139297-appb-000002
为类内间距正则化损失，
Figure PCTCN2020139297-appb-000003
是所有类别的特征图集合；
在所述神经网络训练过程中，在设定比例的迭代次数中插入所述基于类内间距正则化的损失函数，采用网络优化算法与类内间距正则化算法交叠训练的方式对所述神经网络进行迭代训练，得到最优神经网络。
本申请实施例采取的技术方案还包括：所述基于类内间距正则化的损失函数用于提取各个类别的数据的特征图的类别中心，所述类别中心计算方式为：
将所述基于类内间距正则化的损失函数展开，得：
Figure PCTCN2020139297-appb-000004
上述公式中，
Figure PCTCN2020139297-appb-000005
代表输入数据x i经过神经网络后得到的特征图，数据x i的标签为c，记作l(x i)∈c，fm c为标签为c的所有数据的特征图的类别中心，
Figure PCTCN2020139297-appb-000006
代表数据x i的特征图与所属类别中心之间的距离；
所述类别中心的特征图计算公式为：
Figure PCTCN2020139297-appb-000007
上式中，|X c|代表所有样本X中类别属于c的样本数目。
本申请实施例采取的技术方案还包括：所述网络优化算法采用网络优化算法与类内间距正则化算法交叠训练的方式对所述神经网络进行迭代训练包括：
将所述神经网络的一次完整训练记为一次epoch，假设整个训练过程共有T个epoch，首先采用网络优化算法进行迭代训练，每当所述网络优化算法的迭代次数达到设定次数阈值时，在最后一次网络优化算法训练的epoch之 后插入所述设定次数阈值的10％的迭代次数进行类内间距正则化算法迭代训练，并在最后一次类内间距正则化算法训练的epoch之后再次采用网络优化算法进行迭代训练；如此循环，直到完成T个epoch。
本申请实施例采取的技术方案还包括：所述网络优化算法迭代次数的设定次数阈值为20次。
本申请实施例采取的技术方案还包括：所述类内间距正则化算法的损失函数为：
Figure PCTCN2020139297-appb-000008
本申请实施例采取的技术方案还包括：所述网络优化算法为随机梯度下降法。
本申请实施例采取的技术方案还包括：所述随机梯度下降法的损失函数为：
Figure PCTCN2020139297-appb-000009
本申请实施例采取的另一技术方案为：一种神经网络调优系统，包括：
损失函数优化模块：用于在神经网络的损失函数中加入类内间距正则化损失，得到基于类内间距正则化的损失函数；其中，所述类内间距正则化损失表示所述神经网络输出的同一类别的数据的特征图之间的类内间距接近；给定一个神经网络结构z(x i,ω)，ω是神经网络的参数，x i是输入数据，所述基于类内间距正则化的损失函数为：
Figure PCTCN2020139297-appb-000010
上式中，y i是输入数据x i对应的标签，L(z(x i,ω),y i)是经验损失函数，λ是一个超参数，
Figure PCTCN2020139297-appb-000011
为类内间距正则化损失，
Figure PCTCN2020139297-appb-000012
是所有类别的特征图集合；
模型训练模块：用于在所述神经网络训练过程中，在设定比例的迭代次数中插入所述基于类内间距正则化的损失函数，采用网络优化算法与类内间距正则化算法交叠训练的方式对所述神经网络进行迭代训练，得到最优神经网络。
本申请实施例采取的又一技术方案为：一种终端，所述终端包括处理器、与所述处理器耦接的存储器，其中，
所述存储器存储有用于实现所述神经网络调优方法的程序指令；
所述处理器用于执行所述存储器存储的所述程序指令以控制神经网络调优。
本申请实施例采取的又一技术方案为：一种存储介质，存储有处理器可运行的程序指令，所述程序指令用于执行所述神经网络调优方法。
相对于现有技术，本申请实施例产生的有益效果在于：本申请实施例的神经网络调优方法、系统、终端及存储介质通过在神经网络的损失函数中加入类内间距正则化损失，在网络训练的过程中，在一定比例的迭代次数中插入类内间距正则化算法对神经网络进行迭代训练，类内间距正则化算法只需提取各个类别的样本数据的特征图的类别中心，而其余迭代过程与原始网络 的训练方法一致，从而在不改变神经网络结构的情况下，提升网络结构的泛化能力和抗干扰性，且由此带来的额外时间开销较小。
附图说明
图1是本申请实施例的神经网络调优方法的流程图；
图2为神经网络结构示意图；
图3为本申请实施例的神经网络训练示意图；
图4为本申请实施例的神经网络调优系统结构示意图；
图5为本申请实施例的终端结构示意图；
图6为本申请实施例的存储介质的结构示意图。
具体实施方式
为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请进行进一步详细说明。应当理解，此处所描述的具体实施例仅用以解释本申请，并不用于限定本申请。
为了解决现有技术的不足，本申请实施例的神经网络调优方法通过在神经网络的损失函数中加入类内间距正则化损失，在网络训练的过程中，在一定比例的迭代次数中插入类内间距正则化算法对神经网络进行迭代训练，而其余迭代过程与原始网络的训练方法一致，在不改变神经网络结构的情况下，提升网络结构的泛化能力，且由此带来的额外时间开销较小。
请参阅图1，是本申请实施例的神经网络调优方法的流程图。本申请实施例的神经网络调优方法包括以下步骤：
步骤100：在神经网络的损失函数中加入类内间距正则化损失，得到基于类内间距正则化的损失函数；
步骤100中，神经网络通常包括卷积层和全连接层，具体如图2所示，为神经网络结构示意图。其中梯形为卷积层，三角形为全连接层。以图片分类任务为例，当输入一张图片后，首先经过若干层卷积层进行特征提取，并输出特征图(Feature Map)，然后经过若干全连接层得到输出的图片分类结果。
假设输入数据记作X＝{x 1,x 2,…,x n}，其中n为样本数；对于每一个输入x i，其特征图输出记作FM i，输入数据的标签记作Y＝{1,2,…,C}，数据的分类类别一共包括C类。为了使神经网络可以对输入数据进行非常精确的识别分类，假设其输出的特征图也具有非常好的聚类效果，即同一类别的数据的特征图之间的类内间距尽可能的小，基于此假设本申请实施例提出类内间距正则化损失。
给定一个神经网络结构z(x_i,ω)，ω是神经网络的参数，x i是输入数据，其损失函数包括经验损失函数和正则化损失函数，损失函数可以归纳为：
Figure PCTCN2020139297-appb-000013
式(1)中，y i是输入数据x i对应的标签，L(z(x i,ω),y i)是经验损失函数，Ω(ω)是正则化损失函数，λ是一个超参数，用以平衡经验损失函数与正则化损失函数。
基于上式，将类内间距正则化损失加入到损失函数中，得到基于类内间距正则化的损失函数：
Figure PCTCN2020139297-appb-000014
式(2)中，
Figure PCTCN2020139297-appb-000015
代表类内间距正则化损失,
Figure PCTCN2020139297-appb-000016
是所有类别的特征图集合。将其展开为：
Figure PCTCN2020139297-appb-000017
式(3)中，
Figure PCTCN2020139297-appb-000018
代表输入数据x i经过神经网络后得到的特征图，数据x i的标签为c，记作l(x_i)∈c，fm c为标签为c的所有样本的特征图的聚类中心，即类别中心，
Figure PCTCN2020139297-appb-000019
代表样本x i的特征图与所属类别中心之间的距离。
其中，类别中心的特征图通过自学习得到：
Figure PCTCN2020139297-appb-000020
式(4)中，|X c|代表所有样本X中类别属于c的样本数目。
步骤200：在神经网络训练过程中，在设定比例的迭代次数中插入基于类内间距正则化的损失函数(ICR)，采用传统网络优化算法与类内间距正则化算法交叠训练的方式对神经网络进行迭代训练，得到最优神经网络；
步骤200中，神经网络训练过程中，使用训练集的全部数据样本对模型进行一次完整训练记为一次epoch(时期)，整个训练过程总共有T个epoch，所有epoch的索引记作序列[0,1,2,…,T]。传统网络优化算法包括但不限于随机梯度下降法(Stochastic gradient descent,SGD)、小批量梯度下降法(Mini- batch gradient descent，MBGD)等。以SGD为例，随机梯度下降法的损失函数为：
Figure PCTCN2020139297-appb-000021
类内间距正则化算法的损失函数为：
Figure PCTCN2020139297-appb-000022
本申请实施例中，随机梯度下降法与类内间距正则化方法的交叠方式为：在神经网络训练过程中，首先采用SGD进行迭代训练，每当SGD迭代次数达到设定次数阈值时，在最后一次SGD训练的epoch之后插入该设定次数阈值的10％的迭代次数进行ICR迭代训练，然后在最后一次ICR训练的epoch之后再次采用SGD进行迭代训练。如此循环，直到完成T个epoch。具体如图3所示，为本申请实施例的神经网络训练示意图。其中灰色代表SGD的训练方法，黑色代表ICR的训练方法，SGD与ICR交叠训练。优选地，通过实验对比，SGD迭代训练的次数阈值为20次时，神经网络的性能提升效果最好。即：每间隔20个SGD训练的epoch之后插入两次ICR训练，可以理解，该次数阈值以及ICR方法的插入比例可根据实际操作进行调节。
基于上述，本申请实施例通过在训练过程中插入较小比例的类内间距正则化方法，类内间距正则化算法只需提取各个类别的样本数据的特征图的类别中心，从而在不改变神经网络的结构的情况下，提升神经网络的泛化能力和抗干扰性，对神经网络的训练时间带来的额外负担也可以忽略不计。
为了验证本申请实施例的可行性和有效性，使用图片识别的公用数据集CIFAR100进行了实验。实验结果表明，本申请实施例在VGG(Visual Geometry Group，深度卷积神经网络)、ResNet(Residual Network，残差网络)等都取得了非常好的性能提升和对白噪声的抗干扰能力，同时训练过程带来的额外时间开销小于原有训练时间的5％。
请请参阅图4，是本申请实施例的神经网络调优系统的结构示意图。本申请实施例的神经网络调优系统包括：
损失函数优化模块：用于在神经网络的损失函数中加入类内间距正则化损失，得到基于类内间距正则化的损失函数；其中，给定一个神经网络结构z(x i,ω)，ω是神经网络的参数，x i是输入数据，其损失函数包括经验损失函数和正则化损失函数，损失函数可以归纳为：
Figure PCTCN2020139297-appb-000023
式(1)中，y i是输入数据x i对应的标签，L(z(x i,ω),y i)是经验损失函数，Ω(ω)是正则化损失函数，λ是一个超参数，用以平衡经验损失函数与正则化损失函数。
基于上式，将类内间距正则化损失加入到损失函数中，得到基于类内间距正则化的损失函数：
Figure PCTCN2020139297-appb-000024
式(2)中，
Figure PCTCN2020139297-appb-000025
代表类内间距正则化损失，
Figure PCTCN2020139297-appb-000026
是所有类别的特征图集合。将其展开为：
Figure PCTCN2020139297-appb-000027
式(3)中，
Figure PCTCN2020139297-appb-000028
代表输入数据x i经过神经网络后得到的特征图，数据x i的标签为c，记作l(x i)∈c，fm c为标签为c的所有样本的特征图的聚类中心，即类别中心，d(fm xi,fm c)代表样本x i的特征图与所属类别中心之间的距离。
其中，类别中心的特征图通过自学习得到：
Figure PCTCN2020139297-appb-000029
式(4)中，|X C|代表所有样本X中类别属于c的样本数目。
模型训练模块：用于在神经网络训练过程中，在设定比例的迭代次数中插入基于类内间距正则化的损失函数(ICR)，采用传统网络优化算法与类内间距正则化算法交叠训练的方式对神经网络进行迭代训练，得到最优神经网络；
具体的，神经网络训练过程中，使用训练集的全部数据样本对模型进行一次完整训练记为一次epoch(时期)，整个训练过程总共有T个epoch，所有epoch的索引记作序列[0,1,2,…,T]。传统网络优化算法包括但不限于随机梯度下降法(Stochastic gradient descent,SGD)、小批量梯度下降法(Mini-batch gradient descent，MBGD)等。以SGD为例，随机梯度下降法的损失函数为：
Figure PCTCN2020139297-appb-000030
类内间距正则化算法的损失函数为：
Figure PCTCN2020139297-appb-000031
本申请实施例中，随机梯度下降法与类内间距正则化方法的交叠方式为：在神经网络训练过程中，首先采用SGD进行迭代训练，每当SGD迭代次数达到设定次数阈值时，在最后一次SGD训练的epoch之后插入该设定次数阈值的10％的迭代次数进行ICR迭代训练，然后在最后一次ICR训练的epoch之后再次采用SGD进行迭代训练。如此循环，直到完成T个epoch。优选地，通过实验对比，SGD迭代训练的次数阈值为20次时，神经网络的性能提升效果最好。即：每间隔20个SGD训练的epoch之后插入两次ICR训练，可以理解，该次数阈值以及ICR方法的插入比例可根据实际操作进行调节。
请参阅图5，为本申请实施例的终端结构示意图。该终端50包括处理器51、与处理器51耦接的存储器52。
存储器52存储有用于实现上述神经网络调优方法的程序指令。
处理器51用于执行存储器52存储的程序指令以控制神经网络调优。
其中，处理器51还可以称为CPU(Central Processing Unit，中央处理单元)。处理器51可能是一种集成电路芯片，具有信号的处理能力。处理器51还可以是通用处理器、数字信号处理器(DSP)、专用集成电路(ASIC)、现成可编程门阵列(FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
请参阅图6，为本申请实施例的存储介质的结构示意图。本申请实施例的存储介质存储有能够实现上述所有方法的程序文件61，其中，该程序文件61 可以以软件产品的形式存储在上述存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或处理器(processor)执行本发明各个实施方式方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质，或者是计算机、服务器、手机、平板等终端设备。
对所公开的实施例的上述说明，使本领域专业技术人员能够实现或使用本申请。对这些实施例的多种修改对本领域的专业技术人员来说将是显而易见的，本申请中所定义的一般原理可以在不脱离本申请的精神或范围的情况下，在其它实施例中实现。因此，本申请将不会被限制于本申请所示的这些实施例，而是要符合与本申请所公开的原理和新颖特点相一致的最宽的范围。

Claims (10)

  1. 一种神经网络调优方法，其特征在于，包括以下步骤：
    在神经网络的损失函数中加入类内间距正则化损失，得到基于类内间距正则化的损失函数；其中，所述类内间距正则化损失表示所述神经网络输出的同一类别的数据的特征图之间的类内间距接近；给定一个神经网络结构z(x i,ω)，ω是神经网络的参数，x i是输入数据，所述基于类内间距正则化的损失函数为：
    Figure PCTCN2020139297-appb-100001
    上式中，y i是输入数据x i对应的标签，L(z(x i,ω),y i)是经验损失函数，λ是一个超参数，
    Figure PCTCN2020139297-appb-100002
    为类内间距正则化损失，
    Figure PCTCN2020139297-appb-100003
    是所有类别的特征图集合；
    在所述神经网络训练过程中，在设定比例的迭代次数中插入所述基于类内间距正则化的损失函数，采用网络优化算法与类内间距正则化算法交叠训练的方式对所述神经网络进行迭代训练，得到最优神经网络。
  2. 根据权利要求1所述的神经网络调优方法，其特征在于，所述基于类内间距正则化的损失函数用于提取各个类别的数据的特征图的类别中心，所述类别中心计算方式为：
    将所述基于类内间距正则化的损失函数展开，得：
    Figure PCTCN2020139297-appb-100004
    上述公式中，
    Figure PCTCN2020139297-appb-100005
    代表输入数据x i经过神经网络后得到的特征图，数据x i的标签为c，记作l(x i)∈c，fm c为标签为c的所有数据的特征图的类别中心，
    Figure PCTCN2020139297-appb-100006
    代表数据x i的特征图与所属类别中心之间的距离；
    所述类别中心的特征图计算公式为：
    Figure PCTCN2020139297-appb-100007
    上式中，|X c|代表所有样本X中类别属于c的样本数目。
  3. 根据权利要求2所述的神经网络调优方法，其特征在于，所述网络优化算法采用网络优化算法与类内间距正则化算法交叠训练的方式对所述神经网络进行迭代训练包括：
    将所述神经网络的一次完整训练记为一次epoch，假设整个训练过程共有T个epoch，首先采用网络优化算法进行迭代训练，每当所述网络优化算法的迭代次数达到设定次数阈值时，在最后一次网络优化算法训练的epoch之后插入所述设定次数阈值的10％的迭代次数进行类内间距正则化算法迭代训练，并在最后一次类内间距正则化算法训练的epoch之后再次采用网络优化算法进行迭代训练；如此循环，直到完成T个epoch。
  4. 根据权利要求3所述的神经网络调优方法，其特征在于，所述网络优化算法迭代次数的设定次数阈值为20次。
  5. 根据权利要求1至4任一项所述的神经网络调优方法，其特征在于，所述类内间距正则化算法的损失函数为：
    Figure PCTCN2020139297-appb-100008
  6. 根据权利要求1至4任一项所述的神经网络调优方法，其特征在于，所述网络优化算法为随机梯度下降法。
  7. 根据权利要求6所述的神经网络调优方法，其特征在于，所述随机梯度下降法的损失函数为：
    Figure PCTCN2020139297-appb-100009
  8. 一种神经网络调优系统，其特征在于，包括：
    损失函数优化模块：用于在神经网络的损失函数中加入类内间距正则化损失，得到基于类内间距正则化的损失函数；其中，所述类内间距正则化损失表示所述神经网络输出的同一类别的数据的特征图之间的类内间距接近；给定一个神经网络结构z(x i,ω)，ω是神经网络的参数，x i是输入数据，所述基于类内间距正则化的损失函数为：
    Figure PCTCN2020139297-appb-100010
    上式中，y i是输入数据x i对应的标签，L(z(x i,ω),y i)是经验损失函数，λ是一个超参数，
    Figure PCTCN2020139297-appb-100011
    为类内间距正则化损失，
    Figure PCTCN2020139297-appb-100012
    是所有类别的特征图集合；
    模型训练模块：用于在所述神经网络训练过程中，在设定比例的迭代次数中插入所述基于类内间距正则化的损失函数，采用网络优化算法与类内间距正则化算法交叠训练的方式对所述神经网络进行迭代训练，得到最优神经网络。
  9. 一种终端，其特征在于，所述终端包括处理器、与所述处理器耦接的存储器，其中，
    所述存储器存储有用于实现权利要求1-7任一项所述的神经网络调优方法的程序指令；
    所述处理器用于执行所述存储器存储的所述程序指令以控制神经网络调优。
  10. 一种存储介质，其特征在于，存储有处理器可运行的程序指令，所述程序指令用于执行权利要求1至7任一项所述神经网络调优方法。
PCT/CN2020/139297 2020-07-09 2020-12-25 一种神经网络调优方法、系统、终端以及存储介质 Ceased WO2022007349A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010657269.2 2020-07-09
CN202010657269.2A CN111860834B (zh) 2020-07-09 2020-07-09 一种神经网络调优方法、系统、终端以及存储介质

Publications (1)

Publication Number Publication Date
WO2022007349A1 true WO2022007349A1 (zh) 2022-01-13

Family

ID=73151967

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/139297 Ceased WO2022007349A1 (zh) 2020-07-09 2020-12-25 一种神经网络调优方法、系统、终端以及存储介质

Country Status (2)

Country Link
CN (1) CN111860834B (zh)
WO (1) WO2022007349A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115526242A (zh) * 2022-09-14 2022-12-27 华中科技大学 一种日负荷曲线聚类方法

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111860834B (zh) * 2020-07-09 2024-05-24 中国科学院深圳先进技术研究院 一种神经网络调优方法、系统、终端以及存储介质
CN112507895A (zh) * 2020-12-14 2021-03-16 广东电力信息科技有限公司 基于大数据分析自动归类资质证书文件的方法及装置
CN114387457A (zh) * 2021-12-27 2022-04-22 腾晖科技建筑智能(深圳)有限公司 一种基于调参的人脸类内类间距优化方法
CN115470847B (zh) * 2022-09-05 2026-04-17 中国科学院深圳先进技术研究院 一种网络通讯检测方法、装置、设备以及存储介质

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019208564A1 (ja) * 2018-04-26 2019-10-31 日本電信電話株式会社 ニューラルネットワーク学習装置、ニューラルネットワーク学習方法、プログラム
CN110598552A (zh) * 2019-08-09 2019-12-20 吉林大学 基于改进粒子群算法优化卷积神经网络的表情识别方法
CN111860834A (zh) * 2020-07-09 2020-10-30 中国科学院深圳先进技术研究院 一种神经网络调优方法、系统、终端以及存储介质

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10650286B2 (en) * 2017-09-07 2020-05-12 International Business Machines Corporation Classifying medical images using deep convolution neural network (CNN) architecture
US11429862B2 (en) * 2018-03-20 2022-08-30 Sri International Dynamic adaptation of deep neural networks
CA3094822A1 (en) * 2018-03-26 2019-10-03 Pediametrix Inc. Systems and methods of measuring the body based on image analysis
CN109800754B (zh) * 2018-12-06 2020-11-06 杭州电子科技大学 一种基于卷积神经网络的古字体分类方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019208564A1 (ja) * 2018-04-26 2019-10-31 日本電信電話株式会社 ニューラルネットワーク学習装置、ニューラルネットワーク学習方法、プログラム
CN110598552A (zh) * 2019-08-09 2019-12-20 吉林大学 基于改进粒子群算法优化卷积神经网络的表情识别方法
CN111860834A (zh) * 2020-07-09 2020-10-30 中国科学院深圳先进技术研究院 一种神经网络调优方法、系统、终端以及存储介质

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
ANONYMOUS: "Neural Network Optimization: Regularization (Regularized Loss Function)", CSDN, 1 August 2018 (2018-08-01), XP055885477, Retrieved from the Internet <URL:https://blog.csdn.net/congcong7267/article/details/81324467> *
YUI ARAKAGI: "Neural Network and Deep Learning CHAP-Network Optimization and Regularization", DOC88.COM WEBPAGE, CN, pages 1 and - 7, XP009533497, Retrieved from the Internet <URL:http://www.doc88.com/p-8711736075598.html> *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115526242A (zh) * 2022-09-14 2022-12-27 华中科技大学 一种日负荷曲线聚类方法

Also Published As

Publication number Publication date
CN111860834A (zh) 2020-10-30
CN111860834B (zh) 2024-05-24

Similar Documents

Publication Publication Date Title
WO2022007349A1 (zh) 一种神经网络调优方法、系统、终端以及存储介质
US11941721B2 (en) Using watermark information and weight information to train an embedded neural network model
CN113505797B (zh) 模型训练方法、装置、计算机设备和存储介质
CN104463202B (zh) 一种多类图像半监督分类方法及系统
CN109086811B (zh) 多标签图像分类方法、装置及电子设备
CN111666994A (zh) 样本图像数据增强方法、装置、电子设备及存储介质
CN103729428B (zh) 一种大数据分类方法及系统
CN112567355A (zh) 用于知识库补全的端到端的结构感知卷积网络
CN114299305B (zh) 聚合密集和注意力多尺度特征的显著性目标检测算法
WO2022095356A1 (zh) 用于图像分类的迁移学习方法、相关装置及存储介质
WO2022042123A1 (zh) 图像识别模型生成方法、装置、计算机设备和存储介质
CN114565807A (zh) 训练目标图像检索模型的方法和装置
CN111709516A (zh) 神经网络模型的压缩方法及压缩装置、存储介质、设备
CN114333062B (zh) 基于异构双网络和特征一致性的行人重识别模型训练方法
CN112270259B (zh) 基于轻量级卷积神经网络的sar图像舰船目标快速检测方法
CN114677565B (zh) 特征提取网络的训练方法和图像处理方法、装置
CN116450941B (zh) 基于洛伦兹图卷积网络的书籍推荐方法及系统
CN112001401A (zh) 实例分割网络的训练模型及训练方法，实例分割网络
CN110222607A (zh) 人脸关键点检测的方法、装置及系统
CN111783655A (zh) 一种图像处理方法、装置、电子设备和存储介质
CN114677695A (zh) 表格解析方法、装置、计算机设备和存储介质
CN115982346A (zh) 一种问答库构建方法、终端设备及存储介质
CN119415664B (zh) 基于长期记忆的检索增强生成方法、系统、电子设备及存储介质
CN116342955A (zh) 一种基于改进的特征金字塔网络的目标检测方法及系统
CN112634869B (zh) 命令词识别方法、设备及计算机存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20944480

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20944480

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 20944480

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 03.07.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 20944480

Country of ref document: EP

Kind code of ref document: A1