WO2022178908A1 - 神经网络的剪枝方法及其剪枝装置、存储介质 - Google Patents

神经网络的剪枝方法及其剪枝装置、存储介质 Download PDF

Info

Publication number
WO2022178908A1
WO2022178908A1 PCT/CN2021/078455 CN2021078455W WO2022178908A1 WO 2022178908 A1 WO2022178908 A1 WO 2022178908A1 CN 2021078455 W CN2021078455 W CN 2021078455W WO 2022178908 A1 WO2022178908 A1 WO 2022178908A1
Authority
WO
WIPO (PCT)
Prior art keywords
value
neural network
layer
residual layer
importance
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2021/078455
Other languages
English (en)
French (fr)
Inventor
梁栋
王位
朱燕杰
崔卓须
刘新
郑海荣
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Institute of Advanced Technology of CAS
Original Assignee
Shenzhen Institute of Advanced Technology of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Institute of Advanced Technology of CAS filed Critical Shenzhen Institute of Advanced Technology of CAS
Publication of WO2022178908A1 publication Critical patent/WO2022178908A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0495—Quantised networks; Sparse networks; Compressed networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning

Definitions

  • the invention belongs to the technical field of machine learning, and in particular, relates to a pruning method of a neural network, a pruning device, and a computer-readable storage medium.
  • Deep learning is currently widely used in various industries, but in many real-world applications, the deployment of deep learning networks is a big challenge. On the one hand, deep learning network parameters are huge, and the demand for memory is very high. On the other hand, deep learning requires a lot of computation. Especially the commonly used convolutional neural network, which has a large number of redundant parameters, often requires network pruning.
  • the basic idea of network pruning is to use the different importance of different parameters to prune those redundant parameters to make the network more streamlined and reduce storage occupation and calculation amount.
  • the idea of pruning for parameter importance is to establish an evaluation standard, then sort the parameters of the network according to the evaluation standard, set a threshold, and then prune those network parameters that are smaller than the threshold.
  • the network is pruned by the scaling factor of the BN layer, and the scaling factor with a small pruning value corresponds to the convolution kernel.
  • the selection of the scaling factor depends on a global threshold we set for all layers of the entire network, which is defined as a ratio of all scaling factor values. For example, we will cut 70% of the channels in the entire network, then we first set the scaling factor Sort by absolute value, and then take 70% of the scaling factors sorted from small to large as the threshold.
  • the existing technology seems to be universal, but has an obvious defect, that is, it does not take into account the specificity of the network structure, the difference of different layers of the network and the interpretability of the method.
  • different network structures are different, and their sensitivity to parameters is also different, so all networks cannot be treated equally.
  • the network is a hierarchical learning process from front to back, and the sensitivity of parameters in this process is also changing, and the front and rear parameters of the network cannot be treated equally.
  • interpretability of the method which current methods cannot give convincing explanations.
  • the technical problem solved by the present invention is: how to provide a general and interpretable neural network pruning method.
  • a method for pruning a neural network includes:
  • the target neural network includes several residual layers
  • test sample Use the test sample set to evaluate the importance of each residual layer layer by layer, and obtain the importance degree value of each residual layer;
  • the importance of each residual layer is evaluated layer by layer using the test sample set, and the method for obtaining the importance level value of each residual layer includes:
  • test sample set to test the performance index value of the target neural network before and after the convolution kernel of each residual layer is replaced to zero, and calculate the change value of the performance index of the target neural network;
  • the importance value of the residual layer is the first degree value; when the change value of the performance index is greater than or equal to the preset change value, the importance of the residual layer is The degree value is a second degree value, wherein the first degree value is greater than the threshold value and the second degree value is less than the threshold value.
  • the performance index value includes any one or more of precision rate, precision rate, recall rate, Dice value, and Iou value.
  • the application also discloses a neural network pruning device, the pruning device includes:
  • the evaluation unit is used to evaluate the importance of each residual layer by using the test sample set, and obtain the importance degree value of each residual layer;
  • the pruning unit is used to remove the residual layer whose importance value is less than the threshold to obtain the pruned target neural network
  • the training unit is used to train the pruned target neural network by using the training sample set to optimize the parameters of the pruned target neural network.
  • the evaluation unit is specifically used for:
  • test sample set to test the performance index value of the target neural network before and after the convolution kernel of the residual layer is replaced by zero, and calculate the change value of the performance index of the target neural network;
  • the evaluation unit when the performance index change value is less than the preset change value, the evaluation unit is further configured to set the importance degree value of the residual layer to the first degree value; when the performance index change value is greater than or equal to the preset change value When the value is , the evaluation unit is further configured to set the importance degree value of the residual layer to a second degree value, wherein the first degree value is greater than the threshold value, and the second degree value is smaller than the threshold value.
  • the performance index value includes any one or more of precision rate, precision rate, recall rate, Dice value, and Iou value.
  • the present application also discloses a computer-readable storage medium, where the computer-readable storage medium stores a neural network pruning program, and when the neural network pruning program is executed by a processor, implements the above-mentioned neural network pruning method.
  • the present application also discloses a computer device, the computer device comprising a computer-readable storage medium, a processor, and a neural network pruning program stored in the computer-readable storage medium, the neural network pruning program When executed by the processor, the above-mentioned pruning method of the neural network is realized.
  • the invention discloses a neural network pruning method, which has the following technical effects compared with the traditional pruning method:
  • FIG. 1 is a flowchart of a method for pruning a neural network according to Embodiment 1 of the present invention
  • FIG. 2 is a schematic diagram of detailed steps of a method for pruning a neural network according to Embodiment 1 of the present invention
  • FIG. 3 is a schematic diagram of a partial structure of a target neural network according to Embodiment 1 of the present invention.
  • FIG. 4 is a schematic diagram of a neural network pruning device according to Embodiment 2 of the present invention.
  • FIG. 5 is a schematic diagram of a computer device according to Embodiment 4 of the present invention.
  • the existing neural network pruning algorithm does not consider the specificity of the network structure and the differences of different network layers, resulting in the algorithm not having generality and interpretability
  • this scheme first uses the test sample set to evaluate each residual layer of the target neural network after training, and obtains the importance level value of each residual layer, and then sets the importance level value less than The residual layer of the threshold is removed, and other parameters are kept unchanged to obtain the pruned target neural network, which can test the reusability of the parameters of each layer of the network layer by layer. The importance of each parameter improves the generality and interpretability of the pruning method.
  • the neural network pruning method of the first embodiment includes the following steps:
  • Step S10 Obtain a trained target neural network, where the target neural network includes several residual layers.
  • Step S20 Use the test sample set to evaluate the importance of each residual layer layer by layer, and obtain the importance degree value of each residual layer.
  • Step S30 Remove the residual layer whose importance value is less than the threshold to obtain the pruned target neural network.
  • Step S40 Use the training sample set to train the pruned target neural network to optimize the parameters of the pruned target neural network.
  • the target neural network is a ResNet network as an example.
  • the network structure of ResNet includes a short connection structure and several residual layers.
  • the short connection structure is mainly responsible for information storage, and the residual layer is mainly responsible for fine-tuning the learned information. Because all the learned information is stored in the short connection, the residual layer responsible for fine-tuning the information is relatively unimportant relative to the information storage structure, so a part of the residual layer can be deleted.
  • the short-connection feature decomposition layer is responsible for decomposing and combining the features learned above, and is the most important part of a layer block (continuous layers with the same number of features are called a layer block).
  • the short-connection feature decomposition layer is responsible for decomposing and combining the information learned from the previous layer block, and more importantly, storing the learned information.
  • Step S10 uses the training set to train the target neural network to obtain the trained target neural network.
  • step S20 the importance of each residual layer is evaluated layer by layer using the test sample set, and obtaining the importance level value of each residual layer includes the following steps:
  • Step S21 Replace the convolution kernel of the residual layer with zero layer by layer
  • Step S22 use the test sample set to test the performance index value of the target neural network after the convolution kernel of each residual layer is replaced by zero, and calculate the change value of the performance index of the target neural network;
  • Step S23 Determine the importance level value of the residual layer according to the change value of the performance index.
  • step S20 mainly applies the zero-replacement convolution kernel operation.
  • ResNet residual units have various forms and can be expressed as follows:
  • x l+1 ⁇ (x l +BN( ⁇ (BN(x l *w' l ))*w′′ l ))
  • x l , x l+1 represent the input and output feature maps of the lth and l+1st residual units, respectively;
  • x' l+1 , x" l+1 , x"' l+1 represent the output feature map of the l+1th residual unit performing zero-replacement in different ways;
  • w' l , w" l respectively represent the first convolution kernel and the second convolution kernel of the lth residual unit
  • BN Batch normalization
  • ⁇ activation function
  • ⁇ ', ⁇ ", and ⁇ "' represent the biases of different BN layers of the lth residual unit, respectively.
  • the importance degree of the residual layer is the first degree value; when the change value of the performance index is greater than or equal to the preset change value, the importance degree of the residual layer is The value is a second degree value, wherein the first degree value is greater than the threshold value and the second degree value is less than the threshold value.
  • the performance index value is defined according to the task. For example, for the classification task, it can be the precision rate, accuracy rate, and recall rate. For the segmentation task, it can be the Dice value and the Iou value (intersection over union, intersection ratio). Exemplarily, the performance index value is selected as the accuracy rate. When the accuracy rate change value is less than 5%, the importance of the residual layer is low and can be deleted. When the accuracy rate change value is greater than or equal to 5%, Then the residual layer has a high degree of importance and cannot be deleted. The above step S20 is repeated until the importance level value of each residual layer is determined.
  • step S30 the residual layer whose importance degree value is the second degree value is removed to obtain the pruned target neural network.
  • the target neural network needs to be further trained using the training sample set to optimize the parameters of the target neural network.
  • the neural network pruning method provided in the first embodiment uses the test sample set to evaluate each residual layer of the target neural network after training, and obtains the importance value of each residual layer.
  • the residual layer whose importance value is less than the threshold is removed, and other parameters are kept unchanged to obtain the pruned target neural network, so that the reusability of the parameters of each layer of the network can be tested layer by layer, and the performance of the results on the test set is intuitive.
  • the importance of each parameter of each layer is given, which improves the generality and interpretability of the pruning method.
  • the second embodiment provides a neural network pruning device, where the pruning device includes an acquisition unit 100 , an evaluation unit 200 , a pruning unit 300 and a training unit 400 .
  • the obtaining unit 100 is used to obtain the trained target neural network, and the target neural network includes several residual layers;
  • the evaluating unit 200 is used to evaluate the importance of each residual layer by using the test sample set, and obtain each layer The importance level value of the residual layer;
  • the pruning unit 300 is used to remove the residual layer whose importance level value is less than the threshold to obtain the pruned target neural network;
  • the training unit 400 is used to train the pruned neural network using the training sample set target neural network to optimize the parameters of the pruned target neural network.
  • the evaluation unit 200 is specifically configured to: replace the convolution kernel of the residual layer with zero; use the test sample set to test the performance index value of the target neural network before and after the convolution kernel of the residual layer is replaced with zero. , and calculate the change value of the performance index of the target neural network; determine the importance degree value of the residual layer according to the change value of the performance index.
  • the evaluating unit 200 is further configured to set the importance degree value of the residual layer to the first degree value; when the change value of the performance index is greater than or equal to the preset change value, The evaluation unit 200 is further configured to set the importance degree value of the residual layer to a second degree value, wherein the first degree value is greater than the threshold value, and the second degree value is smaller than the threshold value.
  • Embodiment 3 of the present application further discloses a computer-readable storage medium, where the computer-readable storage medium stores a pruning program of a neural network, and when the pruning program of the neural network is executed by a processor, realizes the above-mentioned neural network The pruning method of the network.
  • the terminal includes a processor 12 , an internal bus 13 , a network interface 14 , and a computer-readable storage medium 11 .
  • the processor 12 reads the corresponding computer program from the computer-readable storage medium and then executes it, forming a request processing device on a logical level.
  • the computer-readable storage medium 11 stores a neural network pruning program, and when the neural network pruning program is executed by a processor, implements the above-mentioned neural network pruning method.
  • Computer-readable storage media includes both persistent and non-permanent, removable and non-removable media, and storage of information can be implemented by any method or technology.
  • Information may be computer readable instructions, data structures, modules of programs, or other data.
  • Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash Memory or other memory technology, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage , magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by computing devices.
  • PRAM phase-change memory
  • SRAM static random access memory
  • DRAM dynamic random access memory
  • RAM random access memory
  • ROM read-only memory Memory

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Feedback Control In General (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

一种神经网络的剪枝方法及其剪枝装置、存储介质，其中剪枝方法包括：获取训练完成的目标神经网络，所述目标神经网络包括若干残差层（S10）；利用测试样本集逐层评估每一层残差层的重要性，获得每一层残差层的重要性程度值（S20）；将重要性程度值小于阈值的残差层去除，获得剪枝后的目标神经网络（S30）；利用训练样本集训练剪枝后的目标神经网络，以优化剪枝后的目标神经网络的参数（S40)。该方法可以逐层测试网络各层参数的重要性，通过在测试集上结果的表现直观地给出了各层各参数的重要性，提高了剪枝方法的通用性和可解释性。

Description

神经网络的剪枝方法及其剪枝装置、存储介质 技术领域
本发明属于机器学习技术领域，具体地讲，涉及一种神经网络的剪枝方法及其剪枝装置、计算机可读存储介质。
背景技术
深度学习目前广泛应用于各行，但是在许多现实应用中，深度学习网络的部署是一个很大的难题。一方面深度学习网络参数庞大，对内存的需求很高，另一方面深度学习需要大量计算。尤其是常用的卷积神经网络，其存在大量冗余参数，往往需要进行网络剪枝。网络剪枝的基本思想就是利用不同参数的重要性不同，裁剪掉那些冗余参数，使网络更精简，减少存储占用和计算量。针对参数重要性剪枝思路是建立一个评估标准，然后按照评估标准对网络的参数进行排序，设定阈值，然后剪除小于阈值的那些网络参数。
例如，文章(Li,Kadav，Durdanovic，Samet & Graf，2016)中提出思路是：
1)计算卷积核中所有权值的绝对值之和；2)根据求和大小排列卷积核；3)删除数值较小的卷积核；4)对删除之后的卷积核重新组合，生成新的卷积核矩阵。
另一文章(Liu et al.2017)指出，通过BN层的缩放因子进行网络剪枝，裁剪值小的缩放因子对应卷积核。缩放因子的选取取决于我们为整个网络所有层设置的一个全局阈值，它被定义为所有缩放因子值的一个比例，比如我们将剪掉整个网络中70％的通道，那么我们先对缩放因子的绝对值排个序，然后取从小到大排序的缩放因子中70％的缩放因子为阈值。
现有技术的看似具有通用性，但是具有一个明显的缺陷，那就是没有考虑到网络结构的特异性，网络不同层的差异性和方法的可解释性。首先不同网络结构是不同的，而他们对参数的敏感性也不同，不可以所有网络同等对待。网 络从前到后是一个有层次的学习过程，而在此过程参数的敏感度也在变化，网络的前后参数不可同等对待。最后是方法的可解释性，目前的方法无法给出令人信服的解释。
发明内容
(一)本发明所要解决的技术问题
本发明解决的技术问题是：如何提供一种具有通用性和可解释性的神经网络的剪枝方法。
(二)本发明所采用的技术方案
一种神经网络的剪枝方法，所述剪枝方法包括：
获取训练完成的目标神经网络，所述目标神经网络包括若干残差层；
利用测试样本集逐层评估每一层残差层的重要性，获得每一层残差层的重要性程度值；
将重要性程度值小于阈值的残差层去除，获得剪枝后的目标神经网络；
利用训练样本集训练剪枝后的目标神经网络，以优化剪枝后的目标神经网络的参数。
可选择地，利用测试样本集逐层评估每一层残差层的重要性，获得每一层残差层的重要性程度值的方法包括：
逐层将残差层的卷积核替换为零；
利用测试样本集测试各个残差层的卷积核替换为零之前和之后的目标神经网络的性能指标值，并计算得到目标神经网络的性能指标变化值；
根据性能指标变化值确定残差层的重要性程度值。
可选择地，当性能指标变化值小于预设变化值时，残差层的重要性程度值为第一程度值；当性能指标变化值大于或等于预设变化值时，残差层的重要性程度值为第二程度值，其中所述第一程度值大于所述阈值，所述第二程度值小于所述阈值。
可选择地，所述性能指标值包括精确率、准确率、召回率、Dice值、Iou 值中的任意一种或多种。
本申请还公开了一种神经网络的剪枝装置，所述剪枝装置包括：
获取单元，用于获取训练完成的目标神经网络，所述目标神经网络包括若干残差层；
评估单元，用于利用测试样本集评估每一层残差层的重要性，获得每一层残差层的重要性程度值；
剪枝单元，用于将重要性程度值小于阈值的残差层去除，获得剪枝后的目标神经网络；
训练单元，用于利用训练样本集训练剪枝后的目标神经网络，以优化剪枝后的目标神经网络的参数。
所述评估单元具体用于：
将残差层的卷积核替换为零；
利用测试样本集测试残差层的卷积核替换为零之前和之后的目标神经网络的性能指标值，并计算得到目标神经网络的性能指标变化值；
根据性能指标变化值确定残差层的重要性程度值。
可选择地，当性能指标变化值小于预设变化值时，所述评估单元还用于将残差层的重要性程度值设置为第一程度值；当性能指标变化值大于或等于预设变化值时，所述评估单元还用于将残差层的重要性程度值为第二程度值，其中所述第一程度值大于所述阈值，所述第二程度值小于所述阈值。
可选择地，所述性能指标值包括精确率、准确率、召回率、Dice值、Iou值中的任意一种或多种。
本申请还公开了一种计算机可读存储介质，所述计算机可读存储介质存储有神经网络的剪枝程序，所述神经网络的剪枝程序被处理器执行时实现上述的神经网络的剪枝方法。
本申请还公开了一种计算机设备，所述计算机设备包括计算机可读存储介质、处理器和存储在所述计算机可读存储介质中的神经网络的剪枝程序，所述神经网络的剪枝程序被处理器执行时实现上述的神经网络的剪枝方法。
(三)有益效果
本发明公开了一种神经网络的剪枝方法，相对于传统的剪枝方法，具有如下技术效果：
利用测试样本集分别对训练完成的目标神经网络的每一层残差层进行评估，获得每一层残差层的重要性程度值，接着将重要性程度值小于阈值的残差层去除，保持其他参数不变，获得剪枝后的目标神经网络，这样可以逐层测试网络各层参数可重用性，通过在测试集上结果的表现直观地给出了各层各参数的重要性，提高了剪枝方法的通用性和可解释性。
附图说明
图1为本发明的实施例一的神经网络的剪枝方法的流程图；
图2为本发明的实施例一的神经网络的剪枝方法的详细步骤示意图；
图3为本发明的实施例一的目标神经网络的局部结构示意图；
图4为本发明的实施例二的神经网络的剪枝装置的示意图；
图5为本发明的实施例四的计算机设备示意图。
具体实施方式
为了使本发明的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本发明进一步详细说明。应当理解，此处所描述的具体实施例仅仅用以解释本发明，并不用于限定本发明。
在详细描述本申请的各个实施例之前，首先简单描述本申请的发明构思：现有的神经网络剪枝算法由于没有考虑网络结构的特异性以及不同网络层的差异性，导致算法不具有通用性和可解释性，本方案首先利用测试样本集分别对训练完成的目标神经网络的每一层残差层进行评估，获得每一层残差层的重要性程度值，接着将重要性程度值小于阈值的残差层去除，保持其他参数不变，获得剪枝后的目标神经网络，这样可以逐层测试网络各层参数可重用性，通过在测试集上结果的表现直观地给出了各层各参数的重要性，提高了剪枝方法的通用性和可解释性。
如图1和图2所示，本实施例一的神经网络的剪枝方法包括如下步骤：
步骤S10：获取训练完成的目标神经网络，所述目标神经网络包括若干残差层。
步骤S20：利用测试样本集逐层评估每一层残差层的重要性，获得每一层残差层的重要性程度值。
步骤S30：将重要性程度值小于阈值的残差层去除，获得剪枝后的目标神经网络。
步骤S40：利用训练样本集训练剪枝后的目标神经网络，以优化剪枝后的目标神经网络的参数。
作为优选实施例，在步骤S10中，如图3所示，目标神经网络以ResNet网络为例。
ResNet的网络结构包括短连接结构和若干残差层，其中短连接结构主要负责信息存储，残差层主要负责微调学习到的信息。因为短连接中存储了学习到的全部信息，所以相对信息存储结构，负责微调信息的残差层就相对不是很重要，因此残差层可以删除一部分。其中短连接结构和残差层中有两个特殊的层：短连接特征分解层；参差特征分解层。短连接特征分解层负责对上面学习到的特征进行分解和组合，是一个层块(具有相同特征数的连续层称为一个层块)中最重要的部分。短连接特征分解层负责分解和组合上一个层块学到的信息，更重要的是存储学到的信息。步骤S10利用训练集训练目标神经网络，得到训练完成的目标神经网络。
在步骤S20中，利用测试样本集逐层评估每一层残差层的重要性，获得每一层残差层的重要性程度值包括如下步骤：
步骤S21：逐层将残差层的卷积核替换为零；
步骤S22：利用测试样本集测试各残差层的卷积核替换为零之后的目标神经网络的性能指标值，并计算得到目标神经网络的性能指标变化值；
步骤S23：根据性能指标变化值确定残差层的重要性程度值。
需要说明的是逐层是指每层进行测试时，仅将该层的卷积核替换为零，其他层的卷积核保持不变。具体来说，步骤S20主要应用了零替换卷积核操作。ResNet残差单元具有多种形式，可以表示为如下形式：
x l+1＝σ(x l+BN(σ(BN(x l*w' l))*w″ l))
Figure PCTCN2021078455-appb-000001
用零分别替换各卷积核后可以表示为：
x' l+1＝σ(x l+BN(σ(BN(x l*0))*w″ l))＝σ(x l+BN(σ(β')*w″ l))
x″ l+1＝σ(x l+BN(σ(BN(x l*w' l))*0))＝σ(x l+β″)
Figure PCTCN2021078455-appb-000002
Figure PCTCN2021078455-appb-000003
x″′ l+1＝σ(BN(x l*0)+BN(σ(BN(x l*w' l))*w″ l))＝σ(β″′)+BN(σ(BN(x l*w' l))*w″ l))
x l,x l+1分别表示第l，l+1个残差单元的输入输出特征图；
x' l+1,x” l+1,x”’ l+1表示第l+1个残差单元分别以不同方式进行零替换的输出特征图；
w' l,w” l分别表示第l个残差单元的第一个卷积核和第二个卷积核；
*表示卷积操作；BN(Batch normalization)表示批标准化；σ表示激活函数；
β',β”,β”'分别表示第l个残差单元不同BN层的偏置。
进一步地，当性能指标变化值小于预设变化值时，残差层的重要性程度值为第一程度值；当性能指标变化值大于或等于预设变化值时，残差层的重要性程度值为第二程度值，其中所述第一程度值大于所述阈值，所述第二程度值小于所述阈值。其中性能指标值根据任务定义，如针对分类任务可以是精确率、准确率、召回率，针对分割任务可以是Dice值、Iou值(intersection over union，交并比值)。示例性地，性能指标值选为准确率，当准确率变化值小于5％时，则该残差层的重要性程度较低，可以被删除，当准确率变化值大于或等于5％时，则该残差层的重要性程度较高，不可以被删除。重复上述步骤S20，直至确定每一层残差层的重要性程度值。
在步骤S30中，将重要性程度值为第二程度值的残差层去除，获得剪枝后 的目标神经网络。剪枝完成后，需要利用训练样本集进一步训练目标神经网络，以优化目标神经网络的参数。
本实施例一提供的神经网络的剪枝方法，利用测试样本集分别对训练完成的目标神经网络的每一层残差层进行评估，获得每一层残差层的重要性程度值，接着将重要性程度值小于阈值的残差层去除，保持其他参数不变，获得剪枝后的目标神经网络，这样可以逐层测试网络各层参数可重用性，通过在测试集上结果的表现直观地给出了各层各参数的重要性，提高了剪枝方法的通用性和可解释性。
进一步地，如图4所示，本实施例二提供了一种神经网络的剪枝装置，所述剪枝装置包括获取单元100、评估单元200、剪枝单元300和训练单元400。其中，获取单元100用于获取训练完成的目标神经网络，所述目标神经网络包括若干残差层；评估单元200用于利用测试样本集评估每一层残差层的重要性，获得每一层残差层的重要性程度值；剪枝单元300用于将重要性程度值小于阈值的残差层去除，获得剪枝后的目标神经网络；训练单元400用于利用训练样本集训练剪枝后的目标神经网络，以优化剪枝后的目标神经网络的参数。
进一步地，所述评估单元200具体用于：将残差层的卷积核替换为零；利用测试样本集测试残差层的卷积核替换为零之前和之后的目标神经网络的性能指标值，并计算得到目标神经网络的性能指标变化值；根据性能指标变化值确定残差层的重要性程度值。
当性能指标变化值小于预设变化值时，所述评估单元200还用于将残差层的重要性程度值设置为第一程度值；当性能指标变化值大于或等于预设变化值时，所述评估单元200还用于将残差层的重要性程度值为第二程度值，其中所述第一程度值大于所述阈值，所述第二程度值小于所述阈值。评估单元200的具体评估过程参考实施例一中关于步骤S20的描述，在此不进行赘述。
本申请的实施例三还公开了一种计算机可读存储介质，所述计算机可读存储介质存储有神经网络的剪枝程序，所述神经网络的剪枝程序被处理器执行时实现上述的神经网络的剪枝方法。
本申请的另一实施例还公开了一种计算机设备，在硬件层面，如图5所示，该终端包括处理器12、内部总线13、网络接口14、计算机可读存储介质11。处理器12从计算机可读存储介质中读取对应的计算机程序然后运行，在逻辑 层面上形成请求处理装置。当然，除了软件实现方式之外，本说明书一个或多个实施例并不排除其他实现方式，比如逻辑器件抑或软硬件结合的方式等等，也就是说以下处理流程的执行主体并不限定于各个逻辑单元，也可以是硬件或逻辑器件。所述计算机可读存储介质11上存储有神经网络的剪枝程序，所述神经网络的剪枝程序被处理器执行时实现上述的神经网络的剪枝方法。
计算机可读存储介质包括永久性和非永久性、可移动和非可移动媒体可以由任何方法或技术来实现信息存储。信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机可读存储介质的例子包括，但不限于相变内存(PRAM)、静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、其他类型的随机存取存储器(RAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字多功能光盘(DVD)或其他光学存储、磁盒式磁带、磁盘存储、量子存储器、基于石墨烯的存储介质或其他磁性存储设备或任何其他非传输介质，可用于存储可以被计算设备访问的信息。
上面对本发明的具体实施方式进行了详细描述，虽然已表示和描述了一些实施例，但本领域技术人员应该理解，在不脱离由权利要求及其等同物限定其范围的本发明的原理和精神的情况下，可以对这些实施例进行修改和完善，这些修改和完善也应在本发明的保护范围内。

Claims (12)

  1. 一种神经网络的剪枝方法，其中，所述剪枝方法包括：
    获取训练完成的目标神经网络，所述目标神经网络包括若干残差层；
    利用测试样本集逐层评估每一层残差层的重要性，获得每一层残差层的重要性程度值；
    将重要性程度值小于阈值的残差层去除，获得剪枝后的目标神经网络；
    利用训练样本集训练剪枝后的目标神经网络，以优化剪枝后的目标神经网络的参数。
  2. 根据权利要求1所述的神经网络的剪枝方法，其中，利用测试样本集逐层评估每一层残差层的重要性，获得每一层残差层的重要性程度值的方法包括：
    逐层将残差层的卷积核替换为零；
    利用测试样本集测试各个残差层的卷积核替换为零之前和之后的目标神经网络的性能指标值，并计算得到目标神经网络的性能指标变化值；
    根据性能指标变化值确定残差层的重要性程度值。
  3. 根据权利要求2所述的神经网络的剪枝方法，其中，当性能指标变化值小于预设变化值时，残差层的重要性程度值为第一程度值；当性能指标变化值大于或等于预设变化值时，残差层的重要性程度值为第二程度值，其中所述第一程度值大于所述阈值，所述第二程度值小于所述阈值。
  4. 根据权利要求2所述的神经网络的剪枝方法，其中，所述性能指标值包括精确率、准确率、召回率、Dice值、Iou值中的任意一种或多种。
  5. 一种神经网络的剪枝装置，其中，所述剪枝装置包括：
    获取单元，用于获取训练完成的目标神经网络，所述目标神经网络包括若干残差层；
    评估单元，用于利用测试样本集评估每一层残差层的重要性，获得每一层残差层的重要性程度值；
    剪枝单元，用于将重要性程度值小于阈值的残差层去除，获得剪枝后的目 标神经网络；
    训练单元，用于利用训练样本集训练剪枝后的目标神经网络，以优化剪枝后的目标神经网络的参数。
  6. 根据权利要求5所述的神经网络的剪枝装置，其中，所述评估单元具体用于：
    逐层将残差层的卷积核替换为零；
    利用测试样本集测试各个残差层的卷积核替换为零之前和之后的目标神经网络的性能指标值，并计算得到目标神经网络的性能指标变化值；
    根据性能指标变化值确定残差层的重要性程度值。
  7. 根据权利要求6所述的神经网络的剪枝装置，其中，当性能指标变化值小于预设变化值时，所述评估单元还用于将残差层的重要性程度值设置为第一程度值；当性能指标变化值大于或等于预设变化值时，所述评估单元还用于将残差层的重要性程度值为第二程度值，其中所述第一程度值大于所述阈值，所述第二程度值小于所述阈值。
  8. 根据权利要求6所述的神经网络的剪枝装置，其中，所述性能指标值包括精确率、准确率、召回率、Dice、Iou值中的任意一种或多种。
  9. 一种计算机可读存储介质，其中，所述计算机可读存储介质存储有神经网络的剪枝程序，所述神经网络的剪枝程序被处理器执行时实现权利要求1所述的神经网络的剪枝方法。
  10. 根据权利要求6所述的计算机可读存储介质，其中，利用测试样本集逐层评估每一层残差层的重要性，获得每一层残差层的重要性程度值的方法包括：
    逐层将残差层的卷积核替换为零；
    利用测试样本集测试各个残差层的卷积核替换为零之前和之后的目标神经网络的性能指标值，并计算得到目标神经网络的性能指标变化值；
    根据性能指标变化值确定残差层的重要性程度值。
  11. 根据权利要求10所述的计算机可读存储介质，其中，当性能指标变化值小于预设变化值时，残差层的重要性程度值为第一程度值；当性能指标变 化值大于或等于预设变化值时，残差层的重要性程度值为第二程度值，其中所述第一程度值大于所述阈值，所述第二程度值小于所述阈值。
  12. 根据权利要求2所述的计算机可读存储介质，其中，所述性能指标值包括精确率、准确率、召回率、Dice值、Iou值中的任意一种或多种。
PCT/CN2021/078455 2021-02-26 2021-03-01 神经网络的剪枝方法及其剪枝装置、存储介质 Ceased WO2022178908A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110216506.6A CN114970851A (zh) 2021-02-26 2021-02-26 神经网络的剪枝方法及其剪枝装置、存储介质、设备
CN202110216506.6 2021-02-26

Publications (1)

Publication Number Publication Date
WO2022178908A1 true WO2022178908A1 (zh) 2022-09-01

Family

ID=82972722

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/078455 Ceased WO2022178908A1 (zh) 2021-02-26 2021-03-01 神经网络的剪枝方法及其剪枝装置、存储介质

Country Status (2)

Country Link
CN (1) CN114970851A (zh)
WO (1) WO2022178908A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116227575A (zh) * 2023-02-27 2023-06-06 上海瑾盛通信科技有限公司 模型剪枝方法、装置、电子设备及存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190244103A1 (en) * 2018-02-07 2019-08-08 Royal Bank Of Canada Robust pruned neural networks via adversarial training
CN110689113A (zh) * 2019-09-19 2020-01-14 浙江大学 一种基于大脑共识主动性的深度神经网络压缩方法
CN111008640A (zh) * 2019-10-17 2020-04-14 平安科技（深圳）有限公司 图像识别模型训练及图像识别方法、装置、终端及介质
CN111291637A (zh) * 2020-01-19 2020-06-16 中国科学院上海微系统与信息技术研究所 一种基于卷积神经网络的人脸检测方法、装置及设备
CN112052951A (zh) * 2020-08-31 2020-12-08 北京中科慧眼科技有限公司 一种剪枝神经网络方法、系统、设备及可读存储介质
CN112149797A (zh) * 2020-08-18 2020-12-29 Oppo（重庆）智能科技有限公司 神经网络结构优化方法和装置、电子设备

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190244103A1 (en) * 2018-02-07 2019-08-08 Royal Bank Of Canada Robust pruned neural networks via adversarial training
CN110689113A (zh) * 2019-09-19 2020-01-14 浙江大学 一种基于大脑共识主动性的深度神经网络压缩方法
CN111008640A (zh) * 2019-10-17 2020-04-14 平安科技（深圳）有限公司 图像识别模型训练及图像识别方法、装置、终端及介质
CN111291637A (zh) * 2020-01-19 2020-06-16 中国科学院上海微系统与信息技术研究所 一种基于卷积神经网络的人脸检测方法、装置及设备
CN112149797A (zh) * 2020-08-18 2020-12-29 Oppo（重庆）智能科技有限公司 神经网络结构优化方法和装置、电子设备
CN112052951A (zh) * 2020-08-31 2020-12-08 北京中科慧眼科技有限公司 一种剪枝神经网络方法、系统、设备及可读存储介质

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116227575A (zh) * 2023-02-27 2023-06-06 上海瑾盛通信科技有限公司 模型剪枝方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN114970851A (zh) 2022-08-30

Similar Documents

Publication Publication Date Title
CN109598287B (zh) 基于深度卷积生成对抗网络样本生成的外观瑕疵检测方法
WO2022120901A1 (zh) 基于特征金字塔的图像检测模型的训练方法、介质和设备
US12106466B2 (en) Photographing condition determining method for metal structure, photographing method for metal structure, phase classification method for metal structure, photographing condition determining device for metal structure, photographing device for metal structure, phase classification device for metal structure, material property estimating method for metal material, and material property estimating device for metal material
CN112381791A (zh) 一种基于3d点云的螺栓松动检测方法
CN118193955B (zh) 一种压气机气动噪声获取方法、装置、介质及产品
WO2022166294A1 (zh) 一种目标检测方法和装置
CN116109627B (zh) 基于迁移学习和小样本学习缺陷检测方法、装置及介质
WO2022166293A1 (zh) 一种目标检测方法和装置
CN109726195A (zh) 一种数据增强方法及装置
CN118397001B (zh) 一种基于图像处理的铌钛毛细管质量检测方法及系统
CN115293223A (zh) 深度学习中相位解缠的细节损失优化方法及系统、存储介质
WO2022178908A1 (zh) 神经网络的剪枝方法及其剪枝装置、存储介质
CN113362437A (zh) 一种点云重采样方法、系统、存储介质及设备
CN111127439A (zh) 车辆轮胎的胎纹检测方法、装置、电子设备及存储介质
CN117809060A (zh) 一种商标侵权检测方法和系统
US12057336B2 (en) Estimating heights of defects in a wafer by scaling a 3D model using an artificial neural network
CN118097320B (zh) 一种双分支的晶圆sem缺陷图分类和分割方法、系统
CN115479904B (zh) 一种饲料单宁快速检测方法及系统
CN118537497A (zh) 一种大规模点云的加权泊松盘重采样方法、设备及介质
CN117635521A (zh) 基于改进yolox的金属表面缺陷检测方法及系统
CN116737681A (zh) 一种实时异常日志检测方法、装置、计算机设备和存储介质
JP2024540704A (ja) カーボンブラックの三次元分析のためのシステム、デバイス及び方法
CN120339549A (zh) 基于网格对象的模型分割方法及设备
CN118967847B (zh) 一种三维网格模型压缩方法、设备、存储介质及产品
CN120876660B (zh) 基于改进gan的图像生成模型构建、应用方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21927343

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21927343

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 11/01/2024)

122 Ep: pct application non-entry in european phase

Ref document number: 21927343

Country of ref document: EP

Kind code of ref document: A1