WO2019196223A1 - 一种应用于卷积神经网络的加速方法和加速器 - Google Patents

一种应用于卷积神经网络的加速方法和加速器 Download PDF

Info

Publication number
WO2019196223A1
WO2019196223A1 PCT/CN2018/095365 CN2018095365W WO2019196223A1 WO 2019196223 A1 WO2019196223 A1 WO 2019196223A1 CN 2018095365 W CN2018095365 W CN 2018095365W WO 2019196223 A1 WO2019196223 A1 WO 2019196223A1
Authority
WO
WIPO (PCT)
Prior art keywords
density
feature maps
sparse
neural network
preset threshold
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2018/095365
Other languages
English (en)
French (fr)
Inventor
刘勇攀
袁哲
岳金山
杨华中
李学清
王智博
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tsinghua University
Original Assignee
Tsinghua University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tsinghua University filed Critical Tsinghua University
Publication of WO2019196223A1 publication Critical patent/WO2019196223A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0495—Quantised networks; Sparse networks; Compressed networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means

Definitions

  • the invention belongs to the technical field of operation optimization, and more particularly to an acceleration method and an accelerator applied to a convolutional neural network.
  • the Convolutional Neural Network is a feedforward neural network whose artificial neurons can respond to surrounding units in a part of the coverage and is suitable for processing large images.
  • Convolutional neural networks are widely used in image recognition, speech recognition and other fields, but the amount of calculation is very large.
  • some methods do not perform a multiplication operation when the input data is 0, thereby reducing the operation.
  • these methods are all focused on dealing with the sparse neural network itself, assuming that the neural network is sparse.
  • the output feature maps of each layer in the convolutional neural network may be sparse and may be non-sparse.
  • the weight data of each layer of the convolutional neural network and the density of the feature map are generally distributed between 5% and 90%.
  • a sparse matrix is a matrix in which the number of elements with an index value of 0 is much larger than the number of non-zero elements, and the non-zero element distribution has no regular matrix.
  • the prior art only a sparse convolutional neural network can be processed on the one hand, and the calculation amount is large and the operation speed is low in the case where the convolutional neural network is not sparse; on the other hand, the prior art can only process the convolutional nerve.
  • the weight data or feature map is sparse and cannot handle the case where the weight data and the feature map are sparse.
  • the present invention provides an acceleration method and an accelerator applied to a convolutional neural network.
  • an acceleration method applied to a convolutional neural network comprising:
  • step S1 specifically includes:
  • the number of non-zero elements in the feature map and the total number of all elements in the feature map are counted;
  • the ratio between the number of non-zero elements in the feature map and the total number of all elements in the feature map is taken as the density of the feature map.
  • the preset threshold includes a first preset threshold and a second preset threshold; wherein the first preset threshold is smaller than the second preset threshold;
  • step S2 specifically includes:
  • each of the feature maps is not sparsely encoded.
  • the method further includes:
  • each of the convolution kernels has a density less than the first predetermined threshold, encoding each of the convolution kernels into a sparse matrix storage format
  • each of the convolution kernels is not sparsely encoded.
  • step S3 specifically includes:
  • an accelerator for a convolutional neural network comprising: a neural network computing array module and a dynamic sparse adjustment module;
  • the dynamic sparse adjustment module is configured to calculate the density of each feature image outputted by each layer of the convolutional neural network, compare the density of each of the feature maps with a plurality of preset thresholds, and compare the results according to the comparison results.
  • the feature map is subjected to sparse coding; wherein different comparison results correspond to different sparse coding modes;
  • the neural network computing array module is configured to perform a convolution operation on each of the sparsely encoded feature maps and the pre-sparsely encoded convolutional neural networks.
  • the dynamic sparse adjustment module includes an online density identification module, an output temporary registration module, a dynamic coding module, and a dynamic sparse control module;
  • the online density identification module is configured to count, for any of the feature maps, the number of 0 elements in the feature map and the total number of all elements in the feature map; The ratio between the number and the total number of all elements in the feature map as the density of the feature map;
  • the output temporary registration module is configured to store each of the feature maps output by each layer in the convolutional neural network
  • the dynamic sparse control module is configured to compare the density of each of the feature maps output by the online density identification module with a plurality of preset thresholds;
  • the dynamic encoding module is configured to perform sparse encoding on each of the feature maps in the output temporary registration module according to a comparison result.
  • the preset threshold includes a first preset threshold and a second preset threshold; wherein the first preset threshold is smaller than the second preset threshold;
  • the dynamic coding module is specifically configured to:
  • each of the feature maps is not sparsely encoded.
  • the dynamic coding module is further configured to:
  • each of the pre-computed convolution kernels has a density less than the first predetermined threshold, encoding each of the convolution kernels into a sparse matrix storage format;
  • each of the convolution kernels is not sparsely encoded.
  • the neural network computing array module is specifically configured to:
  • the present invention provides an acceleration method and an accelerator applied to a convolutional neural network, which obtains each of the features by comparing the density of each feature image outputted by each layer in the convolutional neural network with a plurality of preset thresholds.
  • the sparse state of the graph, the feature maps of different sparse states are sparsely coded in different ways, and then based on the thinned coded feature maps of the next layer of each layer and the volumes in the pre-sparse coded convolutional neural network
  • the product core performs a convolution operation to reduce the amount of calculation of the convolution operation in the convolutional neural network and to increase the operation speed.
  • FIG. 1 is a schematic overall flow chart of an acceleration method applied to a convolutional neural network according to an embodiment of the present invention
  • FIG. 2 is a schematic diagram of an overall structure of an accelerator applied to a convolutional neural network according to an embodiment of the present invention
  • FIG. 3 is a schematic diagram of a limit energy efficiency test result of an accelerator applied to a convolutional neural network according to an embodiment of the present invention
  • FIG. 4 is a schematic diagram of comparison of extreme energy efficiency test results in an accelerator applied to a convolutional neural network according to an embodiment of the present invention.
  • FIG. 1 is a schematic flowchart of an overall acceleration method applied to a convolutional neural network according to an embodiment of the present invention, where the method includes:
  • the convolutional neural network may include or not include a pooling layer.
  • the convolutional neural network is trained. After the training is completed, the convolution kernel in the convolutional neural network no longer changes. Therefore, the convolution kernel in the convolutional neural network does not need online dynamic sparse coding, and direct offline sparse coding. Just one time.
  • the line refers to the chip on the accelerator, and the line below refers to the chip not located on the accelerator.
  • the convolutional calculation is performed by directly reading the sparsely encoded convolution kernel at each convolution operation.
  • the original image data When the original image data is input, the original image data is sparsely encoded, and then the sparsely encoded raw data and the sparsely encoded convolution kernel are input to the first layer convolutional layer of the convolutional neural network for convolution calculation. Since the original image data is generally not sparse, the original image data may not be sparsely encoded, and the original image data may be directly input.
  • the sparse coding is to store the data in a sparse format.
  • the density of each characteristic map outputted by each layer in the convolutional neural network is different, the characteristic maps of the output of different layers are also dynamically changed, so the density is also dynamically changed.
  • the density indicates the degree of sparsity of each of the feature maps.
  • the density of each feature image outputted by each layer is calculated, and each characteristic image outputted by each layer is obtained according to the density of each feature image outputted by each layer. Perform sparse coding.
  • each of the feature maps after sparse coding and each convolution kernel in the pre-sparse coded convolutional neural network are used as input of a convolution layer of a layer below the layer, and a convolution operation is performed. Then, the result of the convolution operation is used as an input to the next layer of the convolution layer, and the feature map outputted by the next layer of the convolution layer is continued to perform the above-described sparse coding and convolution operations until the convolution The last layer in the neural network outputs the feature maps.
  • This embodiment is not limited to the sparse coding mode of the convolution kernel.
  • the density of each feature image outputted by each layer in the convolutional neural network is compared with a plurality of preset thresholds, and the sparse state of each of the feature maps is obtained, and the feature maps of different sparse states are sparse in different manners. Coding, and then convolving the sparse-coded feature maps and the convolution kernels in the pre-sparse-coded convolutional neural network based on the convolutional layer of the next layer of each layer to reduce the convolution operation in the convolutional neural network. Calculate the amount and increase the speed of the operation.
  • the step S1 specifically includes: counting the number of non-zero elements in the feature map and the total number of all the elements in the feature map for any of the feature maps. The ratio between the number of non-zero elements in the feature map and the total number of all elements in the feature map is taken as the density of the feature map.
  • the density of each feature map is a ratio between the number of non-zero elements in each feature map and the total number of all elements in each feature map. For example, if the number of non-zero elements in a feature map is 10, and the total number of all elements in the feature map is 100, the density of the feature map is 0.1.
  • the preset threshold includes a first preset threshold and a second preset threshold; the preset threshold includes a first preset threshold and a second preset threshold;
  • the step S2 specifically includes: if the density of each of the feature maps is less than the first preset threshold, encoding each of the feature maps into a sparse matrix storage format; if each of the feature maps has a density greater than Or equal to the first preset threshold, and less than the second preset threshold, marking 0 elements in each of the feature maps; if the density of each of the feature maps is greater than or equal to the second
  • the preset threshold is not sparsely encoded for each of the feature maps.
  • the preset threshold in this embodiment includes a first preset threshold th1 and a second preset threshold th2.
  • the feature map is divided into a completely non-sparse state D.
  • each of the feature maps is encoded into a sparse matrix storage format, where the sparse matrix storage format includes non-zero data activ and sparse index index in each of the feature maps, such as coordinate encoding and Compress sparse row coding.
  • the sparse matrix storage format includes non-zero data activ and sparse index index in each of the feature maps, such as coordinate encoding and Compress sparse row coding.
  • the 0 element in each feature map is marked with a guard, which is used to identify the 0 element. Elements that are tagged may not participate in calculations and storage, thereby reducing power consumption. Marking the 0 elements in each feature map is also a sparse coding method. If each feature map is completely non-sparse state D, dynamic coding is not required, and non-sparse data of each feature map is directly output.
  • the method further includes: calculating a density of each convolution kernel in the trained convolution network; if each of the convolution kernels has a density less than the first a preset threshold, the each convolution kernel is encoded into a sparse matrix storage format; if the density of each of the convolution kernels is greater than or equal to the first preset threshold, and less than the second preset threshold And marking 0 elements in each of the convolution kernels; if each of the convolution kernels has a density greater than or equal to the second predetermined threshold, each of the convolution kernels is not encoded.
  • the density of each convolution kernel is the ratio between the number of non-zero elements in each convolution kernel and the total number of all elements in each convolution kernel.
  • the state WS of each convolution kernel is divided into three states as in the feature map. Each state corresponds to a different sparse coding scheme. Since there are three states in the feature map and the convolution kernel, there are 9 states after the combination, so that the density of the convolutional neural network is more finely divided.
  • the step S3 in the embodiment specifically includes: when each of the feature maps or each of the convolution kernels has the mark, each of the feature maps or each The elements corresponding to the tags in the convolution kernel are not evaluated.
  • FIG. 2 is a schematic diagram of an overall structure of an accelerator applied to a convolutional neural network according to an embodiment of the present invention, including: a neural network computing array module and a dynamic sparse adjustment module, wherein the dynamic sparse adjustment module is configured to calculate a density of each feature image outputted by each layer in the convolutional neural network, and compare the density of each of the feature maps with a plurality of preset thresholds, Performing sparse coding on each of the feature maps according to the comparison result; wherein different comparison results correspond to different sparse coding modes; the neural network calculation array module is configured to perform sparsely coded each of the feature maps and pre-sparse coding Each convolution kernel in the convolutional neural network performs a convolution operation.
  • the convolutional neural network may include or not include a pooling layer. Firstly, the convolutional neural network is trained. After the training is completed, the convolution kernel in the convolutional neural network no longer changes. Therefore, the convolution kernel in the convolutional neural network does not need online dynamic sparse coding, and the direct offline sparse coding once can.
  • the neural network computational array module directly reads the offline sparsely encoded convolution kernel for convolution calculations each time a convolution operation is performed.
  • the convolutional neural network inputs the original image data
  • the dynamic sparse adjustment module sparsely encodes the original image data
  • the neural network calculation array module performs convolution calculation according to the sparsely encoded original data and the sparsely encoded convolution kernel. Since the original image data is generally not sparse, the original image data may not be sparsely encoded, and the original image data may be directly input.
  • the sparse coding is to store the data in a sparse format.
  • the dynamic sparse adjustment module calculates the density of each feature map outputted by each layer, according to the density of each feature map output by each layer.
  • Each feature map output by the layer is sparsely encoded.
  • the dynamic sparse adjustment module acquires a sparse state of each of the feature maps output by the layer according to a plurality of preset thresholds. Therefore, the feature maps of different sparse states are subjected to different forms of sparse coding, and are not limited to one type of sparse coding. In the prior art, all the special maps output by each layer are sparsely encoded, and the calculation amount is large.
  • the neural network computing array module performs a convolution operation according to each of the feature maps after sparse coding and each convolution kernel in the pre-sparse coded convolutional neural network. If the pooling module is included, the pooling module performs a pooling operation on the result of the convolution operation.
  • the accelerator further includes an intermediate data memory module, a main chip controller, and an on-chip data exchange module. Wherein, the main controller controls the running action and timing of the entire chip of the accelerator.
  • the on-chip data exchange module is used to read data from the chip external memory or write the chip-calculated data to external storage.
  • the chip reads the original image data and the initial convolution kernel from the external memory through the chip upper and lower data exchange modules under the control of the main controller.
  • the intermediate data storage module is configured to store intermediate results in the calculation process of the neural network computing array module.
  • the dynamic sparse adjustment module compares the density of each feature image outputted by each layer of the convolutional neural network with a plurality of preset thresholds, obtains a sparse state of each of the feature maps, and differentizes the feature maps of different sparse states.
  • Sparse coding of the method for the neural network computing array module to perform convolution operations on the sparsely encoded feature maps and the convolution kernels in the pre-sparse coded convolutional neural network on the one hand, reducing convolution in convolutional neural networks
  • the calculation amount of the operation increases the calculation speed; on the other hand, according to the sparse state, the processing state of the accelerator is dynamically switched, and the flexibility of the accelerator is improved.
  • the dynamic sparse adjustment module in the embodiment includes an online density identification module, an output temporary registration module, a dynamic coding module, and a dynamic sparse control module; wherein the online density identification module For counting any of the feature maps, counting the number of 0 elements in the feature map and the total number of all the elements in the feature map; the number of 0 elements in the feature map and all the elements in the feature map The ratio between the total number is used as the density of the feature map; the output temporary registration module is configured to store each of the feature maps output by each layer in the convolutional neural network; the dynamic sparse control module is configured to The density of each of the feature maps output by the online density identification module is compared with a plurality of preset thresholds; the dynamic encoding module is configured to sparse each of the feature maps in the output temporary registration module according to a comparison result. coding.
  • the dynamic sparse adjustment module specifically includes four modules.
  • the online density identification module is used to count the number of non-zero elements in each feature map during the calculation process to calculate the density of each feature map.
  • the output temporary registration module is configured to temporarily store the feature map outputted by each layer in the convolutional neural network in a non-sparse format.
  • the dynamic sparse control module is configured to control a sparse state of the feature map by using a preset plurality of preset thresholds.
  • the dynamic encoding module performs sparse encoding on each of the feature maps in the output temporary registration module according to a sparse state of each of the feature maps, thereby increasing the speed of the convolution operation.
  • the preset threshold includes a first preset threshold and a second preset threshold; the preset threshold includes a first preset threshold and a second preset threshold;
  • the dynamic coding module is specifically configured to: if each of the feature maps has a density less than the first preset threshold, encode each of the feature maps into a sparse matrix storage format; if each of the feature maps is dense If the degree is greater than or equal to the first preset threshold, and is less than the second preset threshold, the 0 elements in each of the feature maps are marked; if the density of each of the feature maps is greater than or equal to the The second preset threshold does not encode each of the feature maps.
  • the preset threshold in this embodiment includes a first preset threshold th1 and a second preset threshold th2.
  • the dynamic sparse control module divides the feature state AS of each of the feature maps into three states according to the first preset threshold and the second preset threshold, that is, the density is less than the first preset threshold.
  • the feature map is divided into a completely sparse state S, a feature map having a density greater than or equal to the first predetermined threshold, and a feature map smaller than the second predetermined threshold is classified into a medium sparse state M, and the density is greater than or equal to the
  • the feature map of the second preset threshold is divided into a completely non-sparse state D.
  • each feature map is a sparse state S
  • the dynamic coding module encodes each of the feature maps in the output temporary registration module into a sparse matrix storage format, where the sparse matrix storage format includes non-characteristics in each of the feature maps. 0 data activ and sparse index index, such as coordinate encoding and compressed sparse row encoding. By encoding the feature map as a sparse matrix storage format, a large amount of storage space can be saved while saving a lot of computation time.
  • the dynamic coding module adds a flag guard to the 0 element in each feature map in the output temporary registration module, and may not participate in calculation and storage for the marked element, thereby reducing power consumption. . If each feature map is completely non-sparse state D, dynamic coding is not required, and the dynamic coding module directly outputs non-sparse data of each feature map.
  • the dynamic coding module in this embodiment is further configured to: if the density of each of the convolution kernels calculated in advance is less than the first preset threshold, each of the convolutions The kernel is encoded as a sparse matrix storage format; if the density of each of the convolution kernels is greater than or equal to the first predetermined threshold and less than the second predetermined threshold, then 0 of each of the convolution kernels The element is marked; if the density of each of the convolution kernels is greater than or equal to the second predetermined threshold, each of the convolution kernels is not encoded.
  • the density of each convolution kernel is the ratio between the number of non-zero elements in each convolution kernel and the total number of all elements in each convolution kernel.
  • the state WS of each convolution kernel has three states as in the feature map. Each state corresponds to a different sparse coding scheme. Since there are three states in the feature map and the convolution kernel, there are 9 states after the combination, so that the density of the convolutional neural network is more finely divided.
  • the neural network calculation array module in the embodiment is specifically configured to: when each of the feature maps or each of the convolution kernels has the label, Or the corresponding element in each of the convolution kernels is not calculated.
  • the 0 is reduced before each of the feature maps or each of the convolution kernels is input into the neural network calculation array module.
  • the storage space is not calculated at the same time; when each of the feature maps or each of the convolution kernels is in a medium sparse state M, the 0 elements in each of the feature maps or each of the convolution kernels are stored, However, the elements corresponding to the mark are not calculated, thereby reducing the calculation.
  • the chip of the accelerator is fabricated by a TSMC 65 nm process, the chip having an area of 3 mm*4 mm, an operating frequency of 20-200 MHz, and a power consumption of 20.5-248.4 mW.
  • the ultimate energy efficiency rises rapidly as the feature map and the convolution kernel density decrease, as shown in FIG.
  • the density of the feature map and the convolution kernel are both 5%, the ultimate energy efficiency can reach 62.1 TOPS/W, which is 6.2 times the ultimate energy efficiency when the accelerator of the embodiment is not used.
  • the energy efficiency of the embodiment can be increased by 4.3 times compared to the implementation of only supporting feature data sparse.
  • the energy efficiency of the present invention can be increased by 2.8 times.
  • the energy efficiency of the present invention can be increased by a factor of 2 compared to the implementation without variable density control but variable quantization accuracy.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)
  • Complex Calculations (AREA)

Abstract

本发明提供一种应用于卷积神经网络的加速方法和加速器，所述方法包括：S1，对于卷积神经网络中的任一层，分别计算该层输出的各特征图的稠密度；S2，将该层输出的各所述特征图的稠密度与多个预设阈值进行比较，根据比较结果将各所述特征图进行稀疏编码；其中，不同的比较结果对应不同的稀疏编码方式；S3，基于该层下一层的卷积层对稀疏编码后的各所述特征图和预先稀疏编码的所述卷积神经网络中的各卷积核进行卷积。本发明减少卷积神经网络中卷积运算的计算量，提高了运算速度。

Description

一种应用于卷积神经网络的加速方法和加速器
交叉引用
本申请引用于2018年04月08日提交的专利名称为“一种应用于卷积神经网络的加速方法和加速器”的第2018103065773号中国专利申请，其通过引用被全部并入本申请。
技术领域
本发明属于运算优化技术领域，更具体地，涉及一种应用于卷积神经网络的加速方法和加速器。
背景技术
卷积神经网络(Convolutional Neural Network,CNN)是一种前馈神经网络，其人工神经元可以响应一部分覆盖范围内的周围单元，适用于对大型图像的处理。卷积神经网络广泛应用于图像识别、语音识别等领域，但计算量非常大。
由于卷积神经网络中的激活函数ReLU(Rectified linear unit，修正线性单元)会造成大量稀疏的特征图(feature map)；同时，采用剪枝等方法训练卷积神经网络会造成大量稀疏的权重数据(weight data)。利用特征图和权重数据的稀疏性可以大幅提高卷积神经网络的计算效率。目前，已有很多方法基于卷积神经网络中特征图和权重数据的稀疏性提高计算速度。这些方法大致可以分为两类，一类着眼于跳过0值。例如有的方法去除输入中的0值，从而减少输入为0的无效计算。另一类采取忽略零值的方法。例如有的方法在输入数据为0时，不执行乘法操作，从而减少运算。但这些方法都着眼于处理稀疏神经网络本身，假定神经网络稀疏是前提。然而实际上卷积神经网络中各层输出特征图可能是稀疏的，可能是非稀疏的。实际运用中卷积神经网络各层的权重数据和特征图的稠密度一般在5％-90％之间分布。
稀疏矩阵是指数值为0的元素数目远远多于非0元素的数目，并且非0元素分布没有规律的矩阵。现有技术中一方面只能对稀疏的卷积神经网 络进行处理，在卷积神经网络不是稀疏的情况下计算量很大，运算速度低；另一方面现有技术只能处理卷积神经中权重数据或特征图是稀疏的情况，不能处理权重数据和特征图都是稀疏的情况。
发明内容
为克服上述卷积神经网络运算速度低的问题或者至少部分地解决上述问题，本发明提供了一种应用于卷积神经网络的加速方法和加速器。
根据本发明的第一方面，提供一种应用于卷积神经网络的加速方法，包括：
S1，对于卷积神经网络中的任一层，分别计算该层输出的各特征图的稠密度；
S2，将该层输出的各所述特征图的稠密度与多个预设阈值进行比较，根据比较结果将各所述特征图进行稀疏编码；其中，不同的比较结果对应不同的稀疏编码方式；
S3，基于该层下一层的卷积层对稀疏编码后的各所述特征图和预先稀疏编码的所述卷积神经网络中的各卷积核进行卷积。
具体地，所述步骤S1具体包括：
对于任一所述特征图，统计该特征图中非0元素的个数和该特征图中所有元素的总个数；
将该特征图中非0元素的个数与该特征图中所有元素的总个数之间的比值作为该特征图的稠密度。
具体地，所述预设阈值包括第一预设阈值和第二预设阈值；其中，所述第一预设阈值小于所述第二预设阈值；
相应地，所述步骤S2具体包括：
若各所述特征图的稠密度小于所述第一预设阈值，则将各所述特征图编码为稀疏矩阵存储格式；
若各所述特征图的稠密度大于或等于所述第一预设阈值，且小于所述第二预设阈值，则将各所述特征图中的0元素进行标记；
若各所述特征图的稠密度大于或等于所述第二预设阈值，则不对各所述特征图进行稀疏编码。
具体地，所述步骤S3之前还包括：
计算训练好的卷积网络中各卷积核的稠密度；
若各所述卷积核的稠密度小于所述第一预设阈值，则将各所述卷积核编码为稀疏矩阵存储格式；
若各所述卷积核的稠密度大于或等于所述第一预设阈值，且小于所述第二预设阈值，则将各所述卷积核中的0元素进行标记；
若各所述卷积核的稠密度大于或等于所述第二预设阈值，则不对各所述卷积核进行稀疏编码。
具体地，所述步骤S3具体包括：
当各所述特征图或各所述卷积核中存在所述标记时，对各所述特征图或各所述卷积核中标记对应的元素不进行计算。
根据本发明另一方面提供一种应用于卷积神经网络的加速器，包括：神经网络计算阵列模块和动态稀疏调整模块；
其中，所述动态稀疏调整模块用于计算卷积神经网络各层输出的各特征图的稠密度，将各所述特征图的稠密度与多个预设阈值进行比较，根据比较结果对各所述特征图进行稀疏编码；其中，不同的比较结果对应的不同稀疏编码方式；
所述神经网络计算阵列模块用于对稀疏编码后的各所述特征图和预先稀疏编码的所述卷积神经网络中的各卷积核进行卷积操作。
具体地，所述动态稀疏调整模块包括线上稠密度识别模块、输出临时寄存模块、动态编码模块和动态稀疏控制模块；
其中，所述线上稠密度识别模块用于对于任一所述特征图，统计该特征图中0元素的个数和该特征图中所有元素的总个数；将该特征图中0元素的个数与该特征图中所有元素的总个数之间的比值作为该特征图的稠密度；
所述输出临时寄存模块用于存储卷积神经网络中各层输出的各所述特征图；
所述动态稀疏控制模块用于将所述线上稠密度识别模块输出的各所述特征图的稠密度与多个预设阈值进行比较；
所述动态编码模块用于根据比较结果将所述输出临时寄存模块中各所述特征图进行稀疏编码。
具体地，所述预设阈值包括第一预设阈值和第二预设阈值；其中，所述第一预设阈值小于所述第二预设阈值；
相应地，所述动态编码模块具体用于：
若各所述特征图的稠密度小于所述第一预设阈值，则将各所述特征图编码为稀疏矩阵存储格式；
若各所述特征图的稠密度大于或等于所述第一预设阈值，且小于所述第二预设阈值，则将各所述特征图中的0元素进行标记；
若各所述特征图的稠密度大于或等于所述第二预设阈值，则不对各所述特征图进行稀疏编码。
具体地，所述动态编码模块还用于：
若预先计算的各所述卷积核的稠密度小于所述第一预设阈值，则将各所述卷积核编码为稀疏矩阵存储格式；
若各所述卷积核的稠密度大于或等于所述第一预设阈值，且小于所述第二预设阈值，则将各所述卷积核中的0元素进行标记；
若各所述卷积核的稠密度大于或等于所述第二预设阈值，则不对各所述卷积核进行稀疏编码。
具体地，所述神经网络计算阵列模块具体用于：
当各所述特征图或各所述卷积核中存在所述标记时，对各所述特征图或各所述卷积核中标记对应的元素不进行计算。
本发明提供一种应用于卷积神经网络的加速方法和加速器，该方法通过将卷积神经网络中各层输出的各特征图的稠密度与多个预设阈值进行比较，获取各所述特征图的稀疏状态，将不同稀疏状态的特征图进行不同方式的稀疏编码，然后基于各层下一层的卷积层对稀疏编码后的各特征图和预先稀疏编码的卷积神经网络中的卷积核进行卷积操作，减少卷积神经网络中卷积运算的计算量，提高运算速度。
附图说明
图1为本发明实施例提供的应用于卷积神经网络的加速方法整体流程示意图；
图2为本发明实施例提供的应用于卷积神经网络的加速器整体结构示意图；
图3为本发明实施例提供的应用于卷积神经网络的加速器中极限能量效率测试结果示意图；
图4为本发明实施例提供的应用于卷积神经网络的加速器中极限能量效率测试结果对比示意图。
具体实施方式
下面结合附图和实施例，对本发明的具体实施方式作进一步详细描述。以下实施例用于说明本发明，但不用来限制本发明的范围。
在本发明的一个实施例中提供一种应用于卷积神经网络的加速方法，图1为本发明实施例提供的应用于卷积神经网络的加速方法整体流程示意图，该方法包括：
S1，对于卷积神经网络中的任一层，分别计算该层输出的各特征图的稠密度；
具体地，所述卷积神经网络中可以包括池化层或不包括池化层。首先对所述卷积神经网络进行训练，训练完成后卷积神经网络中的卷积核不再变动，因此卷积神经网络中的卷积核不需要线上动态稀疏编码，直接线下稀疏编码一次即可。所述线上是指位于加速器的芯片上，所述线下是指不位于所述加速器的芯片上。在每次卷积运算时直接读取稀疏编码的卷积核进行卷积计算。当输入原始图像数据时，对原始图像数据进行稀疏编码，然后将稀疏编码的原始数据和稀疏编码的卷积核输入所述卷积神经网络的第一层卷积层进行卷积计算。由于原始图像数据一般不是稀疏的，也可以不对所述原始图像数据进行稀疏编码，直接输入所述原始图像数据。所述稀疏编码是将数据以稀疏格式存储。
S1中，由于卷积神经网络中每层输出的各特征图的稠密度不一样，不同层输出的特征图也是动态变化的，因此稠密度也是动态变化的。所述稠密度表示各所述特征图的稀疏程度。为了更好地提高所述卷积神经网络的运算速度，计算每一层输出的各特征图的稠密度，以根据每一层输出的各特征图的稠密度对每一层输出的各特征图进行稀疏编码。
S2，将该层输出的各所述特征图的稠密度与多个预设阈值进行比较，根据比较结果将各所述特征图进行稀疏编码；其中，不同的比较结果对应不同的稀疏编码方式；
S2中，现有技术中将各层输出的特征图全部进行稀疏编码，计算量大。本实施例根据所述预设阈值获取该层输出的各所述特征图的稀疏状态。从而将不同稀疏状态的特征图进行不同形式的稀疏编码。
S3，基于该层下一层的卷积层对稀疏编码后的各所述特征图和预先稀疏编码的所述卷积神经网络中的各卷积核进行卷积。
S3中，将稀疏编码后的各所述特征图和预先稀疏编码的所述卷积神经网络中的各卷积核作为该层下一层的卷积层的输入，进行卷积操作。然后，将所述卷积操作的结果作为所述卷积层下一层的输入，将所述卷积层下一层输出的特征图继续执行上述稀疏编码和卷积操作，直到所述卷积神经网络中最后一层输出各特征图为止。本实施例中不限于卷积核的稀疏编码方式。
本实施例将卷积神经网络中各层输出的各特征图的稠密度与多个预设阈值进行比较，获取各所述特征图的稀疏状态，将不同稀疏状态的特征图进行不同方式的稀疏编码，然后基于各层下一层的卷积层对稀疏编码后的各特征图和预先稀疏编码的卷积神经网络中的卷积核进行卷积操作，减少卷积神经网络中卷积运算的计算量，提高运算速度。
在上述实施例的基础上，本实施例中，所述步骤S1具体包括：对于任一所述特征图，统计该特征图中非0元素的个数和该特征图中所有元素的总个数；将该特征图中非0元素的个数与该特征图中所有元素的总个数之间的比值作为该特征图的稠密度。
具体地，各特征图的稠密度为各特征图中非0元素的个数与各特征图中所有元素的总个数之间的比值。例如，一个特征图中非0元素的个数为10，该特征图中所有元素的总个数为100，则该特征图的稠密度为0.1。
在上述实施例的基础上，本实施例中所述预设阈值包括第一预设阈值和第二预设阈值；所述预设阈值包括第一预设阈值和第二预设阈值；相应地，所述步骤S2具体包括：若各所述特征图的稠密度小于所述第一预设阈值，则将各所述特征图编码为稀疏矩阵存储格式；若各所述特征图的稠密度大于或等于所述第一预设阈值，且小于所述第二预设阈值，则将各所述特征图中的0元素进行标记；若各所述特征图的稠密度大于或等于所述第二预设阈值，则不对各所述特征图进行稀疏编码。
具体地，本实施例中所述预设阈值包括第一预设阈值th1和第二预设阈值th2。根据所述第一预设阈值和所述第二预设阈值将各所述特征图的特征状态AS分为三种状态，即将稠密度小于所述第一预设阈值的特征图分为完全稀疏状态S，将稠密度大于或等于所述第一预设阈值，且小于所述第二预设阈值的特征图分为中等稀疏状态M，将稠密度大于或等于所述第二预设阈值的特征图分为完全非稀疏状态D。若各特征图为稀疏状态S，则将各所述特征图编码为稀疏矩阵存储格式，所述稀疏矩阵存储格式包括各所述特征图中的非0数据activ和稀疏索引index，例如坐标编码和压缩稀疏行编码。通过将特征图编码为稀疏矩阵存储格式，可以节省大量存储空间，同时节省大量计算时间。若各特征图为中等稀疏状态M，则将各特征图中的0元素添加标记guard，所述标记用于标识0元素。对于标记的元素可以不参与计算和存储，从而降低功耗。将各特征图中的0元素进行标记也是一种稀疏编码方式。若各特征图为完全非稀疏状态D，则不需要动态编码，直接输出各特征图的非稀疏数据。
在上述实施例的基础上，本实施例中所述步骤S3之前还包括：计算训练好的卷积网络中各卷积核的稠密度；若各所述卷积核的稠密度小于所述第一预设阈值，则将各所述卷积核编码为稀疏矩阵存储格式；若各所述卷积核的稠密度大于或等于所述第一预设阈值，且小于所述第二预设阈值，则将各所述卷积核中的0元素进行标记；若各所述卷积核的稠密度大于或等于所述第二预设阈值，则不对各所述卷积核进行编码。
具体地，各卷积核的稠密度为各卷积核中非0元素的个数与各卷积核中所有元素的总个数之间的比值。将各卷积核的状态WS与特征图一样分为三种状态。每种状态对应不同的稀疏编码方式。由于特征图和卷积核各存在三种状态，组合后共有9种状态，从而对卷积神经网络的稠密度进行更细粒度的划分。
在上述各实施例的基础上，本实施例中所述步骤S3具体包括：当各所述特征图或各所述卷积核中存在所述标记时，对各所述特征图或各所述卷积核中标记对应的元素不进行计算。
具体地，当各所述特征图或各所述卷积核为完全稀疏状态S时，在输入前去除0，减少存储空间，同时不用对0元素进行计算；当各所述特征 图或各所述卷积核为中等稀疏状态M时，各所述特征图或各所述卷积核中的0元素虽然存储，但对标记对应的元素不进行计算，从而减少计算。
在本发明的另一个实施例中提供一种应用于卷积神经网络的加速器，图2为本发明实施例提供的应用于卷积神经网络的加速器整体结构示意图，包括：神经网络计算阵列模块和动态稀疏调整模块；其中，所述动态稀疏调整模块用于计算卷积神经网络中各层输出的各特征图的稠密度，将各所述特征图的稠密度与多个预设阈值进行比较，根据比较结果对各所述特征图进行稀疏编码；其中，不同的比较结果对应的不同稀疏编码方式；所述神经网络计算阵列模块用于对稀疏编码后的各所述特征图和预先稀疏编码的所述卷积神经网络中的各卷积核进行卷积操作。
具体地，所述卷积神经网络中可以包括池化层或不包括池化层。首先对卷积神经网络进行训练，训练完成后卷积神经网络中的卷积核不再变动，因此卷积神经网络中的卷积核不需要线上动态稀疏编码，直接线下稀疏编码一次即可。在每次卷积运算时所述神经网络计算阵列模块直接读取线下稀疏编码的卷积核进行卷积计算。当卷积神经网络输入原始图像数据时，动态稀疏调整模块对原始图像数据进行稀疏编码，然后神经网络计算阵列模块根据稀疏编码的原始数据和稀疏编码的卷积核进行卷积计算。由于原始图像数据一般不是稀疏的，也可以不对所述原始图像数据进行稀疏编码，直接输入所述原始图像数据。所述稀疏编码是将数据以稀疏格式存储。
由于卷积神经网络中各层输出的各特征图的稠密度不一样，不同层输出的特征图也是动态变化的，因此稠密度也是动态变化的。所述稠密度表示各所述特征图的稀疏程度。为了更好地提高所述卷积神经网络的运算速度，所述动态稀疏调整模块计算每一层输出的各特征图的稠密度，以根据每一层输出的各特征图的稠密度对每一层输出的各特征图进行稀疏编码。
所述动态稀疏调整模块根据多个预设阈值获取该层输出的各所述特征图的稀疏状态。从而将不同稀疏状态的特征图进行不同形式的稀疏编码，而不仅限于一种稀疏编码。现有技术中将各层输出的特证图全部进行稀疏编码，计算量大。
所述神经网络计算阵列模块根据稀疏编码后的各所述特征图和预先 稀疏编码的所述卷积神经网络中的各卷积核进行卷积操作。如果包括池化模块，所述池化模块将所述卷积操作的结果进行池化操作。此外，所述加速器还包括中间数据存储器模块、主芯片控制器和芯片上下数据交流模块。其中，所述主控制器控制加速器整个芯片的运行动作与时序。芯片上下数据交流模块用于从所述芯片外部存储器读取数据或将芯片计算好的数据写入外部存储。例如，初始化后，芯片在所述主控制器的控制下通过所述芯片上下数据交流模块从外部存储器读取原始图像数据和初始卷积核。所述中间数据存储器模块用于存储所述神经网络计算阵列模块计算过程中的中间结果。
本实施例动态稀疏调整模块将卷积神经网络各层输出的各特征图的稠密度与多个预设阈值进行比较，获取各所述特征图的稀疏状态，将不同稀疏状态的特征图进行不同方式的稀疏编码，以供神经网络计算阵列模块对稀疏编码后的各特征图和预先稀疏编码的卷积神经网络中的卷积核进行卷积操作，一方面，减少卷积神经网络中卷积运算的计算量，提高运算速度；另一方面，根据稀疏状态的不同，动态切换加速器的处理状态，提高了加速器的灵活性。
在上述实施例的基础上，本实施例中所述动态稀疏调整模块包括线上稠密度识别模块、输出临时寄存模块、动态编码模块和动态稀疏控制模块；其中，所述线上稠密度识别模块用于对于任一所述特征图，统计该特征图中0元素的个数和该特征图中所有元素的总个数；将该特征图中0元素的个数与该特征图中所有元素的总个数之间的比值作为该特征图的稠密度；所述输出临时寄存模块用于存储卷积神经网络中各层输出的各所述特征图；所述动态稀疏控制模块用于将所述线上稠密度识别模块输出的各所述特征图的稠密度与多个预设阈值进行比较；所述动态编码模块用于根据比较结果将所述输出临时寄存模块中各所述特征图进行稀疏编码。
具体地，所述动态稀疏调整模块具体包括四个模块。所述线上稠密度识别模块用于统计计算过程中各特征图中非0元素的个数，以计算各特征图的稠密度。所述输出临时寄存模块用于将卷积神经网络中各层输出的特征图以非稀疏格式进行暂存。所述动态稀疏控制模块用于通过预设的多个预设阈值控制所述特征图的稀疏状态。所述动态编码模块根据各所述特征 图的稀疏状态将所述输出临时寄存模块中各所述特征图进行稀疏编码，从而提高卷积运算的速度。
在上述实施例的基础上，本实施例中所述预设阈值包括第一预设阈值和第二预设阈值；所述预设阈值包括第一预设阈值和第二预设阈值；相应地，所述动态编码模块具体用于：若各所述特征图的稠密度小于所述第一预设阈值，则将各所述特征图编码为稀疏矩阵存储格式；若各所述特征图的稠密度大于或等于所述第一预设阈值，且小于所述第二预设阈值，则将各所述特征图中的0元素进行标记；若各所述特征图的稠密度大于或等于所述第二预设阈值，则不对各所述特征图进行编码。
具体地，本实施例中所述预设阈值包括第一预设阈值th1和第二预设阈值th2。所述动态稀疏控制模块根据所述第一预设阈值和所述第二预设阈值将各所述特征图的特征状态AS分为三种状态，即将稠密度小于所述第一预设阈值的特征图分为完全稀疏状态S，将稠密度大于或等于所述第一预设阈值，且小于所述第二预设阈值的特征图分为中等稀疏状态M，将稠密度大于或等于所述第二预设阈值的特征图分为完全非稀疏状态D。
若各特征图为稀疏状态S，则所述动态编码模块将所述输出临时寄存模块中各所述特征图编码为稀疏矩阵存储格式，所述稀疏矩阵存储格式包括各所述特征图中的非0数据activ和稀疏索引index，例如坐标编码和压缩稀疏行编码。通过将特征图编码为稀疏矩阵存储格式，可以节省大量存储空间，同时节省大量计算时间。若各特征图为中等稀疏状态M，则所述动态编码模块将所述输出临时寄存模块中各特征图中的0元素添加标记guard，对于标记的元素可以不参与计算和存储，从而降低功耗。若各特征图为完全非稀疏状态D，则不需要动态编码，所述动态编码模块直接输出各特征图的非稀疏数据。
在上述实施例的基础上，本实施例中所述动态编码模块还用于：若预先计算的各所述卷积核的稠密度小于所述第一预设阈值，则将各所述卷积核编码为稀疏矩阵存储格式；若各所述卷积核的稠密度大于或等于所述第一预设阈值，且小于所述第二预设阈值，则将各所述卷积核中的0元素进行标记；若各所述卷积核的稠密度大于或等于所述第二预设阈值，则不对各所述卷积核进行编码。
具体地，各卷积核的稠密度为各卷积核中非0元素的个数与各卷积核中所有元素的总个数之间的比值。各卷积核的状态WS与特征图一样有三种状态。每种状态对应不同的稀疏编码方式。由于特征图和卷积核各存在三种状态，组合后共有9种状态，从而对卷积神经网络的稠密度进行更细粒度的划分。
在上述各实施例的基础上，本实施例中所述神经网络计算阵列模块具体用于：当各所述特征图或各所述卷积核中存在所述标记时，对各所述特征图或各所述卷积核中标记对应的元素不进行计算。
具体地，当各所述特征图或各所述卷积核为完全稀疏状态S时，在将各所述特征图或各所述卷积核输入所述神经网络计算阵列模块前去除0，减少存储空间，同时不用对0元素进行计算；当各所述特征图或各所述卷积核为中等稀疏状态M时，各所述特征图或各所述卷积核中的0元素虽然存储，但对标记对应的元素不进行计算，从而减少计算。
例如，采用台积电65nm工艺制作所述加速器的芯片，所述芯片的面积为3mm*4mm，运行频率为20-200MHz，功耗为20.5-248.4毫瓦。本实施例中极限能量效率会随着特征图和卷积核稠密度的下降而快速上升，如图3所示。当特征图和卷积核的稠密度均为5％时，极限能量效率可达到62.1TOPS/W，是不采用本实施例加速器时极限能量效率的6.2倍。如图4所示，相比于只支持特征数据稀疏的实现，本实施例能量效率可提升4.3倍。相比于无自适应稀疏控制的实现，本发明能量效率可提升2.8倍。相比于无稠密度控制但量化精度可变的实现，本发明能量效率可提升2倍。
最后，本申请的方法仅为较佳的实施方案，并非用于限定本发明的保护范围。凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。

Claims (10)

  1. 一种应用于卷积神经网络的加速方法，其特征在于，包括：
    S1，对于卷积神经网络中的任一层，分别计算该层输出的各特征图的稠密度；
    S2，将该层输出的各所述特征图的稠密度与多个预设阈值进行比较，根据比较结果将各所述特征图进行稀疏编码；其中，不同的比较结果对应不同的稀疏编码方式；
    S3，基于该层下一层的卷积层对稀疏编码后的各所述特征图和预先稀疏编码的所述卷积神经网络中的各卷积核进行卷积。
  2. 根据权利要求1所述的方法，其特征在于，所述步骤S1具体包括：
    对于任一所述特征图，统计该特征图中非0元素的个数和该特征图中所有元素的总个数；
    将该特征图中非0元素的个数与该特征图中所有元素的总个数之间的比值作为该特征图的稠密度。
  3. 根据权利要求1所述的方法，其特征在于，所述预设阈值包括第一预设阈值和第二预设阈值；其中，所述第一预设阈值小于所述第二预设阈值；
    相应地，所述步骤S2具体包括：
    若各所述特征图的稠密度小于所述第一预设阈值，则将各所述特征图编码为稀疏矩阵存储格式；
    若各所述特征图的稠密度大于或等于所述第一预设阈值，且小于所述第二预设阈值，则将各所述特征图中的0元素进行标记；
    若各所述特征图的稠密度大于或等于所述第二预设阈值，则不对各所述特征图进行稀疏编码。
  4. 根据权利要求3所述的方法，其特征在于，所述步骤S3之前还包括：
    计算训练好的卷积网络中各卷积核的稠密度；
    若各所述卷积核的稠密度小于所述第一预设阈值，则将各所述卷积核编码为稀疏矩阵存储格式；
    若各所述卷积核的稠密度大于或等于所述第一预设阈值，且小于所述 第二预设阈值，则将各所述卷积核中的0元素进行标记；
    若各所述卷积核的稠密度大于或等于所述第二预设阈值，则不对各所述卷积核进行稀疏编码。
  5. 根据权利要求3或4所述的方法，其特征在于，所述步骤S3具体包括：
    当各所述特征图或各所述卷积核中存在所述标记时，对各所述特征图或各所述卷积核中标记对应的元素不进行计算。
  6. 一种应用于卷积神经网络的加速器，其特征在于，包括：神经网络计算阵列模块和动态稀疏调整模块；
    其中，所述动态稀疏调整模块用于计算卷积神经网络中各层输出的各特征图的稠密度，将各所述特征图的稠密度与多个预设阈值进行比较，根据比较结果对各所述特征图进行稀疏编码；其中，不同的比较结果对应的不同稀疏编码方式；
    所述神经网络计算阵列模块用于对稀疏编码后的各所述特征图和预先稀疏编码的所述卷积神经网络中的各卷积核进行卷积操作。
  7. 根据权利要求6所述的加速器，其特征在于，所述动态稀疏调整模块包括线上稠密度识别模块、输出临时寄存模块、动态编码模块和动态稀疏控制模块；
    其中，所述线上稠密度识别模块用于对于任一所述特征图，统计该特征图中0元素的个数和该特征图中所有元素的总个数；将该特征图中0元素的个数与该特征图中所有元素的总个数之间的比值作为该特征图的稠密度；
    所述输出临时寄存模块用于存储卷积神经网络中各层输出的各所述特征图；
    所述动态稀疏控制模块用于将所述线上稠密度识别模块输出的各所述特征图的稠密度与多个预设阈值进行比较；
    所述动态编码模块用于根据比较结果将所述输出临时寄存模块中各所述特征图进行稀疏编码。
  8. 根据权利要求7所述的加速器，其特征在于，所述预设阈值包括第一预设阈值和第二预设阈值；其中，所述第一预设阈值小于所述第二预 设阈值；
    相应地，所述动态编码模块具体用于：
    若各所述特征图的稠密度小于所述第一预设阈值，则将各所述特征图编码为稀疏矩阵存储格式；
    若各所述特征图的稠密度大于或等于所述第一预设阈值，且小于所述第二预设阈值，则将各所述特征图中的0元素进行标记；
    若各所述特征图的稠密度大于或等于所述第二预设阈值，则不对各所述特征图进行稀疏编码。
  9. 根据权利要求8所述的加速器，其特征在于，所述动态编码模块还用于：
    若预先计算的各所述卷积核的稠密度小于所述第一预设阈值，则将各所述卷积核编码为稀疏矩阵存储格式；
    若各所述卷积核的稠密度大于或等于所述第一预设阈值，且小于所述第二预设阈值，则将各所述卷积核中的0元素进行标记；
    若各所述卷积核的稠密度大于或等于所述第二预设阈值，则不对各所述卷积核进行稀疏编码。
  10. 根据权利要求7或8所述的加速器，其特征在于，所述神经网络计算阵列模块具体用于：
    当各所述特征图或各所述卷积核中存在所述标记时，对各所述特征图或各所述卷积核中标记对应的元素不进行计算。
PCT/CN2018/095365 2018-04-08 2018-07-12 一种应用于卷积神经网络的加速方法和加速器 Ceased WO2019196223A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810306577.3 2018-04-08
CN201810306577.3A CN108510063B (zh) 2018-04-08 2018-04-08 一种应用于卷积神经网络的加速方法和加速器

Publications (1)

Publication Number Publication Date
WO2019196223A1 true WO2019196223A1 (zh) 2019-10-17

Family

ID=63380995

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/095365 Ceased WO2019196223A1 (zh) 2018-04-08 2018-07-12 一种应用于卷积神经网络的加速方法和加速器

Country Status (2)

Country Link
CN (1) CN108510063B (zh)
WO (1) WO2019196223A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111401554A (zh) * 2020-03-12 2020-07-10 交叉信息核心技术研究院(西安)有限公司 支持多粒度稀疏与多模式量化的卷积神经网络的加速器
WO2023164855A1 (en) * 2022-03-03 2023-09-07 Intel Corporation Apparatus and method for 3d dynamic sparse convolution
WO2025095929A1 (en) * 2023-10-30 2025-05-08 Google Llc Controllable neural network sparsity through dynamic activation functions

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109389043B (zh) * 2018-09-10 2021-11-23 中国人民解放军陆军工程大学 一种无人机航拍图片的人群密度估计方法
CN109409518B (zh) * 2018-10-11 2021-05-04 北京旷视科技有限公司 神经网络模型处理方法、装置及终端
US20200143226A1 (en) * 2018-11-05 2020-05-07 Samsung Electronics Co., Ltd. Lossy compression of neural network activation maps
CN109784484A (zh) * 2019-01-31 2019-05-21 深兰科技（上海）有限公司 神经网络加速方法、装置、神经网络加速芯片及存储介质
WO2020168423A1 (en) * 2019-02-19 2020-08-27 Lei Zhang Method and system for convolution model multi-mode hardware accelerator
CN110163340B (zh) * 2019-03-08 2025-04-08 腾讯科技（深圳）有限公司 使用卷积神经网络进行计算的方法、装置和计算机可读存储介质
CN110097172B (zh) * 2019-03-18 2021-10-29 中国科学院计算技术研究所 一种基于winograd卷积运算的卷积神经网络数据处理方法及装置
CN109858575B (zh) * 2019-03-19 2024-01-05 苏州市爱生生物技术有限公司 基于卷积神经网络的数据分类方法
CN110443357B (zh) * 2019-08-07 2020-09-15 上海燧原智能科技有限公司 卷积神经网络计算优化方法、装置、计算机设备及介质
CN110909801B (zh) * 2019-11-26 2020-10-09 山东师范大学 基于卷积神经网络的数据分类方法、系统、介质及设备
CN111291230B (zh) * 2020-02-06 2023-09-15 北京奇艺世纪科技有限公司 特征处理方法、装置、电子设备及计算机可读存储介质
CN113537465B (zh) * 2021-07-07 2024-10-08 深圳市易成自动驾驶技术有限公司 Lstm模型优化方法、加速器、装置及介质
CN115293339B (zh) * 2022-07-29 2026-02-17 浙江大学 基于神经网络中数据稠密特性的智能训练加速方法及系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105184362A (zh) * 2015-08-21 2015-12-23 中国科学院自动化研究所 基于参数量化的深度卷积神经网络的加速与压缩方法
US20170103309A1 (en) * 2015-10-08 2017-04-13 International Business Machines Corporation Acceleration of convolutional neural network training using stochastic perforation
CN107239824A (zh) * 2016-12-05 2017-10-10 北京深鉴智能科技有限公司 用于实现稀疏卷积神经网络加速器的装置和方法
CN107239825A (zh) * 2016-08-22 2017-10-10 北京深鉴智能科技有限公司 考虑负载均衡的深度神经网络压缩方法
CN107609641A (zh) * 2017-08-30 2018-01-19 清华大学 稀疏神经网络架构及其实现方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105184362A (zh) * 2015-08-21 2015-12-23 中国科学院自动化研究所 基于参数量化的深度卷积神经网络的加速与压缩方法
US20170103309A1 (en) * 2015-10-08 2017-04-13 International Business Machines Corporation Acceleration of convolutional neural network training using stochastic perforation
CN107239825A (zh) * 2016-08-22 2017-10-10 北京深鉴智能科技有限公司 考虑负载均衡的深度神经网络压缩方法
CN107239824A (zh) * 2016-12-05 2017-10-10 北京深鉴智能科技有限公司 用于实现稀疏卷积神经网络加速器的装置和方法
CN107609641A (zh) * 2017-08-30 2018-01-19 清华大学 稀疏神经网络架构及其实现方法

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111401554A (zh) * 2020-03-12 2020-07-10 交叉信息核心技术研究院(西安)有限公司 支持多粒度稀疏与多模式量化的卷积神经网络的加速器
CN111401554B (zh) * 2020-03-12 2023-03-24 交叉信息核心技术研究院(西安)有限公司 支持多粒度稀疏与多模式量化的卷积神经网络的加速器
WO2023164855A1 (en) * 2022-03-03 2023-09-07 Intel Corporation Apparatus and method for 3d dynamic sparse convolution
WO2025095929A1 (en) * 2023-10-30 2025-05-08 Google Llc Controllable neural network sparsity through dynamic activation functions

Also Published As

Publication number Publication date
CN108510063B (zh) 2020-03-20
CN108510063A (zh) 2018-09-07

Similar Documents

Publication Publication Date Title
CN108510063B (zh) 一种应用于卷积神经网络的加速方法和加速器
CN108510066B (zh) 一种应用于卷积神经网络的处理器
CN113657560B (zh) 基于节点分类的弱监督图像语义分割方法及系统
KR20200060302A (ko) 처리방법 및 장치
Zong et al. ASHF-Net: Adaptive sampling and hierarchical folding network for robust point cloud completion
CN116343340B (zh) 一种基于Transformer改进的人体骨架数据行为识别方法及系统
CN113255910A (zh) 卷积神经网络的剪枝方法、装置、电子设备和存储介质
CN117036832B (zh) 一种基于随机多尺度分块的图像分类方法、装置及介质
CN115659823B (zh) 基于属性约简的翼型气动力系数预测方法、电子设备及存储介质
CN113177580A (zh) 基于通道重要性剪枝与二值量化的图像分类系统
Jain et al. Symmetric $ k $-means for deep neural network compression and hardware acceleration on FPGAs
CN111160523A (zh) 基于特征值区域的动态量化方法、系统及介质
CN104318271B (zh) 一种基于适应性编码和几何平滑汇合的图像分类方法
CN119849576A (zh) 预训练模型的参数微调方法、装置、设备及介质
CN112926658B (zh) 基于二维数据嵌入与邻接拓扑图的图像聚类方法及设备
CN110728303A (zh) 基于卷积神经网络数据复杂度的动态自适应计算阵列
CN115994571B (zh) 一种基于fpga的卷积神经网络块剪枝方法
CN111126501B (zh) 一种图像识别方法、终端设备及存储介质
CN110363287B (zh) 一种面向内存计算和室内是否有人的神经网络设计方法
CN113705784B (zh) 一种基于矩阵共享的神经网络权重编码方法及硬件系统
Chung et al. Using quantization-aware training technique with post-training fine-tuning quantization to implement a MOBILENET hardware accelerator
CN104951651B (zh) 一种基于黑森正则约束与a优化的非负图像数据降维方法
CN118506094A (zh) 一种基于时间可逆脉冲Transformer的图像分类方法及装置
CN119206245A (zh) 由电子设备执行的方法、电子设备、存储介质及程序产品
CN117273081A (zh) 一种基于忆阻器的神经网络压缩方法及神经网络加速器

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18914271

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18914271

Country of ref document: EP

Kind code of ref document: A1