WO2021174790A1 - 稀疏量化神经网络编码模式识别方法与系统 - Google Patents

稀疏量化神经网络编码模式识别方法与系统 Download PDF

Info

Publication number
WO2021174790A1
WO2021174790A1 PCT/CN2020/113050 CN2020113050W WO2021174790A1 WO 2021174790 A1 WO2021174790 A1 WO 2021174790A1 CN 2020113050 W CN2020113050 W CN 2020113050W WO 2021174790 A1 WO2021174790 A1 WO 2021174790A1
Authority
WO
WIPO (PCT)
Prior art keywords
neural network
data
bit
calculation
sparse
Prior art date
Application number
PCT/CN2020/113050
Other languages
English (en)
French (fr)
Inventor
周喜川
刘奎
梁松红
Original Assignee
重庆大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 重庆大学 filed Critical 重庆大学
Publication of WO2021174790A1 publication Critical patent/WO2021174790A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G06V20/13—Satellite images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/94—Hardware or software architectures specially adapted for image or video understanding

Definitions

  • the invention belongs to the technical field of neural networks, in particular the field of sparse quantization neural networks, and relates to a sparse quantization neural network coding pattern recognition method and system.
  • Convolutional neural networks have proven to be successful for artificial intelligence and have great potential in different computer vision applications, such as autonomous driving and robotics.
  • deep learning based on pattern recognition on resource-constrained mobile embedded platforms is still a challenge.
  • energy-efficient neural network hardware design has attracted attention from academia to industry.
  • a common optimization method is to quantify and compress neural network activations.
  • the commonly used quantification method is to continuously perform iterative rounding quantization during the back propagation of the neural network.
  • This rounding method will cause the loss of information, especially when the data is small, it will cause great data distortion. Moreover, the rounding calculation is not guided, and the optimal solution cannot be found in the back propagation process of the neural network based on gradient descent, which leads to the decline of the quantized performance of the neural network.
  • Another method of extreme compression of deep neural networks uses a one-digit binary number to represent the neural network.
  • the neural network solves the non-convex programming problem and uses +1 and -1 connections to approximate the parameters of the neural network. Although this method can greatly compress the neural network, it will cause a significant drop in the performance of the neural network.
  • the purpose of the present invention is to provide a sparse quantization neural network coding pattern recognition method and system.
  • the quantized neural network has a higher accuracy rate and a lower code rate.
  • the above-mentioned quantization and compression methods are both non-differentiable and non-convex. In practice, it is almost impossible to obtain the optimal solution, resulting in the loss of the activation completeness and classification accuracy of the neural network.
  • the present invention proposes an information bit bottleneck method based on sparse convex optimization to achieve optimal quantization and compression.
  • Rate Distortion Theory as a lossy data compression coding method, trying to determine the most important bits in fixed-point activation and minimize the quantization loss.
  • the bit bottleneck transforms the quantization and compression of the neural network into a sparse convex optimization problem, which can estimate the most important bits in the activation and achieve the smallest square difference distortion. Because the unimportant bits are usually nearly zero coefficients, the activation can be optimally compressed bitwise.
  • This method can effectively compress the current fixed-point neural network activation, so that the code rate of the neural network activation representation is lower, and the accuracy rate is almost not lost compared with the original network. Save the hardware resources and power consumption of the neural network hardware system.
  • the present invention provides the following technical solutions:
  • Sparse quantization neural network coding pattern recognition method the method includes the following steps:
  • R D ⁇ I d is a quantization function, which converts the input D-bit real-valued data into a d-bit integer;
  • g( ⁇ ) is a function indicating the code rate of the input data; according to the rate-distortion theory, the typical The lossy data compression method tries to minimize the distortion function d( ⁇ ) given the maximum number of bits:
  • the standard quantization function uses D binary vectors x i1 ,x i2 ,...,x iD to approximate Referred to as x ij ⁇ 0,1 ⁇ M , represents the M-dimensional binary vector of the j-th bit of x i; assuming real value activation Is normalized to the area of [0, 2 D ], the standard rounding quantization method uses the following fixed-point data:
  • Each bit is assigned a constant coefficient ⁇ 2 0 ,..., 2 D-1 ⁇ , and the value of x ij is rounded get;
  • the minimum standard square distortion rate on N training samples is defined as
  • Formula (4) is equivalent to the following sparse solution problem based on L1 norm:
  • the bit bottleneck operation solves formula (5) to determine the sparse important bits and achieves the smallest rate distortion; solves formula (5) by solving its dual form:
  • ⁇ is a hyperparameter used to control the optimal error rate and code rate:
  • formula (6) leads to a sparse solution of the coefficient ⁇ , and the activation bits related to the zero coefficient of D-d are deleted in the inference stage to improve calculation efficiency;
  • Step 1 Select a pre-trained neural network model, set a compressed peak signal-to-noise ratio PSNR loss threshold, denoted as T, determine an initial quantization rate, denoted as D;
  • Step 2 Apply any basic method to quantify the activation of the pre-trained neural network, and obtain the activation with D-bit fixed-point representation, that is, x ij ⁇ ⁇ 0, 1 ⁇ M in formula (5);
  • Step 3 Before each convolutional layer, train the information bit bottleneck by solving formula (6), and insert it into the network model to reduce the loss of basic quantization; taking the same PNSR loss threshold T as a reference, each bit bottleneck With different sparsity levels and different numbers of non-zero coefficients ⁇ j , it is equivalent to the effective code rate after compression being less than D;
  • Step 4 Through the back propagation training process, the weight of the deep neural network with information bottleneck is tuned and quantified
  • Step 5 Load the trained neural network weights, bit coefficient vector ⁇ , and bias to the neural network hardware system, and the hardware system deployment is completed.
  • CPU direct memory access DMA, serial port, bus, controller, on-chip memory and neural network accelerator
  • the CPU controls the operation of the entire computing process
  • the serial port is used to receive data
  • the DMA is used to schedule the transmission of input data
  • the bus is used to transmit control commands and data
  • the neural network accelerator includes a computing array, an input frame buffer and an output frame buffer, which are used to realize data buffering and data multiplexing; the input frame buffer is used to capture the data required for each convolution and rearrange the data;
  • each calculation array includes multiple calculation units, and each calculation unit implements all the calculations of a macro module in the sparse quantization neural network;
  • the controller is used to control the coordinated operation of each module, and communicates with the CPU through the bus, and receives instructions from the CPU.
  • the calculation array includes a plurality of calculation units, and each calculation unit implements all calculations of a macro module in the sparse quantization neural network.
  • all calculations of the one macromodule include batch normalization calculation, bit bottleneck calculation, convolution calculation, activation calculation, and pooling calculation; bit bottleneck calculation can be performed simultaneously with convolution calculation.
  • the calculation unit includes a data cache module, a normalization and convolution module, and an activation and pooling module;
  • data caching module After the data caching module performs data caching, batch normalization, convolution bit multiplication array, and bias caching are performed in the normalization and convolution module; then activation and pooling are performed in the activation and pooling module; then The output is activated.
  • bit bottleneck compression quantization method is convex, it is more powerful and robust than conventional and other data-driven activation compression methods.
  • bit bottleneck operation is very flexible, which can be used to quantize and compress floating-point activation to any size of code rate, so that the hardware system architecture of the neural network can balance the efficiency and accuracy of different applications and reduce the hardware design Time-consuming computing resources and storage consumption, and can reduce the power consumption of the entire system architecture.
  • bit bottleneck can minimize the loss of activation information caused by quantization and compression
  • a deep neural network hardware system architecture with a bit bottleneck can achieve higher classification accuracy than hardware system architectures that use other activation compression methods.
  • Figure 1 is a sparse quantitative neural network architecture diagram
  • Figure 2 shows the hardware architecture diagram of the sparse quantitative neural network system
  • Figure 3 shows the use method of neural network system architecture based on sparse quantization coding
  • Figure 4 is a schematic diagram of an embodiment.
  • the goal of lossy data compression is to achieve the smallest rate distortion under the limitation of the maximum coding rate.
  • R D ⁇ I d is a quantization function, which converts the input D-bit real-valued data into a d-bit integer.
  • g( ⁇ ) is a function indicating the code rate of the input data.
  • typical lossy data compression methods try to minimize the distortion function d( ⁇ ) given the maximum number of bits.
  • the quantization function f(x) is non-differentiable and non-convex, which makes it difficult to solve. Different from the typical rounding-based quantization method, it only provides a sub-optimal solution. This article attempts to find the best solution by re-transforming formula (1) into a sparse coding problem.
  • the standard quantization function uses D binary vectors x i1 ,x i2 ,...,x iD to approximate Referred to as x ij ⁇ 0,1 ⁇ M , represents the M-dimensional binary vector of the j-th bit of x i; assuming real value activation Is normalized to the area of [0, 2 D ], the standard rounding quantization method uses the following fixed-point data:
  • Each bit is assigned a constant coefficient ⁇ 2 0 ,..., 2 D-1 ⁇ , and the value of x ij is rounded get;
  • this method attempts to find and estimate the optimal sparsity coefficient ⁇ associated with the most important bits.
  • this method attempts to define the minimum standard square distortion rate on N training samples as
  • the constraint based on the L0 norm limits the total number of non-zero coefficients, which will cause the coefficient ⁇ to be sparse. Since the number of non-zero coefficients in ⁇ is equal to the number of bits in the fixed-point representation, the constraint function of formula (4) will actually limit the maximum number of bits in the quantized representation according to the requirements of rate-distortion theory.
  • ⁇ is a hyperparameter used to control the optimal error rate and code rate.
  • Formula (6) usually leads to a sparse solution of the coefficient ⁇ , and the activation bits related to the D-d zero coefficient are deleted in the inference stage, and the calculation efficiency can be significantly improved.
  • the information bit bottleneck operation calculates the optimal coefficient associated with each bit of the compressed activation representation, so that the minimum distortion rate can be achieved when the maximum coding rate is given. This section shows how information bottlenecks work in deep neural networks.
  • the bit bottleneck can be inserted into the macro blocks of different networks.
  • a typical macro block package includes the batch normalization layer, the bit bottleneck layer, the convolution layer, the activation layer, and the pooling layer (optional) in sequence. Since the bit bottleneck layer converts normalized floating-point activations into compressed fixed-point activations, the convolutional layer can replace computationally intensive floating-point multiplications with efficient fixed-point integer multiplications. Fixed-point integer multiplication is usually implemented in bitwise calculations, and the number of active bits is proportional to the actual number of bitwise calculations. Therefore, the time and memory efficiency of computationally intensive convolution calculations can be greatly improved.
  • the overall hardware architecture includes six parts: CPU, DMA (direct memory access), serial port, bus, on-chip memory, and neural network accelerator.
  • the CPU controls the operation of the entire computing process
  • the serial port is used to receive data
  • the DMA is used to schedule the transmission of input data
  • the bus is used to transmit control commands and data.
  • the neural network computing accelerator is mainly composed of computing array, input frame buffer and output frame buffer.
  • the input and output frame buffer is used to realize data buffering and data multiplexing.
  • the calculation array is used to realize the calculation inside the neural network, the design of multiple arrays, and the design of multiple calculation units per array, the purpose is to realize the parallel calculation of the neural network and increase the calculation efficiency.
  • each computing unit implements all the calculations of a macro module in the sparse quantization neural network, including batch normalization calculations, bit bottleneck calculations, convolution calculations, activation calculations, and pooling calculations.
  • the input block buffer is used to capture the data needed for each convolution and rearrange the data.
  • the bit bottleneck calculation can be performed at the same time as the convolution calculation.
  • the controller is used to control the coordinated operation of each module, and communicates with the CPU through the bus, and receives instructions from the CPU.
  • This patent proposes that a sparse quantization neural network system uses bit bottleneck operations to extract the sparseness between activated data bits, so as to reduce the loss of activation integrity and classification accuracy caused by quantization. Assuming that someone intends to obtain a sparse quantized neural network system with a neural network activation less than D-bit, the following five steps are required.
  • Step 1 Select a pre-trained neural network model, set a compressed peak signal-to-noise ratio (PSNR) loss threshold, denoted as T, and determine an initial quantization rate, denoted as D.
  • PSNR signal-to-noise ratio
  • Step 2 Apply any basic method to quantify the activation of the pre-trained neural network, and obtain the activation with D-bit fixed-point representation, that is, x ij ⁇ ⁇ 0, 1 ⁇ M in formula (5).
  • Step 3 Before each convolutional layer, the information bit bottleneck can be trained by solving formula (6) and inserted into the network model to reduce the loss of basic quantization. Taking the same PNSR loss threshold T as a reference, each bit bottleneck may have a different sparsity level and a different number of non-zero coefficients ⁇ j , which is equivalent to an effective code rate less than D after compression.
  • Step 4 Through the back-propagation training process, the weights of the deep neural network with information bottleneck are tuned and quantified.
  • Step 5 Load the trained neural network weights, bit coefficient vector ⁇ and bias into the neural network hardware system. At this point, the entire hardware system has been deployed and can be put into practical applications.
  • the sparse quantitative neural network hardware system can be applied to the UAV platform with recognition function.
  • the camera captures images, sends them to the hardware system for intelligent information processing, detects and recognizes video image information, and sends the results back to the drone control system.
  • the control system can send the information to the ground receiving station, and the control system can also base on the recognition results Control the motor to control the flight attitude and path of the aircraft.
  • the system can realize functions such as forest fire detection, specific target tracking, urban smart fire protection, and transmission line inspection.
  • the sparse quantitative neural network hardware system can speed up information processing and realize high-speed recognition. It can be applied to mobile embedded systems that require high-speed image information processing, such as drones, self-driving cars, and smart robots.

Abstract

本发明涉及一种稀疏量化神经网络编码模式识别方法与系统，属于神经网络技术领域。该系统包括CPU、直接存储器访问DMA、串口、总线、控制器、片上内存和神经网络加速器；CPU控制整个计算进程的运行，串口用于接收数据，DMA用于调度输入数据的传输，总线用于传输控制命令和数据；神经网络加速器包括计算阵列与输入帧缓存和输出帧缓存，用于实现数据缓冲和数据复用；输入帧缓存用于抓取每次卷积需要的数据，并将数据进行重新排列；计算阵列用于实现神经网络内部的计算，每个计算阵列包括多个计算单元，每个计算单元实现稀疏量化神经网络中一个宏模块所有计算。本发明中比常规的和其他数据驱动的激活压缩方法更强大，具有更强的鲁棒性。

Description

稀疏量化神经网络编码模式识别方法与系统 技术领域
本发明属于神经网络技术领域，特别是稀疏量化神经网络领域，涉及一种稀疏量化神经网络编码模式识别方法与系统。
背景技术
卷积神经网络对于人工智能已被证明是成功的，在不同的计算机视觉应用上具有巨大的潜力，比如自动驾驶和机器人。但是由于其大量的参数，导致其在资源受限的移动嵌入式平台上的基于模式识别的深度学习依然是一个挑战。为了应对这个挑战，高能效神经网络硬件设计的研究吸引了从学术界到工业界的关注。为了使神经网络硬件系统具有更高的能效，常常需要进行软硬件协同设计，即在算法层面需要对神经网络进行面向硬件实现的优化。目前一种常见的优化方法是对神经网络激活进行量化压缩。通常采用的量化方法是在神经网络反向传播过程中，不断进行迭代的四舍五入取整量化。这种舍入的方法会导致信息的丢失，尤其当数据较小时，会造成很大的数据失真。并且舍入计算不可导，无法在基于梯度下降的神经网络反向传播过程中找到其最优解，从而导致神经网络量化后的性能下降。
另外一种极限压缩深度神经网络的方法使用一位二值制数来表示神经网络，该神经网络通过解决非凸规划问题，使用+1和-1连接来近似神经网络的参数。这种方法虽然可以极大的压缩神经网络，但会引起神经网络性能的大幅下降。
近来，有人提出了一种基于比特感知的方法，证明定点激活中的某些比特可以被删除，并且不会显著影响分类结果。这种方法是基于实验结果删除激活中的某些比特，不具备理论基础，结果并不鲁棒。
发明内容
有鉴于此，本发明的目的在于提供一种稀疏量化神经网络编码模式识别方法与系统。为了使神经网络硬件设计后具有更高的性能，使用的硬件资源更少，期望量化的神经网络准确率更高，码率更低。而上述的量化和压缩方法都是不可微和非凸的，在实际中几乎无法得到其最优的解，导致神经网络激活完整性和分类准确率丢失。代替使用传统的迭代方法来量化和压缩卷积神经网络激活，本发明提出了一种基于稀疏凸优化的信息比特瓶颈方法来实现最优的量化和压缩。该方法基于率失真理论(Rate Distortion Theory)，作为是一种有损的数据压缩编码方法，试图用于确定在定点激活中最重要的比特位，并且使量化损失最小。从技术 上讲，比特瓶颈将神经网络的量化与压缩转化为了一个稀疏凸优化的问题，能够估量激活中最重要的比特，达到最小的平方差失真。因为不重要的比特通常是近乎零的系数，因此激活能够被最优的进行按位压缩。该方法能够有效压缩目前的定点神经网络激活，使神经网络激活表示的码率更低，且准确率相比原网络几乎不丢失，能够
Figure PCTCN2020113050-appb-000001
节省神经网络硬件系统的硬件资源和功耗。
为达到上述目的，本发明提供如下技术方案：
稀疏量化神经网络编码模式识别方法，该方法包括以下步骤：
假设f：R D→I d是量化函数，将输入的D-bit实值数据转换为d-bit整数；假设g(·)是指示输入数据码率的函数；根据率失真理论，典型的有损数据压缩方法尝试在给定最大比特数的情况下，最小化失真函数d(·)：
min d(x，f(x))st.g(x)≤η    (1)
假定
Figure PCTCN2020113050-appb-000002
是与第i个样本相关的浮点激活矢量；标准的量化函数使用D个二值矢量x i1,x i2,…,x iD来近似
Figure PCTCN2020113050-appb-000003
记作
Figure PCTCN2020113050-appb-000004
x ij∈{0，1} M，表示x i的第j个比特位的M维二值矢量；假设实值激活
Figure PCTCN2020113050-appb-000005
被归一化到[0，2 D]的区域，则标准的舍入量化方法使用如下定点数据：
Figure PCTCN2020113050-appb-000006
其中每个比特分配有一个常数系数{2 0，...，2 D-1}，并且x ij的值通过四舍五入
Figure PCTCN2020113050-appb-000007
得到；
用变量α∈R D替换固定系数，如下：
Figure PCTCN2020113050-appb-000008
找到并估计与最重要比特相关的最优稀疏系数α；
将N个训练样本上的最小化标准平方失真率定义为
Figure PCTCN2020113050-appb-000009
公式(4)等效于以下基于L1范数求解稀疏解的问题：
Figure PCTCN2020113050-appb-000010
比特瓶颈操作通过求解公式(5)，以确定稀疏的重要的比特位，并实现最小的率失真；通过求解它的对偶形式来求解公式(5)：
Figure PCTCN2020113050-appb-000011
其中λ是用于控制最优的错误率和码率的超参数：公式(6)导致系数α的稀疏解，并且在推理阶段将与D-d零系数相关的激活比特位删除，提高计算效率；
接着进行以下步骤：
步骤1：选择一个预训练好的神经网络模型，设置一个压缩的峰值信噪比PSNR损失阈值，记为T，确定一个初始量化码率，记为D；
步骤2：应用任意基础方法来量化预训练神经网络的激活，得到具有D-bit定点表示的激活，即公式(5)中的x ij∈{0，1} M；
步骤3：在每个卷积层之前，通过求解公式(6)来训练信息比特瓶颈，并将其插入网络模型中以减少基础量化的损失；以相同PNSR损失阈值T为参考，每个比特瓶颈具有不同的稀疏性等级和不同数量的非零系数α j，等效于压缩后的有效码率小于D；
步骤4：通过反向传播训练过程，对具有信息瓶颈的深层神经网络的权值进行调优和量化；
步骤5：将训练好的神经网络的权值、比特系数矢量α和偏置加载到神经网络硬件系统，硬件系统部署完成。
可选的，包括CPU、直接存储器访问DMA、串口、总线、控制器、片上内存和神经网络加速器；
CPU控制整个计算进程的运行，串口用于接收数据，DMA用于调度输入数据的传输，总线用于传输控制命令和数据；
神经网络加速器包括计算阵列与输入帧缓存和输出帧缓存，用于实现数据缓冲和数据复用；输入帧缓存用于抓取每次卷积需要的数据，并将数据进行重新排列；
计算阵列用于实现神经网络内部的计算，每个计算阵列包括多个计算单元，每个计算单元实现稀疏量化神经网络中一个宏模块所有计算；
控制器用于控制各个模块的协调运作，并且通过总线与CPU通信，接收CPU的指令。
可选的，所述计算阵列包括多个计算单元，每个计算单元实现稀疏量化神经网络中一个宏模块所有计算。
可选的，所述一个宏模块所有计算包括批量归一化计算、比特瓶颈计算、卷积计算、激活计算和池化计算；比特瓶颈计算能够和卷积计算同时进行。
可选的，所述计算单元包括数据缓存模块、归一化和卷积模块、激活和池化模块；
所述数据缓存模块进行数据缓存后，在归一化和卷积模块中进行批量归一化、卷积位乘法阵列和偏置缓存；然后在激活和池化模块中进行激活和池化；然后输出激活。
本发明的有益效果在于：
1、首先，由于比特瓶颈压缩量化方法是凸的，因此它比常规的和其他数据驱动的激活压缩方法更强大，具有更强的鲁棒性。
2、其次，比特瓶颈操作非常灵活，可用于将浮点激活量化和压缩为任意大小的码率，从而使神经网络的硬件系统架构能够在不同应用的效率和精度之间进行权衡，减少硬件设计时的计算资源和存储消耗，以及可以降低整个系统架构的功耗。
3、最后，由于比特瓶颈可以将量化和压缩导致的激活信息丢失降至最低，因此具有比特瓶颈的深度神经网络硬件系统架构可以比使用其他激活压缩方法的硬件系统架构实现更高的分类精度。
本发明的其他优点、目标和特征在某种程度上将在随后的说明书中进行阐述，并且在某种程度上，基于对下文的考察研究对本领域技术人员而言将是显而易见的，或者可以从本发明的实践中得到教导。本发明的目标和其他优点可以通过下面的说明书来实现和获得。
附图说明
为了使本发明的目的、技术方案和优点更加清楚，下面将结合附图对本发明作优选的详细描述，其中：
图1为稀疏量化神经网络架构图；
图2为稀疏量化神经网络系统硬件架构图；
图3为基于稀疏量化编码神经网络系统架构使用方法；
图4为实施例示意图。
具体实施方式
以下通过特定的具体实例说明本发明的实施方式，本领域技术人员可由本说明书所揭露的内容轻易地了解本发明的其他优点与功效。本发明还可以通过另外不同的具体实施方式加以实施或应用，本说明书中的各项细节也可以基于不同观点与应用，在没有背离本发明的精神下进行各种修饰或改变。需要说明的是，以下实施例中所提供的图示仅以示意方式说明本发明的基本构想，在不冲突的情况下，以下实施例及实施例中的特征可以相互组合。
其中，附图仅用于示例性说明，表示的仅是示意图，而非实物图，不能理解为对本发明的限制；为了更好地说明本发明的实施例，附图某些部件会有省略、放大或缩小，并不代表实际产品的尺寸；对本领域技术人员来说，附图中某些公知结构及其说明可能省略是可以理解的。
本发明实施例的附图中相同或相似的标号对应相同或相似的部件；在本发明的描述中，需要理解的是，若有术语“上”、“下”、“左”、“右”、“前”、“后”等指示的方位或位置关系为基于附图所示的方位或位置关系，仅是为了便于描述本发明和简化描述，而不是指示或暗示所 指的装置或元件必须具有特定的方位、以特定的方位构造和操作，因此附图中描述位置关系的用语仅用于示例性说明，不能理解为对本发明的限制，对于本领域的普通技术人员而言，可以根据具体情况理解上述术语的具体含义。
最近，人们越来越有兴趣在深度学习研究中阐明信息理论，寻求最佳神经网络激活压缩。通过将深度神经网络视为一种有损数据压缩方法，人们打开了深度神经网络的黑匣子，并使用率失真理论工具分析其性能，该方法已广泛应用于通信领域。
1、提出的稀疏量化编码算法原理阐述
有损数据压缩的目标是在最大编码率的限制下实现最小的率失真。假设f：R D→I d是量化函数，它将输入的D-bit实值数据转换为d-bit整数。假设g(·)是指示输入数据码率的函数。根据率失真理论，典型的有损数据压缩方法尝试在给定最大比特数的情况下，最小化失真函数d(·)。
min d(x，f(x))st.g(x)≤η     (1)
实际上，由于其整数输出，量化函数f(x)是不可微且非凸的，这使其难以求解。与典型基于四舍五入的量化方法不同，其只提供了一个次优解决方案，本文尝试通过将公式(1)重新转化为稀疏编码问题来找到最佳解。
假定
Figure PCTCN2020113050-appb-000012
是与第i个样本相关的浮点激活矢量；标准的量化函数使用D个二值矢量x i1,x i2,…,x iD来近似
Figure PCTCN2020113050-appb-000013
记作
Figure PCTCN2020113050-appb-000014
x ij∈{0，1} M，表示x i的第j个比特位的M维二值矢量；假设实值激活
Figure PCTCN2020113050-appb-000015
被归一化到[0，2 D]的区域，则标准的舍入量化方法使用如下定点数据：
Figure PCTCN2020113050-appb-000016
其中每个比特分配有一个常数系数{2 0，...，2 D-1}，并且x ij的值通过四舍五入
Figure PCTCN2020113050-appb-000017
得到；
用变量α∈R D替换固定系数，如下：
Figure PCTCN2020113050-appb-000018
而且，该方法试图找到并估计与最重要比特相关的最优稀疏系数α。
受到公式(1)的率失真理论的启发，该方法尝试将N个训练样本上的最小化标准平方失真率定义为
Figure PCTCN2020113050-appb-000019
其中基于L0范数的约束限制了非零系数的总数，这将导致系数α稀疏。由于α中非零系数的数量等于定点表示中的比特位数，因此公式(4)的约束函数实际上会按照率失真理论的要求限制量化表示中的最大比特位数。
最近的研究表明，公式(4)等效于以下基于L1范数求解稀疏解的问题。
Figure PCTCN2020113050-appb-000020
比特瓶颈操作通过求解公式(5)，以确定稀疏的重要的比特位，并实现最小的率失真。在实践中，通常通过求解它的对偶形式来求解公式(5)。
Figure PCTCN2020113050-appb-000021
其中λ是用于控制最优的错误率和码率的超参数。公式(6)通常导致系数α的稀疏解，并且在推理阶段将与D-d零系数相关的激活比特位删除，计算效率可以显着提高。
2、基于稀疏量化编码神经网络架构与系统硬件架构设计
如公式(5)所示，信息比特瓶颈操作计算与压缩激活表示形式的每个比特位相关的最佳系数，以便在给出最大编码率的情况下可以实现最小失真率。本部分显示了信息瓶颈在深度神经网络中的工作方式。
比特瓶颈可以插入不同的网络的宏块中，典型的宏块包依次含批量归一化层，比特瓶颈层，卷积层，激活层，池化层(可选)。由于位瓶颈层将归一化的浮点激活转换为压缩的定点激活，卷积层可以将计算量大的浮点乘法替换为高效的定点整数乘法。定点整数乘法通常是以按位计算的方式实现，激活比特的数量与实际按位计算的次数成正比。因此，计算密集型的卷积计算的时间和内存效率能够被极大的提升。
如图1所示，硬件整体架构包括CPU、DMA(直接存储器访问)、串口、总线、片上内存和神经网络加速器6个部分组成。CPU控制整个计算进程的运行，串口用于接收数据，DMA用于调度输入数据的传输，总线用于传输控制命令和数据。神经网络计算加速器主要由计算阵列与输入帧缓存和输出帧缓存组成。输入输出帧缓存用于实现数据缓冲和数据复用。计算阵列用于实现神经网络内部的计算，多个阵列设计，以及每个阵列多个计算单元设计，目的都是实现神经网络的并行计算，增加计算效率。
如图2所示，每个计算单元实现稀疏量化神经网络中一个宏模块所有计算，包括批量归一化计算、比特瓶颈计算、卷积计算、激活计算和池化计算。输入块缓存用于抓取每次卷积需要的数据，并将数据进行重新排列。在硬件设计时，比特瓶颈计算能够和卷积计算同时进行。控制器用于控制各个模块的协调运作，并且通过总线与CPU通信，接收CPU的指令。
3、基于稀疏量化编码神经网络系统架构使用步骤
本专利提出稀疏量化神经网络系统利用比特瓶颈操作提取激活的数据比特间的稀疏性，以减少量化导致的激活完整性和分类准确性的损失。假设有人打算得到一个神经网络激活小于D-bit的稀疏量化神经网络系统，需要进行如下五个步骤。
步骤1：选择一个预训练好的神经网络模型，设置一个压缩的峰值信噪比(PSNR)损失 阈值，记为T，确定一个初始量化码率，记为D。
步骤2：应用任意基础方法来量化预训练神经网络的激活，得到具有D-bit定点表示的激活，即公式(5)中的x ij∈{0，1} M。
步骤3：在每个卷积层之前，可通过求解公式(6)来训练信息比特瓶颈，并将其插入网络模型中以减少基础量化的损失。以相同PNSR损失阈值T为参考，每个比特瓶颈都可能具有不同的稀疏性等级和不同数量的非零系数α j，这等效于压缩后的有效码率小于D。
步骤4：通过反向传播训练过程，对具有信息瓶颈的深层神经网络的权值进行了调优和量化。
步骤5：将训练好的神经网络的权值、比特系数矢量α和偏置加载到神经网络硬件系统。至此，整个硬件系统部署完成，能够投入到实际应用中。
有关使用方法的更多详细信息，请参见图3流程图。
如图4所示，该稀疏量化神经网络硬件系统能够应用在带识别功能的无人机平台上。摄像头拍摄图像，传给硬件系统进行智能信息处理，对视频图像信息进行检测识别，将结果传回无人机控制系统，控制系统可以将信息发给地面接收站，并且控制系统也可以根据识别结果控制电机，从而控制飞机的飞行姿态和路径。该系统可以实现森林火灾检测、特定目标追踪、城市智慧消防、输电线路巡检等功能。稀疏量化神经网络硬件系统能够加快信息处理，实现高速的识别，可以应用在需要进行高速图片信息处理的移动嵌入式系统上，比如无人机、自动驾驶汽车、智能机器人等。
最后说明的是，以上实施例仅用以说明本发明的技术方案而非限制，尽管参照较佳实施例对本发明进行了详细说明，本领域的普通技术人员应当理解，可以对本发明的技术方案进行修改或者等同替换，而不脱离本技术方案的宗旨和范围，其均应涵盖在本发明的权利要求范围当中。

Claims (5)

  1. 稀疏量化神经网络编码模式识别方法，其特征在于：该方法包括以下步骤：
    假设f：R D→I d是量化函数，将输入的D-bit实值数据转换为d-bit整数；假设g(·)是指示输入数据码率的函数；根据率失真理论，典型的有损数据压缩方法尝试在给定最大比特数的情况下，最小化失真函数d(·)：
    min d(x，f(x))st.g(x)≤η  (1)假定
    Figure PCTCN2020113050-appb-100001
    是与第i个样本相关的浮点激活矢量；标准的量化函数使用D个二值矢量x i1,x i2,…,x iD来近似
    Figure PCTCN2020113050-appb-100002
    记作
    Figure PCTCN2020113050-appb-100003
    x ij∈{0，1} M，表示x i的第j个比特位的M维二值矢量；假设实值激活
    Figure PCTCN2020113050-appb-100004
    被归一化到[0，2 D]的区域，则标准的舍入量化方法使用如下定点数据：
    Figure PCTCN2020113050-appb-100005
    其中每个比特分配有一个常数系数{2 0，...，2 D-1}，并且x ij的值通过四舍五入
    Figure PCTCN2020113050-appb-100006
    得到；
    用变量α∈R D替换固定系数，如下：
    Figure PCTCN2020113050-appb-100007
    找到并估计与最重要比特相关的最优稀疏系数α；
    将N个训练样本上的最小化标准平方失真率定义为
    Figure PCTCN2020113050-appb-100008
    公式(4)等效于以下基于L1范数求解稀疏解的问题：
    Figure PCTCN2020113050-appb-100009
    比特瓶颈操作通过求解公式(5)，以确定稀疏的重要的比特位，并实现最小的率失真；通过求解它的对偶形式来求解公式(5)：
    Figure PCTCN2020113050-appb-100010
    其中λ是用于控制最优的错误率和码率的超参数：公式(6)导致系数α的稀疏解，并且在推理阶段将与D-d零系数相关的激活比特位删除，提高计算效率；
    接着进行以下步骤：
    步骤1：选择一个预训练好的神经网络模型，设置一个压缩的峰值信噪比PSNR损失阈值，记为T，确定一个初始量化码率，记为D；
    步骤2：应用任意基础方法来量化预训练神经网络的激活，得到具有D-bit定点表示的激活，即公式(5)中的x ij∈{0，1} M；
    步骤3：在每个卷积层之前，通过求解公式(6)来训练信息比特瓶颈，并将其插入网络模型中以减少基础量化的损失；以相同PNSR损失阈值T为参考，每个比特瓶颈具有不同的稀 疏性等级和不同数量的非零系数α j，等效于压缩后的有效码率小于D；
    步骤4：通过反向传播训练过程，对具有信息瓶颈的深层神经网络的权值进行调优和量化；
    步骤5：将训练好的神经网络的权值、比特系数矢量α和偏置加载到神经网络硬件系统，硬件系统部署完成。
  2. 稀疏量化神经网络编码模式识别系统，其特征在于：包括CPU、直接存储器访问DMA、串口、总线、控制器、片上内存和神经网络加速器；
    CPU控制整个计算进程的运行，串口用于接收数据，DMA用于调度输入数据的传输，总线用于传输控制命令和数据；
    神经网络加速器包括计算阵列与输入帧缓存和输出帧缓存，用于实现数据缓冲和数据复用；输入帧缓存用于抓取每次卷积需要的数据，并将数据进行重新排列；
    计算阵列用于实现神经网络内部的计算；
    控制器用于控制各个模块的协调运作，并且通过总线与CPU通信，接收CPU的指令。
  3. 根据权利要求2所述的稀疏量化神经网络编码模式识别系统，其特征在于：所述计算阵列包括多个计算单元，每个计算单元实现稀疏量化神经网络中一个宏模块所有计算。
  4. 根据权利要求3所述的稀疏量化神经网络编码模式识别系统，其特征在于：所述一个宏模块所有计算包括批量归一化计算、比特瓶颈计算、卷积计算、激活计算和池化计算；比特瓶颈计算能够和卷积计算同时进行。
  5. 根据权利要求4所述的稀疏量化神经网络编码模式识别系统，其特征在于：所述计算单元包括数据缓存模块、归一化和卷积模块、激活和池化模块；
    所述数据缓存模块进行数据缓存后，在归一化和卷积模块中进行批量归一化、卷积位乘法阵列和偏置缓存；然后在激活和池化模块中进行激活和池化；然后输出激活。
PCT/CN2020/113050 2020-03-05 2020-09-02 稀疏量化神经网络编码模式识别方法与系统 WO2021174790A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010148404.0A CN111460905A (zh) 2020-03-05 2020-03-05 稀疏量化神经网络编码模式识别方法与系统
CN202010148404.0 2020-03-05

Publications (1)

Publication Number Publication Date
WO2021174790A1 true WO2021174790A1 (zh) 2021-09-10

Family

ID=71685575

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/113050 WO2021174790A1 (zh) 2020-03-05 2020-09-02 稀疏量化神经网络编码模式识别方法与系统

Country Status (2)

Country Link
CN (1) CN111460905A (zh)
WO (1) WO2021174790A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114997386A (zh) * 2022-06-29 2022-09-02 桂林电子科技大学 一种基于多fpga异构架构的cnn神经网络加速设计方法
CN116151340A (zh) * 2022-12-26 2023-05-23 辉羲智能科技(上海)有限公司 并行随机计算神经网络系统及其硬件压缩方法、系统
CN116299247A (zh) * 2023-05-19 2023-06-23 中国科学院精密测量科学与技术创新研究院 一种基于稀疏卷积神经网络的InSAR大气校正方法

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111460905A (zh) * 2020-03-05 2020-07-28 重庆大学 稀疏量化神经网络编码模式识别方法与系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109361404A (zh) * 2018-09-28 2019-02-19 华南理工大学 一种基于半监督深度学习网络的ldpc译码系统及译码方法
CN109635936A (zh) * 2018-12-29 2019-04-16 杭州国芯科技股份有限公司 一种基于重训练的神经网络剪枝量化方法
US20190392323A1 (en) * 2018-06-22 2019-12-26 Moffett AI, Inc. Neural network acceleration and embedding compression systems and methods with activation sparsification
CN110852428A (zh) * 2019-09-08 2020-02-28 天津大学 基于fpga的神经网络加速方法和加速器
CN111460905A (zh) * 2020-03-05 2020-07-28 重庆大学 稀疏量化神经网络编码模式识别方法与系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190392323A1 (en) * 2018-06-22 2019-12-26 Moffett AI, Inc. Neural network acceleration and embedding compression systems and methods with activation sparsification
CN109361404A (zh) * 2018-09-28 2019-02-19 华南理工大学 一种基于半监督深度学习网络的ldpc译码系统及译码方法
CN109635936A (zh) * 2018-12-29 2019-04-16 杭州国芯科技股份有限公司 一种基于重训练的神经网络剪枝量化方法
CN110852428A (zh) * 2019-09-08 2020-02-28 天津大学 基于fpga的神经网络加速方法和加速器
CN111460905A (zh) * 2020-03-05 2020-07-28 重庆大学 稀疏量化神经网络编码模式识别方法与系统

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
XU JIAHUI: "Research on Neural Network Compression Techniques: Model Pruning", INFORMATION & COMMUNICATIONS, no. 204, 31 December 2019 (2019-12-31), pages 165 - 167, XP055842096, ISSN: 1673-1131 *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114997386A (zh) * 2022-06-29 2022-09-02 桂林电子科技大学 一种基于多fpga异构架构的cnn神经网络加速设计方法
CN114997386B (zh) * 2022-06-29 2024-03-22 桂林电子科技大学 一种基于多fpga异构架构的cnn神经网络加速设计方法
CN116151340A (zh) * 2022-12-26 2023-05-23 辉羲智能科技(上海)有限公司 并行随机计算神经网络系统及其硬件压缩方法、系统
CN116151340B (zh) * 2022-12-26 2023-09-01 辉羲智能科技(上海)有限公司 并行随机计算神经网络系统及其硬件压缩方法、系统
CN116299247A (zh) * 2023-05-19 2023-06-23 中国科学院精密测量科学与技术创新研究院 一种基于稀疏卷积神经网络的InSAR大气校正方法
CN116299247B (zh) * 2023-05-19 2023-08-04 中国科学院精密测量科学与技术创新研究院 一种基于稀疏卷积神经网络的InSAR大气校正方法

Also Published As

Publication number Publication date
CN111460905A (zh) 2020-07-28

Similar Documents

Publication Publication Date Title
WO2021174790A1 (zh) 稀疏量化神经网络编码模式识别方法与系统
Tung et al. Clip-q: Deep network compression learning by in-parallel pruning-quantization
CN110084281B (zh) 图像生成方法、神经网络的压缩方法及相关装置、设备
CN111445026A (zh) 面向边缘智能应用的深度神经网络多路径推理加速方法
CN113326930B (zh) 数据处理方法、神经网络的训练方法及相关装置、设备
Passalis et al. Learning bag-of-features pooling for deep convolutional neural networks
He et al. Multi-task zipping via layer-wise neuron sharing
CN108764317B (zh) 一种基于多路特征加权的残差卷积神经网络图像分类方法
KR20180004898A (ko) 딥러닝 기반의 이미지 처리 기술 및 그 방법
US20220414432A1 (en) Method and system for splitting and bit-width assignment of deep learning models for inference on distributed systems
KR20220137076A (ko) 이미지 프로세싱 방법 및 관련된 디바이스
WO2023231794A1 (zh) 一种神经网络参数量化方法和装置
CN112766467B (zh) 基于卷积神经网络模型的图像识别方法
CN112633477A (zh) 一种基于现场可编程阵列的量化神经网络加速方法
Mehta et al. Deepsplit: Dynamic splitting of collaborative edge-cloud convolutional neural networks
CN113610192A (zh) 一种基于连续性剪枝的神经网络轻量化方法及系统
CN109886160A (zh) 一种非限定条件下的人脸识别方法
Valdenegro-Toro Real-time convolutional networks for sonar image classification in low-power embedded systems
US11881020B1 (en) Method for small object detection in drone scene based on deep learning
Li et al. Attention-based feature compression for cnn inference offloading in edge computing
CN109542513B (zh) 一种卷积神经网络指令数据存储系统及方法
US20230143985A1 (en) Data feature extraction method and related apparatus
Ling et al. TaiJiNet: Towards partial binarized convolutional neural network for embedded systems
CN115546556A (zh) 用于图像分类的脉冲神经网络的训练方法
WO2022022176A1 (zh) 一种图像处理方法以及相关设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20922949

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20922949

Country of ref document: EP

Kind code of ref document: A1