WO2021004366A1 - 基于结构化剪枝和低比特量化的神经网络加速器及方法 - Google Patents

基于结构化剪枝和低比特量化的神经网络加速器及方法 Download PDF

Info

Publication number
WO2021004366A1
WO2021004366A1 PCT/CN2020/099891 CN2020099891W WO2021004366A1 WO 2021004366 A1 WO2021004366 A1 WO 2021004366A1 CN 2020099891 W CN2020099891 W CN 2020099891W WO 2021004366 A1 WO2021004366 A1 WO 2021004366A1
Authority
WO
WIPO (PCT)
Prior art keywords
excitation
weight
input
buffer
output
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/099891
Other languages
English (en)
French (fr)
Inventor
黄科杰
朱超阳
沈海斌
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang University ZJU
Original Assignee
Zhejiang University ZJU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang University ZJU filed Critical Zhejiang University ZJU
Publication of WO2021004366A1 publication Critical patent/WO2021004366A1/zh
Priority to US17/485,645 priority Critical patent/US12271820B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/40—Transformation of program code
    • G06F8/41—Compilation
    • G06F8/42—Syntactic analysis
    • G06F8/427—Parsing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0495—Quantised networks; Sparse networks; Compressed networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections

Definitions

  • the present invention relates to the technical field of sparse convolutional neural network accelerator design, in particular to a neural network accelerator and method based on structured pruning and low-bit quantization.
  • Convolutional neural networks are rapidly playing a role in a wide range of applications such as computer vision, robot control, video analysis, speech recognition, natural language processing, targeted advertising recommendation and web search.
  • the convolutional neural network can provide excellent results, it has a large amount of calculation parameters and high computational complexity. Due to the limitations of traditional processor performance and power consumption, large-scale convolutional neural networks are difficult to implement on embedded or terminal devices. Therefore, under the premise of ensuring the prediction accuracy of the convolutional neural network, the compression network model and the design of a dedicated hardware accelerator have become a hot research topic.
  • the methods of network model compression include: pruning, quantization, and block loop representation. After pruning, the neural network becomes sparse, and the convolution kernel is compressed and stored. The weights that are pruned in different convolution kernels may be different, resulting in irregularities in the network's reasoning operation.
  • the accelerator architecture based on sparse neural network needs to overcome this irregularity in order to use pruning to further improve accelerator performance and energy efficiency.
  • the first method performs Cartesian product operation on the input excitation vector and weight vector, and calculates the coordinates of the output excitation in the output characteristic map at the same time, and then uses the calculated coordinate control matrix
  • the switch stores the result of the Cartesian product operation in the corresponding position
  • the second method is to read the compression weight first, then restore the position of the compression weight value in the original weight matrix, and then AND the input excitation vector and the weight vector to filter out
  • the input stimulus and the weight value of the effective calculation are calculated, and the effective calculation is only when the two are non-zero values at the same time.
  • the effective calculation means that when the input excitation and weight are both non-zero values, the process of multiplying them is called effective calculation.
  • the problem with the first method is that it needs to calculate the coordinates of the output excitation, which increases the calculation power consumption.
  • the Cartesian product operation on the convolutional layer will cause the output to have a hash conflict, and the Cartesian product operation on the fully connected layer is only valid for part of the output.
  • the problem with the second method is that each input stimulus in the convolution operation needs to be multiplied by the weights of different three-dimensional convolution kernels. Therefore, the input stimulus and different three-dimensional convolution kernels must be filtered. The selection process will bring a lot of power consumption. And the high fan-out of the output of the screening unit will increase the difficulty of design routing and timing optimization. Both of these methods have a negative impact on convolutional neural network accelerators that use pruning to improve performance and energy efficiency.
  • the purpose of the present invention is to provide a neural network accelerator and method based on structured pruning and low-bit quantization, so as to improve the accelerator's acceleration effect and reduce the accelerator's power consumption.
  • the present invention provides a neural network accelerator based on structured pruning and low-bit quantization, and the neural network accelerator includes:
  • Main controller excitation selection unit, scalable computing array, multi-function processing unit, DMA, DRAM, and buffer; the main controller is connected to the excitation selection unit, the scalable computing array and the DMA, respectively, The DMA is respectively connected to the buffer and the DRAM, the buffer is connected to the multi-function processing unit and the excitation selection unit, and the expandable computing array is connected to the excitation selection unit and the buffer. ⁇ ; Connect;
  • the main controller is used to parse the instruction set, generate the first storage address of the input excitation, the weight, the storage address of the output excitation, and the control signal;
  • the buffer is used to store the input excitation, the output excitation and the weight index
  • the excitation selection unit is configured to filter the input excitation input from the buffer according to the control signal generated by the main controller, and transmit the input excitation to the expandable computing array;
  • the scalable computing array includes: N ⁇ M processing units PE; N and M are the rows and columns of the processing units respectively; each PE stores part of the weight of the neural network, and the second storage address of the weight is determined according to the received weight index According to the second storage address of the weight, the weight corresponding to the input stimulus is obtained by the stimulus selection unit, and it is judged whether the received input stimulus is zero to control the reading of the weight and the switching of the multiplier in the PE;
  • the control signal generated by the controller determines whether the currently calculated output stimulus has completed the convolution of the input stimulus and weight of an input channel, and if so, the output stimulus is stored in the excitation selection unit through the excitation selection unit.
  • the output in the buffer stimulates the buffer;
  • the multifunctional processing unit is used to complete network pooling, activation and normalization operations
  • the DMA is used to read the weight stored in the DRAM according to the first storage address of the weight, read the output stimulus stored in the buffer according to the storage address of the current output stimulus, and combine all The output stimulus is sent to the DRAM for storage.
  • the main controller includes:
  • An instruction set decoder an address generator, a control vector generator and seven counters; the instruction set decoder is respectively connected to the input end of each counter, and the output end of each counter is respectively connected to the address generator Connected to the control vector generator, the address generator is connected to the DMA, the control vector generator is respectively connected to the excitation selection unit and the scalable computing array, seven counters are cascaded in sequence, the previous stage The overflow signal is used as the next level increase signal;
  • the instruction set decoder is used to decode the instruction set, generate the upper limit and lower limit of the judgment of each of the counters, and control the cycle count output of seven counters according to the upper limit and the lower limit of the judgment ⁇ kw, kh, ow, oh, ic, oc, l ⁇ ;
  • the address generator is used to generate the first storage address of the input excitation and weight required for the next clock cycle calculation according to ⁇ kw, kh, ow, oh, ic, oc, l ⁇ output by the seven counters, and the current output excitation And transfer to the DMA;
  • the control vector generator is used to generate a control signal ctrl according to ⁇ kw, kh, ow, oh, ic, oc, l ⁇ output by seven counters, and then control the excitation selection unit and the control signal ctrl according to the control signal ctrl Expand the computing array.
  • the excitation selection unit includes:
  • Input excitation register, index decoder and selector the selector is connected to the input excitation register and the index decoder respectively;
  • the input excitation register is used to read in and output input excitation according to the control signal generated by the main controller
  • the index decoder is used to decode the weight index to generate a jump variable
  • the selector is used to filter the input stimulus according to the jump variable and send it to the scalable computing array.
  • the buffer includes:
  • Input excitation buffer, output excitation buffer and weight index buffer said weight index buffer and said input excitation buffer are respectively connected to said DMA, said weight index buffer and said input excitation buffer are respectively connected to The excitation selection unit is connected; the output excitation buffer is respectively connected with the DMA, the multi-function processing unit and the expandable computing array;
  • the input excitation buffer is used to store the input excitation
  • the output excitation buffer is used to store the output excitation
  • the weight index buffer is used to store the weight index.
  • the present invention also provides a neural network acceleration method based on structured pruning and low-bit quantization, and the method includes:
  • S01 Perform structured pruning and low-bit quantization on the neural network to obtain a compression model and an index set that characterizes the compression weight position;
  • S02 Analyze the compression model to obtain network layer parameters and operand bit widths, and determine the size of the scalable computing array in combination with the given hardware resources;
  • S03 Call the main controller with IP core, excitation selection unit and processing unit to form a neural network accelerator according to the size of the scalable computing array;
  • the structural pruning of the neural network specifically includes:
  • the weights of the same positions of different three-dimensional convolution kernels in the convolutional layer are cut off at the same time to obtain different convolution kernels with the same index; the input feature map and a convolution kernel index are filtered by the excitation selection unit to obtain all valid Calculated input excitation and weight index; cut some rows of the weight matrix in the fully connected layer.
  • mapping relationship between the compressed network and hardware resources specifically includes:
  • each row of the processing unit PE of the scalable computing array shares the same weight, and each column of the processing unit PE shares the same input excitation, so The entire scalable computing array shares N input stimuli and one weight index; in the fully connected layer calculation, each row of the scalable computing array shares the same input stimulus, and the weight of each processing unit PE is different, but the structure Pruning makes the weight indexes of the processing units PE of the same column the same, so the scalable computing array shares N input excitations and N weight indexes.
  • w represents the weight vector
  • x represents the input excitation
  • x i represents the i-th input excitation
  • w i represents the i-th weight in the weight vector
  • w′ i is the quantized value of w i
  • n represents the part that needs to be accumulated
  • Bitshift(a,b) means to move a by b bits
  • the moving direction is determined by the sign of b
  • the present invention discloses the following technical effects:
  • the present invention makes full use of the data reusability in the process of neural network inference operations, reduces the power consumption of input incentives and weights for filtering effective calculations, and relieves incentives
  • the high transmission bandwidth pressure between the selection unit and the scalable computing array combined with low-bit quantization technology, reduces the number of weight parameters and storage bit width, and further improves the throughput and energy efficiency of the convolutional neural network accelerator.
  • Figure 1 is a top-level architecture diagram of a neural network accelerator according to an embodiment of the present invention
  • FIG. 2 is a structural diagram of a main controller according to an embodiment of the present invention.
  • FIG. 3 is a structural diagram of an incentive selection unit according to an embodiment of the present invention.
  • Fig. 4 is a working flowchart of an accelerator according to an embodiment of the present invention.
  • FIG. 5 is a schematic diagram of structured pruning according to an embodiment of the present invention.
  • FIG. 6 is a diagram of compressed weights and indexes according to an embodiment of the present invention.
  • FIG. 7 is a diagram of a convolution operation mode according to an embodiment of the present invention.
  • FIG. 8 is a diagram of a fully connected layer operation mode according to an embodiment of the present invention.
  • the purpose of the present invention is to provide a neural network accelerator and method based on structured pruning and low-bit quantization, so as to improve the acceleration effect of the accelerator and reduce the power consumption of the accelerator.
  • the present invention discloses a neural network accelerator based on structured pruning and low-bit quantization.
  • the neural network accelerator includes:
  • Main controller excitation selection unit, scalable computing array, multi-function processing unit, DMA, DRAM, and buffer; the main controller is connected to the excitation selection unit, the scalable computing array and the DMA, respectively, The DMA is respectively connected to the buffer and the DRAM, the buffer is connected to the multi-function processing unit and the excitation selection unit, and the expandable computing array is connected to the excitation selection unit and the buffer. ⁇ ; Connect;
  • the main controller is used to parse the instruction set, generate the first storage address of the input excitation, the weight, the storage address of the output excitation, and the control signal;
  • the buffer is used to store the input excitation, the output excitation and the weight index
  • the excitation selection unit is configured to filter the input excitation input from the buffer according to the control signal generated by the main controller, and transmit the input excitation to the expandable computing array;
  • the scalable computing array includes: N ⁇ M processing units PE; N and M are the rows and columns of the processing units respectively; each PE stores part of the weight of the neural network, and the second storage address of the weight is determined according to the received weight index According to the second storage address of the weight, the weight corresponding to the input stimulus is obtained by the stimulus selection unit, and it is judged whether the received input stimulus is zero to control the reading of the weight and the switching of the multiplier in the PE;
  • the control signal generated by the controller determines whether the currently calculated output stimulus has completed the convolution of the input stimulus and weight of an input channel, and if so, the output stimulus is stored in the excitation selection unit through the excitation selection unit.
  • the output in the buffer stimulates the buffer;
  • the multifunctional processing unit is used to complete network pooling, activation and normalization operations
  • the DMA is used to read the weight stored in the DRAM according to the first storage address of the weight, read the output stimulus stored in the buffer according to the storage address of the current output stimulus, and combine all The output stimulus is sent to the DRAM for storage.
  • the main controller includes an instruction set decoder, an address generator, a control vector generator and seven counters; the instruction set decoder is connected to the input terminals of the counters, respectively, The output end of each of the counters is respectively connected to an address generator and the control vector generator, the address generator is connected to DMA, and the control vector generator is connected to the excitation selection unit and the scalable computing array respectively connection.
  • the overflow signal of the previous level is used as the next increase signal inc, and the seven counters output ⁇ kw,kh,ow,oh,ic,oc,l ⁇ , ⁇ kw,kh,ow, oh, ic, oc, l ⁇ represents the acceleration process, where kw represents the number of columns of the currently calculated weight, kh represents the number of rows of the currently calculated weight, ow represents the number of columns of the currently calculated output excitation, and oh represents the current calculated output
  • the number of excitation rows, ic represents the number of input excitation channels currently calculated, oc represents the number of output excitation channels currently calculated, and l represents the number of input excitation batches currently calculated;
  • the instruction set decoder is used to decode the instruction set , Generate the upper limit and lower limit of the judgment of each of the counters, and control seven counters to cycle counting output ⁇ kw, kh, ow, oh, ic, oc, l ⁇ according to
  • the address generator is used to generate the first storage address of the input excitation and weight required for the next clock cycle calculation according to ⁇ kw, kh, ow, oh, ic, oc, l ⁇ output by the seven counters, and the current output excitation The storage address and transfer to the DMA.
  • the control vector generator is used to generate a control signal ctrl according to ⁇ kw, kh, ow, oh, ic, oc, l ⁇ output by seven counters, and then control the excitation selection unit and the scalable calculation according to the control signal ctrl Array.
  • the excitation selection unit includes: an input excitation register, an index decoder, and a selector; the selector is connected to the input excitation register and the index decoder, respectively.
  • the input excitation register completes the read-in and output of the input excitation according to the control signal of the main controller; the index decoder decodes the weight index to generate a jump variable jump; the selector filters the input excitation according to the jump variable jump signal Specifically, the calculation process corresponding to the zero-value weight is skipped according to the index set, and the input excitation and the weight index corresponding to the non-zero weight are selected and transmitted to the scalable computing array.
  • the scalable computing array includes: N ⁇ M processing units PE; N and M are the rows and columns of the processing units respectively; it can be divided into N groups, each group of PEs calculates different output channels in parallel, and M PEs in each group are parallel Calculate different output excitations in the same output channel.
  • each PE stores part of the weight of the neural network, and determines the second storage address of the weight according to the received weight index.
  • the read address is the address of the weight in the on-chip buffer.
  • the second storage address of the weight obtains the weight corresponding to the input stimulus, and determines whether the received input stimulus is zero to control the reading of the weight and the switching of the multiplier in the PE; specifically, if the input stimulus is not zero, read the corresponding Weight, and enable the multiplier, perform the multiplication calculation of the input excitation and the weight, and obtain a partial sum; if the input excitation is zero, the corresponding weight is no longer read, thereby reducing the memory access power consumption, and turning off the multiplier Reduce computational power consumption, that is, the sum of this part is zero.
  • the partial sum and the previously obtained partial sum are accumulated to obtain a new partial sum, that is, the current output excitation.
  • the control signal generated by the main controller it is judged whether the current output stimulus has completed the convolution of the input stimulus and weight of an input channel, and if so, the output stimulus is stored in the buffer through the stimulus selection unit The output in the stimulus buffer is otherwise stored in each PE.
  • the multi-function processing unit is used to complete network pooling, activation and normalization operations.
  • the output excitation channel generated by the scalable computing array in Figure 1 is first stored in the output excitation buffer, and then the multi-function processing unit performs network Pooling, activation and normalization operations are processed, and the final output excitation is returned to the output excitation buffer after the processing is completed.
  • DRAM is used to store weights and output excitations.
  • the neural network has a huge number of parameters. For large networks such as VGG-16, even after model compression, it is difficult to completely store the weights in the buffer. At the same time, the output feature map size of some layers of the network is huge, which also requires DRAM.
  • the buffer includes an input excitation buffer, an output excitation buffer and an index buffer; the weight index buffer and the input excitation buffer are respectively connected to the DMA, the weight index buffer and the input excitation buffer
  • the buffer is respectively connected with the excitation selection unit; the output excitation buffer is connected with the DMA, the multi-function processing unit and the expandable computing array; the input excitation buffer is used to store the input excitation
  • the output excitation buffer is used to store the output excitation;
  • the weight index buffer is used to store the weight index.
  • the network is divided into blocks for calculation, and the data stored in the buffer needs to be reused to reduce repeated reading of the DRAM and reduce the reading power consumption.
  • Fig. 4 is a working flow chart of an accelerator according to an embodiment of the present invention. As shown in Fig. 4, the present invention discloses a neural network acceleration method based on structured pruning and low-bit quantization. The method includes:
  • S01 Perform structured pruning and low-bit quantization on the neural network to obtain a compression model and an index set that characterizes the compression weight position;
  • S02 Analyze the compression model to obtain the network layer parameters and operand bit width, and combine the given hardware resources to determine the size of the scalable computing array;
  • the network layer parameters include bit width, weight, and excitation;
  • the hardware resources include the size of on-chip memory and the number of DSPs;
  • S03 Call the main controller with IP core, excitation selection unit and processing unit to form a neural network accelerator according to the size of the scalable computing array;
  • Fig. 5 is a schematic diagram of structured pruning according to an embodiment of the present invention
  • Fig. 5 (a) a schematic diagram of structured pruning of a convolution kernel
  • Fig. 5 (b) a schematic diagram of structured pruning of a weight matrix, as shown in Fig. 5
  • Structured pruning of neural networks includes:
  • the weights of the same positions of different three-dimensional convolution kernels in the convolutional layer are cut off at the same time to obtain different convolution kernels with the same index; the input feature map and a convolution kernel index are filtered by the excitation selection unit to obtain all valid Calculated input excitation and weight index; cut some rows of the weight matrix in the fully connected layer, and the corresponding column elements of the input excitation row vector do not need to be calculated, so they are not stored.
  • mapping relationship between the compressed network and hardware resources specifically includes:
  • each row of the processing unit PE of the scalable computing array shares the same weight, and each column of the processing unit PE shares the same input excitation, so The entire scalable computing array shares N input stimuli and one weight index; in the fully connected layer calculation, each row of the scalable computing array shares the same input stimulus, and the weight of each processing unit PE is different, but the structure Pruning makes the weight indexes of the processing units PE of the same column the same, so the scalable computing array shares N input excitations and N weight indexes.
  • w represents the weight vector
  • x represents the input excitation
  • x i represents the i-th input excitation
  • w i represents the i-th weight in the weight vector
  • w′ i is the quantized value of w i
  • n represents the part that needs to be accumulated
  • Bitshift(a,b) means to move a by b bits
  • the moving direction is determined by the sign of b
  • different networks have different quantization bit width requirements.
  • This embodiment performs structured pruning on the neural network. After pruning, the weights of the sparse network are compressed and then stored in rows. At the same time, it is necessary to store the index of the weight position and the row pointer. As shown in Figure 6, the index represents two non-zero weights. The number of "0"s in between, the index of the first non-zero weight in each row is equal to the number of "0"s before this non-zero weight in the row, and the row pointer indicates the total number of non-zero weights in each row.
  • the convolutional layer is calculated in blocks, and the loop of the convolution operation is expanded, so as to make full use of the parallelism of the scalable computing array and improve the reuse of data.
  • the pseudo code for block calculation and loop unrolling of the convolutional layer is as follows:
  • H is the height of the input feature map
  • W is the width of the input feature map
  • C is the channel number of the input feature map
  • R is the height and width of the weight
  • F is the channel number of the output feature map
  • U is the output feature map Height
  • V is the width of the output feature map
  • kw is the number of columns of the currently calculated weight
  • kh is the number of rows of the currently calculated weight
  • ow is the number of columns of the currently calculated output excitation
  • oh is the row of the currently calculated output excitation Number
  • ic represents the number of input excitation channels currently calculated
  • oc represents the number of currently calculated output excitation channels
  • l represents the number of batches of currently calculated input excitation
  • Toc is the number of channels of the output feature map
  • Tow is the number of output feature maps Number of columns
  • Y is output excitation
  • W is weight
  • X is input excitation.
  • loop expansion is used to make the expandable computing array calculate the M output excitations of the N channels of the output feature map at the same time, which can expand the different rows of the computing array to calculate the different channels of the output feature map, and different processing units in the same row calculate the output.
  • the weights input to different rows come from the same position of different 3D convolution kernels. After structured pruning, different 3D convolution kernels in the same layer have the same index, so the entire scalable computing array shares the same weight index.
  • mapping relationship is as follows:
  • F is the number of channels of the output feature map
  • N is the total number of channels for calculating the output feature map at the same time
  • V is the width of the output feature map
  • M is the total number of output excitations for calculating the output feature map at the same time
  • oc max is the current The maximum number of channels of the calculated output excitation
  • ow max is the maximum number of columns of the currently calculated output excitation.
  • Figure 7 describes the input feature map X(10,10,1) and the four-dimensional convolution kernel W(3,3,1,16)
  • the convolution process is the operation mode of a row of the scalable computing array, assuming that each row of the scalable computing array has only 4 processing units PE.
  • Figure 7 (b) is the convolution operation mode of the structured pruning network. Compared with Figure 7 (a), Figure 7 (b) judges whether the weight is zero according to the index. When the weight is zero, the corresponding calculation can be Skip, the judgment process is implemented in the excitation selection unit.
  • the operation of the fully connected layer is shown in Figure 8. Some rows of the weight matrix are cut off, so the corresponding column elements of the input excitation vector do not participate in the calculation.
  • the weight index of the next layer is used to filter the output excitation of the current layer, as shown in Figure 8 (b), and a pipeline is constructed to hide the filtering process in the current layer.
  • Output excitation Between calculation and the next layer of input excitation reading, as shown in Figure 8 (c).
  • the excitation selection unit implements the selection of input excitation.
  • the excitation selection unit reads the weight index jump according to the control information of the main controller.
  • the input excitation vector Data_in contains 2M input excitations, and the selector sets Data_in>>jump to skip the calculation when the weight is 0.
  • the main controller sends a push signal to the excitation selection unit and stores the new input excitation in the excitation register; when the weight of a row is calculated, the main controller sends a pop signal to the excitation unit and outputs a new The input excitation vector Data_in.
  • the selector reads the output stimulus of the current layer and the index of the next layer, and filters out the output stimulus corresponding to the non-zero weight.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Neurology (AREA)
  • Complex Calculations (AREA)

Abstract

一种基于结构化剪枝和低比特量化的神经网络加速器及方法，该神经网络加速器包括：主控制器、激励选择单元、可扩展计算阵列、多功能处理单元、DMA、DRAM和缓存器。通过结构化剪枝和可扩展计算阵列上的数据共享，充分利用了神经网络推理运算过程中的数据可重利用性，降低了筛选有效计算的输入激励和权重的功耗，缓解了激励选择单元和计算阵列之间的高传输带宽压力，结合低比特量化技术，减小了权重参数的数量和存储位宽，进一步提升了卷积神经网络加速器的吞吐率和能效。

Description

基于结构化剪枝和低比特量化的神经网络加速器及方法
本申请要求于2019年7月8日提交中国专利局、申请号为201910609993.5、发明名称为“一种基于结构化剪枝和低比特量化的神经网络加速器”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明涉及稀疏卷积神经网络加速器设计技术领域，特别是涉及一种基于结构化剪枝和低比特量化的神经网络加速器及方法。
背景技术
卷积神经网络在计算机视觉、机器人控制、视频分析、语音识别、自然语言处理、定向广告推荐和网络搜索等广泛应用中迅速发挥作用。虽然卷积神经网络能提供出色的效果，但是它运算参数量大，计算复杂度高。由于传统处理器性能和功耗的限制，使得大规模的卷积神经网络很难在嵌入式或终端设备上进行实现。因此，在保证卷积神经网络预测准确度的前提下，压缩网络模型和设计专用的硬件加速器成为了目前研究的热点。
网络模型压缩的方法包括：剪枝、量化、块循环表示等。剪枝后，神经网络变得稀疏，卷积核被压缩存储，不同卷积核中被剪去的权重可能不同，导致网络的推理运算存在不规则性。基于稀疏神经网络的加速器架构需要克服这种不规则性，以利用剪枝进一步改善加速器性能和能效。目前克服上述不规则性的方法主要有两种：第一种方法将输入激励向量和权重向量进行笛卡尔积运算，同时计算输出激励在输出特征图中的坐标，再用计算所得的坐标控制矩阵开关将笛卡尔积运算结果存到对应的位置；第二种方法是先读取压缩权重，再恢复压缩权重值在原权重矩阵中的位置，然后将输入激励向量和权重向量进行相与，筛选出有效计算的输入激励和权重值进行运算，只有两种同时为非零值时才是有效计算。所述有效计算指输入激励和权重均为非零值时，它们相乘的过程称为有效计算。第一种方法的问题是需要额外计算输出激励的坐标，增加了计算功耗。同时对卷积层进行笛卡尔积运算会使输出存在哈希冲突，而对全连接层进行笛卡尔积运算只有部分输出有效。第二种方法的问题是卷积运算中每个输入激励需 要和不同三维卷积核中权重相乘，因此输入激励和不同三维卷积核都要进行筛选，筛选过程会带来大量功耗，并且筛选单元输出的高扇出会增加设计布线和时序优化的难度。这两种方法都对卷积神经网络加速器利用剪枝提升性能和能效带来了负面影响。
发明内容
基于此，本发明的目的是提供一种基于结构化剪枝和低比特量化的神经网络加速器及方法，以提升加速器加速效果，降低加速器功耗。
为实现上述目的，本发明提供了一种基于结构化剪枝和低比特量化的神经网络加速器，所述神经网络加速器包括：
主控制器、激励选择单元、可扩展计算阵列、多功能处理单元、DMA、DRAM和缓存器；所述主控制器分别与所述激励选择单元、所述可扩展计算阵列和DMA连接，所述DMA分别与所述缓存器和所述DRAM连接，所述缓存器分别与所述多功能处理单元和所述激励选择单元连接，所述可扩展计算阵列分别与所述激励选择单元和所述缓存器连接；
所述主控制器用于解析指令集，产生输入激励、权重的第一存储地址和输出激励的存储地址以及控制信号；
所述缓存器用于存储所述输入激励、所述输出激励和权重索引；
所述激励选择单元用于根据所述主控制器生成的所述控制信号筛选从所述缓存器输入的输入激励，并传输给所述可扩展计算阵列；
所述可扩展计算阵列包括：N×M个处理单元PE；N、M分别为处理单元的行和列；每个PE存储神经网络的部分权重，根据接收的权重索引确定权重的第二存储地址，根据所述权重的第二存储地址通过所述激励选择单元获取输入激励对应的权重，并判断接收的输入激励是否为零从而控制权重的读取和PE内乘法器的开关；根据所述主控制器生成的所述控制信号判断当前计算的所述输出激励是否完成了一个输入通道的输入激励和权重的卷积，如果是，则将所述输出激励通过所述激励选择单元存储到所述缓存器中的输出激励缓存器；
所述多功能处理单元用于完成网络的池化、激活和归一化运算；
所述DMA用于根据所述权重的第一存储地址读取所述DRAM中存储的权重，根据所述当前输出激励的存储地址读取所述缓存器中存储的所 述输出激励，并将所述输出激励发送至所述DRAM进行存储。
可选地，所述主控制器包括：
一个指令集解码器、一个地址生成器、一个控制向量生成器和七个计数器；所述指令集解码器分别与各所述计数器的输入端连接，各所述计数器的输出端分别与地址生成器和所述控制向量生成器连接，所述地址生成器与DMA连接，所述控制向量生成器分别与所述激励选择单元和所述可扩展计算阵列连接，七个计数器依次级联，前一级上溢信号作为下一级增信号；
所述指令集解码器用于对指令集进行解码，生成各所述计数器的判断上限和下限，根据判断上限和下限控制七个计数器循环计数输出{kw,kh,ow,oh,ic,oc,l}；
所述地址生成器用于根据七个计数器输出的{kw,kh,ow,oh,ic,oc,l}生成下一个时钟周期计算所需的输入激励和权重的第一存储地址，以及当前输出激励的存储地址，并传输给所述DMA；
所述控制向量生成器用于根据七个计数器输出的{kw,kh,ow,oh,ic,oc,l}生成控制信号ctrl，进而根据所述控制信号ctrl控制所述激励选择单元和所述可扩展计算阵列。
可选地，所述激励选择单元包括：
输入激励寄存器、索引解码器和选择器；所述选择器分别与所述输入激励寄存器和所述索引解码器连接；
所述输入激励寄存器用于根据所述主控制器生成的控制信号读入和输出输入激励；
所述索引解码器用于将权重索引解码，生成跳变量；
所述选择器用于根据所述跳变量筛选输入激励，并输送至所述可扩展计算阵列。
可选地，所述缓存器包括：
输入激励缓存器、输出激励缓存器和权重索引缓存器；所述权重索引缓存器和所述输入激励缓存器分别与所述DMA连接，所述权重索引缓存器和所述输入激励缓存器分别与所述激励选择单元连接；所述输出激励缓存器分别与所述DMA、所述多功能处理单元和所述可扩展计算阵列连接；
所述输入激励缓存器用于存储所述输入激励；
所述输出激励缓存器用于存储所述输出激励；
所述权重索引缓存器用于存储所述权重索引。
本发明还提供一种基于结构化剪枝和低比特量化的神经网络加速方法，所述方法包括：
S01：对神经网络进行结构化剪枝和低比特量化，得到压缩模型和表征压缩权重位置的索引集；
S02：对所述压缩模型进行分析，得到网络各层参数和操作数位宽，并结合给定的硬件资源确定可扩展计算阵列的尺寸；
S03：根据可扩展计算阵列的尺寸调用带有IP核的主控制器、激励选择单元和处理单元组成神经网络加速器；
S04：读入原始的输入激励、所述压缩模型、所述索引集和指令集，开启神经网络加速器；所述指令集为结合网络各层参数和所述可扩展计算阵列的尺寸完成压缩网络到硬件资源的映射关系。
可选地，所述对神经网络进行结构化剪枝，具体包括：
将卷积层中不同三维卷积核相同位置的权重同时剪去，获得不同卷积核具有相同的索引；利用所述激励选择单元将输入特征图和一个卷积核索引进行筛选，获得所有有效计算的输入激励和权重索引；将全连接层中权重矩阵的某些行剪去。
可选地，所述压缩网络到硬件资源的映射关系，具体包括：
对于包括N×M个处理单元PE的可扩展计算阵列，卷积层计算时，所述可扩展计算阵列的每一行处理单元PE共享相同的权重，每一列处理单元PE共享相同的输入激励，因此整个所述可扩展计算阵列共享N个输入激励和一个权重索引；全连接层计算时，所述可扩展计算阵列每一行共享相同的输入激励，每个处理单元PE的权重各不相同，但是结构化剪枝使得同一列处理单元PE的权重索引相同，因此所述可扩展计算阵列共享N个输入激励和N个权重索引。
可选地，所述低比特量化具体公式为：
Figure PCTCN2020099891-appb-000001
其中，w表示权重向量，x表示输入激励，x i表示第i个输入激励，w i表示权重向量中的第i个权重，w′ i为w i量化后的值，n表示需要累加的部分和的总个数，函数Bitshift(a,b)表示将a移动b位，移动方向由b的符号决定，函数Quantize[y]对y进行取整。
根据本发明提供的具体实施例，本发明公开了以下技术效果：
本发明通过结构化剪枝和可扩展计算阵列上的数据共享，充分利用了神经网络推理运算过程中的数据可重利用性，降低了筛选有效计算的输入激励和权重的功耗，缓解了激励选择单元和可扩展计算阵列之间的高传输带宽压力，结合低比特量化技术，减小了权重参数的数量和存储位宽，进一步提升了卷积神经网络加速器的吞吐率和能效。
说明书附图
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为本发明实施例神经网络加速器顶层架构图；
图2为本发明实施例主控制器结构图；
图3为本发明实施例激励选择单元结构图；
图4为本发明实施例加速器工作流程图；
图5为本发明实施例结构化剪枝示意图；
图6为本发明实施例压缩权重和索引图；
图7为本发明实施例卷积运算模式图；
图8为本发明实施例全连接层运算模式图。
具体实施方式
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没 有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
本发明的目的是提供一种基于结构化剪枝和低比特量化的神经网络加速器及方法，以提升加速器加速效果，降低加速器功耗。
为使本发明的上述目的、特征和优点能够更加明显易懂，下面结合附图和具体实施方式对本发明作进一步详细的说明。
如图1所示，本发明公开一种基于结构化剪枝和低比特量化的神经网络加速器，所述神经网络加速器包括：
主控制器、激励选择单元、可扩展计算阵列、多功能处理单元、DMA、DRAM和缓存器；所述主控制器分别与所述激励选择单元、所述可扩展计算阵列和DMA连接，所述DMA分别与所述缓存器和所述DRAM连接，所述缓存器分别与所述多功能处理单元和所述激励选择单元连接，所述可扩展计算阵列分别与所述激励选择单元和所述缓存器连接；
所述主控制器用于解析指令集，产生输入激励、权重的第一存储地址和输出激励的存储地址以及控制信号；
所述缓存器用于存储所述输入激励、所述输出激励和权重索引；
所述激励选择单元用于根据所述主控制器生成的所述控制信号筛选从所述缓存器输入的输入激励，并传输给所述可扩展计算阵列；
所述可扩展计算阵列包括：N×M个处理单元PE；N、M分别为处理单元的行和列；每个PE存储神经网络的部分权重，根据接收的权重索引确定权重的第二存储地址，根据所述权重的第二存储地址通过所述激励选择单元获取输入激励对应的权重，并判断接收的输入激励是否为零从而控制权重的读取和PE内乘法器的开关；根据所述主控制器生成的所述控制信号判断当前计算的所述输出激励是否完成了一个输入通道的输入激励和权重的卷积，如果是，则将所述输出激励通过所述激励选择单元存储到所述缓存器中的输出激励缓存器；
所述多功能处理单元用于完成网络的池化、激活和归一化运算；
所述DMA用于根据所述权重的第一存储地址读取所述DRAM中存储的权重，根据所述当前输出激励的存储地址读取所述缓存器中存储的所述输出激励，并将所述输出激励发送至所述DRAM进行存储。
如图2所示，所述主控制器包括一个指令集解码器、一个地址生成器、一个控制向量生成器和七个计数器；所述指令集解码器分别与各所述计数器的输入端连接，各所述计数器的输出端分别与地址生成器和所述控制向量生成器连接，所述地址生成器与DMA连接，所述控制向量生成器分别与所述激励选择单元和所述可扩展计算阵列连接。七个计数器依次级联，前一级上溢信号overflow作为下一级增信号inc，七个计数器依次输出{kw,kh,ow,oh,ic,oc,l}，{kw,kh,ow,oh,ic,oc,l}表示加速进程，其中kw表示当前计算的权重的列数，kh表示当前计算的权重的行数，ow表示当前计算的输出激励的列数，oh表示当前计算的输出激励的行数，ic表示当前计算的输入激励的通道数，oc表示当前计算的输出激励的通道数，l表示当前计算的输入激励的批数；所述指令集解码器用于对指令集进行解码，生成各所述计数器的判断上限和下限，根据判断上限和下限控制七个计数器循环计数输出{kw,kh,ow,oh,ic,oc,l}。所述地址生成器用于根据七个计数器输出的{kw,kh,ow,oh,ic,oc,l}生成下一个时钟周期计算所需的输入激励和权重的第一存储地址，以及当前输出激励的存储地址，并传输给所述DMA。控制向量生成器用于根据七个计数器输出的{kw,kh,ow,oh,ic,oc,l}生成控制信号ctrl，进而根据所述控制信号ctrl控制所述激励选择单元和所述可扩展计算阵列。
如图3所示，所述激励选择单元包括：输入激励寄存器、索引解码器和选择器；所述选择器分别与所述输入激励寄存器和所述索引解码器连接。所述输入激励寄存器根据主控制器的控制信号完成输入激励的读入和输出；所述索引解码器将权重索引解码，生成跳变量jump；所述选择器根据所述跳变量jump信号筛选输入激励，具体的，根据索引集跳过零值权重对应的计算过程，选出非零值权重对应的输入激励和权重索引传输给所述可扩展计算阵列。
可扩展计算阵列包括：N×M个处理单元PE；N、M分别为处理单元的行和列；可分为N组，各组PE并行计算不同的输出通道，每组中的M个PE并行计算同一输出通道中不同的输出激励。
所述可扩展计算阵列计算流程为：每个PE存储神经网络的部分权重，根据接收的权重索引确定权重的第二存储地址，该读取地址为权重在片上 缓存器中的地址，根据所述权重的第二存储地址获取输入激励对应的权重，并判断接收的输入激励是否为零从而控制权重的读取和PE内乘法器的开关；具体的，若输入激励不为零，则读取相应权重，并使能乘法器，执行输入激励与权重的乘法计算，得到一个部分和；若输入激励为零，则不再读取对应权重，从而减小内存访问功耗，并关断乘法器来减小计算功耗，即该部分和为0。然后该部分和与之前得到的部分和进行累加，得到一个新的部分和，即当前输出激励。根据所述主控制器生成的控制信号判断当前输出激励是否完成了一个输入通道的输入激励和权重的卷积，如果是，则将所述输出激励通过所述激励选择单元存储到所述缓存器中的输出激励缓存器，否则存储在每个PE内部。
多功能处理单元用于完成网络的池化、激活和归一化运算，图1中可扩展计算阵列生成的输出激励通首先存入输出激励缓存器，然后在所述多功能处理单元进行网络的池化、激活和归一化运算处理，处理完成后返回最终的输出激励到输出激励缓存器中。
DRAM用于存储权重和输出激励，神经网络的参数数量巨大，对于VGG-16等大型网络而言，即使经过模型压缩，也难以把权重完全存在缓存器内。同时网络部分层的输出特征图尺寸巨大，同样需要DRAM。
所述缓存器包括输入激励缓存器、输出激励缓存器和索引缓存器；所述权重索引缓存器和所述输入激励缓存器分别与所述DMA连接，所述权重索引缓存器和所述输入激励缓存器分别与所述激励选择单元连接；所述输出激励缓存器分别与所述DMA、所述多功能处理单元和所述可扩展计算阵列连接；所述输入激励缓存器用于存储所述输入激励；所述输出激励缓存器用于存储所述输出激励；所述权重索引缓存器用于存储所述权重索引。网络分块进行计算，需要重复使用存在缓存器中的数据，以减少对DRAM的重复读取，降低读取功耗。
图4为本发明实施例加速器工作流程图，如图4所示，本发明公开一种基于结构化剪枝和低比特量化的神经网络加速方法，所述方法包括：
S01：对神经网络进行结构化剪枝和低比特量化，得到压缩模型和表征压缩权重位置的索引集；
S02：对所述压缩模型进行分析，得到网络各层参数和操作数位宽， 并结合给定的硬件资源确定可扩展计算阵列的尺寸；所述网络各层参数包括位宽、权重和激励；所述硬件资源包括片上内存大小和DSP数量；
S03：根据可扩展计算阵列的尺寸调用带有IP核的主控制器、激励选择单元和处理单元组成神经网络加速器；
S04：读入原始的输入激励、所述压缩模型、所述索引集和指令集，开启神经网络加速器；所述指令集为结合网络各层参数和所述可扩展计算阵列的尺寸完成压缩网络到硬件资源的映射关系；
图5为本发明实施例结构化剪枝示意图，图5中(a)卷积核的结构化剪枝示意图，图5中(b)权重矩阵的结构化剪枝示意图，如图5所示，对神经网络进行结构化剪枝，具体包括：
将卷积层中不同三维卷积核相同位置的权重同时剪去，获得不同卷积核具有相同的索引；利用所述激励选择单元将输入特征图和一个卷积核索引进行筛选，获得所有有效计算的输入激励和权重索引；将全连接层中权重矩阵的某些行剪去，输入激励行向量的对应列元素不需要进行计算，因此不被存储。
进一步的，所述压缩网络到硬件资源的映射关系，具体包括：
对于包括N×M个处理单元PE的可扩展计算阵列，卷积层计算时，所述可扩展计算阵列的每一行处理单元PE共享相同的权重，每一列处理单元PE共享相同的输入激励，因此整个所述可扩展计算阵列共享N个输入激励和一个权重索引；全连接层计算时，所述可扩展计算阵列每一行共享相同的输入激励，每个处理单元PE的权重各不相同，但是结构化剪枝使得同一列处理单元PE的权重索引相同，因此所述可扩展计算阵列共享N个输入激励和N个权重索引。
进一步的，所述低比特量化具体公式为：
Figure PCTCN2020099891-appb-000002
其中，w表示权重向量，x表示输入激励，x i表示第i个输入激励，w i表示权重向量中的第i个权重，w′ i为w i量化后的值，n表示需要累加的部分和的总个数，函数Bitshift(a,b)表示将a移动b位，移动方向由b的符 号决定，函数Quantize[y]对y进行取整；当b>0时，Bitshift(a,b)将a向左移b位，反之，向右移；当量化结果
Figure PCTCN2020099891-appb-000003
时，向上取整；反之，向下取整；另外不同网络具有不同的量化位宽要求，计算输入激励和权重的乘积时，可以用移位操作实现乘法运算，经过量化后，运算功耗大大降低。
本实施例对神经网络进行结构化剪枝，剪枝后稀疏网络的权重经过按行压缩再存储，同时还需存储记录权重位置的索引和行指针，如图6，索引表示两个非零权重之间“0”的个数，每一行的第一个非零权重的索引等于该行内此非零权重之前“0”的个数，行指针表示每一行内非零权重的总数。
本实施例将卷积层分块计算，并将卷积运算的循环展开，以充分利用可扩展计算阵列的并行性，提高数据的重利用。卷积层分块计算和循环展开的伪代码如下：
Figure PCTCN2020099891-appb-000004
其中，H为输入特征图的高度，W为输入特征图的宽度，C为输入特征图的通道数，R为权重的高度和宽度，F为输出特征图的通道数，U为输出特征图的高度，V为输出特征图的宽度，kw表示当前计算的权重的列数，kh表示当前计算的权重的行数，ow表示当前计算的输出激励的列数，oh表示当前计算的输出激励的行数，ic表示当前计算的输入激励的通道数，oc表示当前计算的输出激励的通道数，l表示当前计算的输入 激励的批数，Toc为输出特征图的通道数，Tow为输出特征图的列数，Y为输出激励，W为权重，X为输入激励。
本实施例运用循环展开使可扩展计算阵列同时计算输出特征图的N个通道的M个输出激励，即可扩展计算阵列的不同行计算输出特征图的不同通道，同一行内的不同处理单元计算输出特征图某一通道的不同输出激励。因此，可扩展计算阵列的每一行共享相同的权重，每一列共享相同的输入激励。同时，输入到不同行的权重来自不同三维卷积核的相同位置，结构化剪枝后同一层内不同三维卷积核具有相同的索引，因此整个可扩展计算阵列共享同一个权重索引。
本实施例将输入特征图X(H,W,C)与四维卷积核W(R,R,C,F)卷积得到输出特征图(U，V，F)的过程映射到N×M可扩展计算阵列，映射关系如下：
Figure PCTCN2020099891-appb-000005
其中，F为输出特征图的通道数，N为同时计算输出特征图的通道总个数，V为输出特征图的宽度，M为同时计算输出特征图的输出激励总个数，oc max为当前计算的输出激励的通道数最大值，ow max为当前计算的输出激励的列数最大值。
本实施例中，卷积到可扩展计算阵列每一行的映射是相同的，图7描述了输入特征图X(10,10,1)与四维卷积核W(3,3,1,16)的卷积过程在可扩展计算阵列某一行的运算模式，假设可扩展计算阵列每一行只有4个处理单元PE。图7中(a)为不经过结构化剪枝的网络卷积运算模式，因为V＝8，所以ow max＝2，即每个处理单元PE需要计算两个输出激励。图7中(b)为结构化剪枝网络的卷积运算模式，相比于图7中(a)，图7中(b)根据索引判断权重是否为零，权重为零时对应的计算可以跳过，判断过程在激励选择单元实现。
本实施例中，全连接层的运算如图8，权重矩阵的某些行剪去，因此输入激励向量的对应列元素不参与计算。本实施例在当前层的下一层是全连接层时，用下一层的权重索引筛选当前层的输出激励，如图8中(b)，并构建流水线将筛选过程隐藏在当前层输出激励计算和下一层输入激励读取之间，如图8中(c)。
本实施例中，激励选择单元实现输入激励的筛选。计算卷积层时，激励选择单元根据主控制器的控制信息读取权重索引jump，输入激励向量Data_in包含2M个输入激励，选择器将Data_in>>jump，以跳过权重为0时的计算。当权重计算到每一行最后一个权重时，主控制器给激励选择单元push信号，把新的输入激励存入激励寄存器中；当计算完一行权重时，主控制器给激励单元pop信号，输出新的输入激励向量Data_in。计算全连接层时，选择器读入当前层的输出激励和下一层的索引，筛选出非零权重对应的输出激励。

Claims (8)

  1. 一种基于结构化剪枝和低比特量化的神经网络加速器，其特征在于，所述神经网络加速器包括：
    主控制器、激励选择单元、可扩展计算阵列、多功能处理单元、DMA、DRAM和缓存器；所述主控制器分别与所述激励选择单元、所述可扩展计算阵列和DMA连接，所述DMA分别与所述缓存器和所述DRAM连接，所述缓存器分别与所述多功能处理单元和所述激励选择单元连接，所述可扩展计算阵列分别与所述激励选择单元和所述缓存器连接；
    所述主控制器用于解析指令集，产生输入激励、权重的第一存储地址和输出激励的存储地址以及控制信号；
    所述缓存器用于存储所述输入激励、所述输出激励和权重索引；
    所述激励选择单元用于根据所述主控制器生成的所述控制信号筛选从所述缓存器输入的输入激励，并传输给所述可扩展计算阵列；
    所述可扩展计算阵列包括：N×M个处理单元PE；N、M分别为处理单元的行和列；每个PE存储神经网络的部分权重，根据接收的权重索引确定权重的第二存储地址，根据所述权重的第二存储地址获取输入激励对应的权重，并判断接收的输入激励是否为零从而控制权重的读取和PE内乘法器的开关；根据所述主控制器生成的所述控制信号判断当前计算的所述输出激励是否完成了一个输入通道的输入激励和权重的卷积，如果是，则将所述输出激励通过所述激励选择单元存储到所述缓存器中的输出激励缓存器；
    所述多功能处理单元用于完成网络的池化、激活和归一化运算；
    所述DMA用于根据所述权重的第一存储地址读取所述DRAM中存储的权重，根据所述当前输出激励的存储地址读取所述缓存器中存储的所述输出激励，并将所述输出激励发送至所述DRAM进行存储。
  2. 根据权利要求1所述的基于结构化剪枝和低比特量化的神经网络加速器，其特征在于，所述主控制器包括：
    一个指令集解码器、一个地址生成器、一个控制向量生成器和七个计数器；所述指令集解码器分别与各所述计数器的输入端连接，各所述计数器的输出端分别与地址生成器和所述控制向量生成器连接，所述地址生成器与DMA连接，所述控制向量生成器分别与所述激励选择单元和所述可 扩展计算阵列连接，七个计数器依次级联，前一级上溢信号作为下一级增信号；
    所述指令集解码器用于对指令集进行解码，生成各所述计数器的判断上限和下限，根据判断上限和下限控制七个计数器循环计数输出{kw,kh,ow,oh,ic,oc,l}；
    所述地址生成器用于根据七个计数器输出的{kw,kh,ow,oh,ic,oc,l}生成下一个时钟周期计算所需的输入激励和权重的第一存储地址，以及当前输出激励的存储地址，并传输给所述DMA；
    所述控制向量生成器用于根据七个计数器输出的{kw,kh,ow,oh,ic,oc,l}生成控制信号ctrl，进而根据所述控制信号ctrl控制所述激励选择单元和所述可扩展计算阵列。
  3. 根据权利要求1所述的基于结构化剪枝和低比特量化的神经网络加速器，其特征在于，所述激励选择单元包括：
    输入激励寄存器、索引解码器和选择器；所述选择器分别与所述输入激励寄存器和所述索引解码器连接；
    所述输入激励寄存器用于根据所述主控制器生成的控制信号读入和输出输入激励；
    所述索引解码器用于将权重索引解码，生成跳变量；
    所述选择器用于根据所述跳变量筛选输入激励，并输送至所述可扩展计算阵列。
  4. 根据权利要求1所述的基于结构化剪枝和低比特量化的神经网络加速器，其特征在于，所述缓存器包括：
    输入激励缓存器、输出激励缓存器和权重索引缓存器；所述权重索引缓存器和所述输入激励缓存器分别与所述DMA连接，所述权重索引缓存器和所述输入激励缓存器分别与所述激励选择单元连接；所述输出激励缓存器分别与所述DMA、所述多功能处理单元和所述可扩展计算阵列连接；
    所述输入激励缓存器用于存储所述输入激励；
    所述输出激励缓存器用于存储所述输出激励；
    所述权重索引缓存器用于存储所述权重索引。
  5. 一种基于结构化剪枝和低比特量化的神经网络加速方法，其特征 在于，所述方法包括：
    S01：对神经网络进行结构化剪枝和低比特量化，得到压缩模型和表征压缩权重位置的索引集；
    S02：对所述压缩模型进行分析，得到网络各层参数和操作数位宽，并结合给定的硬件资源确定可扩展计算阵列的尺寸；
    S03：根据可扩展计算阵列的尺寸调用带有IP核的主控制器、激励选择单元和处理单元组成神经网络加速器；
    S04：读入原始的输入激励、所述压缩模型、所述索引集和指令集，开启神经网络加速器；所述指令集为结合网络各层参数和所述可扩展计算阵列的尺寸完成压缩网络到硬件资源的映射关系。
  6. 根据权利要求5所述的基于结构化剪枝和低比特量化的神经网络加速方法，其特征在于，所述对神经网络进行结构化剪枝，具体包括：
    将卷积层中不同三维卷积核相同位置的权重同时剪去，获得不同卷积核具有相同的索引；利用所述激励选择单元将输入特征图和一个卷积核索引进行筛选，获得所有有效计算的输入激励和权重索引；将全连接层中权重矩阵的某些行剪去。
  7. 根据权利要求5所述的基于结构化剪枝和低比特量化的神经网络加速方法，其特征在于，所述压缩网络到硬件资源的映射关系，具体包括：
    对于包括N×M个处理单元PE的可扩展计算阵列，卷积层计算时，所述可扩展计算阵列的每一行处理单元PE共享相同的权重，每一列处理单元PE共享相同的输入激励，因此整个所述可扩展计算阵列共享N个输入激励和一个权重索引；全连接层计算时，所述可扩展计算阵列每一行共享相同的输入激励，每个处理单元PE的权重各不相同，但是结构化剪枝使得同一列处理单元PE的权重索引相同，因此所述可扩展计算阵列共享N个输入激励和N个权重索引。
  8. 根据权利要求5所述的基于结构化剪枝和低比特量化的神经网络加速方法，其特征在于，所述低比特量化具体公式为：
    Figure PCTCN2020099891-appb-100001
    其中，w表示权重向量，x表示输入激励，x i表示第i个输入激励，w i表示权重向量中的第i个权重，w′ i为w i量化后的值，n表示需要累加的部分和的总个数，函数Bitshift(a,b)表示将a移动b位，移动方向由b的符号决定，函数Quantize[y]对y进行取整。
PCT/CN2020/099891 2019-07-08 2020-07-02 基于结构化剪枝和低比特量化的神经网络加速器及方法 Ceased WO2021004366A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/485,645 US12271820B2 (en) 2019-07-08 2021-09-27 Neural network acceleration and neural network acceleration method based on structured pruning and low-bit quantization

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910609993.5 2019-07-08
CN201910609993.5A CN110378468B (zh) 2019-07-08 2019-07-08 一种基于结构化剪枝和低比特量化的神经网络加速器

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/485,645 Continuation US12271820B2 (en) 2019-07-08 2021-09-27 Neural network acceleration and neural network acceleration method based on structured pruning and low-bit quantization

Publications (1)

Publication Number Publication Date
WO2021004366A1 true WO2021004366A1 (zh) 2021-01-14

Family

ID=68252417

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/099891 Ceased WO2021004366A1 (zh) 2019-07-08 2020-07-02 基于结构化剪枝和低比特量化的神经网络加速器及方法

Country Status (3)

Country Link
US (1) US12271820B2 (zh)
CN (1) CN110378468B (zh)
WO (1) WO2021004366A1 (zh)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112819684A (zh) * 2021-03-02 2021-05-18 成都视海芯图微电子有限公司 一种面向图像文本识别的加速装置
CN113361695A (zh) * 2021-06-30 2021-09-07 南方电网数字电网研究院有限公司 卷积神经网络加速器
CN113487020A (zh) * 2021-07-08 2021-10-08 中国科学院半导体研究所 用于神经网络计算的参差存储结构及神经网络计算方法
CN113657595A (zh) * 2021-08-20 2021-11-16 中国科学院计算技术研究所 神经网络实时剪枝方法、系统及神经网络加速器
CN114463161A (zh) * 2022-04-12 2022-05-10 之江实验室 一种基于忆阻器的神经网络处理连续图像的方法和装置
CN115358383A (zh) * 2022-07-06 2022-11-18 华南理工大学 一种联合比特量化和比特权重搜索的模型压缩方法
CN115393174A (zh) * 2022-10-27 2022-11-25 之江实验室 一种粗粒度的图像神经网络加速器指令集架构方法及装置
CN115600647A (zh) * 2022-10-21 2023-01-13 中国科学技术大学（Cn） 一种面向稀疏神经网络加速的bit级计算的模型架构系统
CN115965067A (zh) * 2023-02-01 2023-04-14 苏州亿铸智能科技有限公司 一种针对ReRAM的神经网络加速器
CN118409872A (zh) * 2024-07-02 2024-07-30 山东海量信息技术研究院 一种基于内存算力感知的模型剪枝方法、设备及介质

Families Citing this family (46)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111738428B (zh) * 2019-03-25 2023-08-25 上海寒武纪信息科技有限公司 计算装置、方法及相关产品
US11487998B2 (en) 2019-06-17 2022-11-01 Qualcomm Incorporated Depth-first convolution in deep neural networks
CN110378468B (zh) * 2019-07-08 2020-11-20 浙江大学 一种基于结构化剪枝和低比特量化的神经网络加速器
CN110796238B (zh) * 2019-10-29 2020-12-08 上海安路信息科技有限公司 基于arm架构fpga硬件系统的卷积神经网络权重压缩方法及其装置
CN110889204B (zh) * 2019-11-06 2021-07-30 北京中科胜芯科技有限公司 一种神经网络模型参数压缩方法
CN110852422B (zh) * 2019-11-12 2022-08-16 吉林大学 基于脉冲阵列的卷积神经网络优化方法及装置
CN110991608B (zh) * 2019-11-25 2021-08-13 恒烁半导体(合肥)股份有限公司 一种卷积神经网络量化计算方法及系统
CN111062472B (zh) * 2019-12-11 2023-05-12 浙江大学 一种基于结构化剪枝的稀疏神经网络加速器及其加速方法
CN113095468B (zh) * 2019-12-23 2024-04-16 上海商汤智能科技有限公司 神经网络加速器及其数据处理方法
CN111242277B (zh) * 2019-12-27 2023-05-05 中国电子科技集团公司第五十二研究所 一种基于fpga设计的支持稀疏剪枝的卷积神经网络加速器
CN111401554B (zh) * 2020-03-12 2023-03-24 交叉信息核心技术研究院(西安)有限公司 支持多粒度稀疏与多模式量化的卷积神经网络的加速器
KR102474787B1 (ko) * 2020-03-30 2022-12-06 포항공과대학교 산학협력단 일정한 확률의 인덱스 매칭을 수행하는 희소성 인식 신경 처리 유닛 및 처리 방법
CN111563589B (zh) * 2020-04-14 2024-01-16 中科物栖(南京)科技有限公司 一种神经网络模型的量化方法及装置
CN111582446B (zh) * 2020-04-28 2022-12-06 北京达佳互联信息技术有限公司 用于神经网络剪枝的系统和神经网络的剪枝处理方法
US12361266B2 (en) * 2020-05-14 2025-07-15 Samsung Electronics Co., Ltd. Hierarchical weight preprocessing for neural network accelerator
CN111738427B (zh) * 2020-08-14 2020-12-29 电子科技大学 一种神经网络的运算电路
CN114118433B (zh) * 2020-08-26 2025-01-14 华为技术有限公司 一种设备的配置参数的推荐方法及装置
CN114186677B (zh) * 2020-09-15 2025-10-14 中兴通讯股份有限公司 加速器参数确定方法及装置、计算机可读介质
CN113159302B (zh) * 2020-12-15 2022-07-19 浙江大学 一种用于可重构神经网络处理器的路由结构
CN112631983B (zh) * 2020-12-28 2023-05-02 电子科技大学 一种面向稀疏神经网络的片上系统
CN112926733B (zh) * 2021-03-10 2022-09-16 之江实验室 一种语音关键词检测专用芯片
CN113052307B (zh) * 2021-03-16 2022-09-06 上海交通大学 一种面向忆阻器加速器的神经网络模型压缩方法及系统
CN115222015A (zh) 2021-04-21 2022-10-21 阿里巴巴新加坡控股有限公司 指令处理装置、加速单元和服务器
US11775317B2 (en) * 2021-04-30 2023-10-03 International Business Machines Corporation Locate neural network performance hot spots
CN113408723B (zh) * 2021-05-19 2023-04-07 北京理工大学 一种面向遥感应用的卷积神经网络剪枝和量化同步压缩方法
CN113269312B (zh) * 2021-06-03 2021-11-09 华南理工大学 一种联合量化与剪枝搜索的模型压缩方法及其系统
US20230013599A1 (en) * 2021-07-08 2023-01-19 Black Sesame International Holding Limited Adaptive mac array scheduling in a convolutional neural network
CN113688976B (zh) * 2021-08-26 2025-08-19 伟光有限公司 一种神经网络加速方法、装置、设备及存储介质
CN114565801A (zh) * 2021-10-22 2022-05-31 上海航天控制技术研究所 一种卷积神经网络的红外图像目标检测fpga加速器设计方法
US20230168644A1 (en) * 2021-12-01 2023-06-01 Low Power Futures, Inc. Optimized processing engine of an internet of things (iot) device and a method of generating the same
FR3131429B1 (fr) * 2021-12-29 2024-11-08 Commissariat Energie Atomique Système de transfert direct de données
CN114781629B (zh) * 2022-04-06 2024-03-05 合肥工业大学 基于并行复用的卷积神经网络的硬件加速器及并行复用方法
US12437195B2 (en) * 2022-07-07 2025-10-07 Taiwan Semiconductor Manufacturing Company, Ltd. Systems and methods for pipelined heterogeneous dataflow for artificial intelligence accelerators
CN115374935B (zh) * 2022-09-15 2023-08-11 重庆大学 一种神经网络的剪枝方法
CN115759219A (zh) * 2022-12-06 2023-03-07 上海安路信息科技股份有限公司 卷积加速计算控制单元及系统
CN115828044B (zh) * 2023-02-17 2023-05-19 绍兴埃瓦科技有限公司 基于神经网络双重稀疏性矩阵乘法运算电路、方法和装置
CN115879530B (zh) * 2023-03-02 2023-05-05 湖北大学 一种面向rram存内计算系统阵列结构优化的方法
US12200101B2 (en) * 2023-06-14 2025-01-14 Trustees Of Boston University Semi-custom accelerator device for bootstrappable fully homomorphic encryption
CN116957022B (zh) * 2023-07-08 2025-08-12 复旦大学 面向手势识别的稀疏二值神经网络硬件加速器
CN117273099B (zh) * 2023-09-20 2025-11-28 西安科技大学 一种可重构阵列下Transformer网络模型数据复用方案与实现方法
CN118333108B (zh) * 2024-04-18 2025-03-21 北京交通大学 一种用于卷积神经网络输入激活值4比特量化方法
CN118364897B (zh) * 2024-06-14 2024-09-03 湘江实验室 一种基于混合精度量化与模拟退火剪枝的联邦学习训练方法
CN118820568B (zh) * 2024-09-19 2025-01-24 浙江大华技术股份有限公司 模型量化方法、电子设备和计算机可读存储介质
CN119443167B (zh) * 2025-01-09 2025-04-08 安徽大学 一种面向Grounding DINO的量化推理加速方法及系统
CN120146122B (zh) * 2025-02-26 2025-10-28 哈尔滨工业大学 一种面向神经网络加速器的算存融合新型架构的设计方法
CN121051064B (zh) * 2025-11-04 2026-02-13 华南理工大学 一种比特级动态稀疏的存内计算架构

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9916531B1 (en) * 2017-06-22 2018-03-13 Intel Corporation Accumulator constrained quantization of convolutional neural networks
CN107977704A (zh) * 2017-11-10 2018-05-01 中国科学院计算技术研究所 权重数据存储方法和基于该方法的神经网络处理器
CN109472350A (zh) * 2018-10-30 2019-03-15 南京大学 一种基于块循环稀疏矩阵的神经网络加速系统
CN109598338A (zh) * 2018-12-07 2019-04-09 东南大学 一种基于fpga的计算优化的卷积神经网络加速器
CN109886397A (zh) * 2019-03-21 2019-06-14 西安交通大学 一种针对卷积层的神经网络结构化剪枝压缩优化方法
CN110378468A (zh) * 2019-07-08 2019-10-25 浙江大学 一种基于结构化剪枝和低比特量化的神经网络加速器

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109389208B (zh) * 2017-08-09 2021-08-31 上海寒武纪信息科技有限公司 数据的量化装置及量化方法
US10872290B2 (en) * 2017-09-21 2020-12-22 Raytheon Company Neural network processor with direct memory access and hardware acceleration circuits
US11429862B2 (en) * 2018-03-20 2022-08-30 Sri International Dynamic adaptation of deep neural networks
US11461631B2 (en) * 2018-03-22 2022-10-04 Amazon Technologies, Inc. Scheduling neural network computations based on memory capacity
US12112256B2 (en) * 2018-07-26 2024-10-08 Intel Corporation Loss-error-aware quantization of a low-bit neural network
US10747502B2 (en) * 2018-09-19 2020-08-18 Xilinx, Inc. Multiply and accumulate circuit
US10746792B1 (en) * 2018-11-30 2020-08-18 Amazon Technologies, Inc. Debug mechanisms for a processor circuit
US11475308B2 (en) * 2019-03-15 2022-10-18 Samsung Electronics Co., Ltd. Jointly pruning and quantizing deep neural networks

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9916531B1 (en) * 2017-06-22 2018-03-13 Intel Corporation Accumulator constrained quantization of convolutional neural networks
CN107977704A (zh) * 2017-11-10 2018-05-01 中国科学院计算技术研究所 权重数据存储方法和基于该方法的神经网络处理器
CN109472350A (zh) * 2018-10-30 2019-03-15 南京大学 一种基于块循环稀疏矩阵的神经网络加速系统
CN109598338A (zh) * 2018-12-07 2019-04-09 东南大学 一种基于fpga的计算优化的卷积神经网络加速器
CN109886397A (zh) * 2019-03-21 2019-06-14 西安交通大学 一种针对卷积层的神经网络结构化剪枝压缩优化方法
CN110378468A (zh) * 2019-07-08 2019-10-25 浙江大学 一种基于结构化剪枝和低比特量化的神经网络加速器

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
JI RONGRONG , LIN SHAOHUI , CHAO FEI , WU YONGJIAN , HUANG FEIYUE: "Deep Neural Network Compression and Acceleration: A Review", JOURNAL OF COMPUTER RESEARCH AND DEVELOPMENT, vol. 55, no. 9, 15 September 2018 (2018-09-15), pages 1871 - 1888, XP055772304, ISSN: 1000-1239, DOI: 10.7544/issn.1000-1239.2018.20180129 *
ZHANG LILI: "Research on the Acceleration of Tiny-yolo Convolution Neural Network Based on HLS", CHINESE MASTER'S THESES FULL-TEXT DATABASE, 1 April 2017 (2017-04-01), pages 1 - 69, XP055772310 *

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112819684A (zh) * 2021-03-02 2021-05-18 成都视海芯图微电子有限公司 一种面向图像文本识别的加速装置
CN113361695B (zh) * 2021-06-30 2023-03-24 南方电网数字电网研究院有限公司 卷积神经网络加速器
CN113361695A (zh) * 2021-06-30 2021-09-07 南方电网数字电网研究院有限公司 卷积神经网络加速器
CN113487020A (zh) * 2021-07-08 2021-10-08 中国科学院半导体研究所 用于神经网络计算的参差存储结构及神经网络计算方法
CN113487020B (zh) * 2021-07-08 2023-10-17 中国科学院半导体研究所 用于神经网络计算的参差存储结构及神经网络计算方法
CN113657595A (zh) * 2021-08-20 2021-11-16 中国科学院计算技术研究所 神经网络实时剪枝方法、系统及神经网络加速器
CN113657595B (zh) * 2021-08-20 2024-03-12 中国科学院计算技术研究所 基于神经网络实时剪枝的神经网络加速器
CN114463161B (zh) * 2022-04-12 2022-09-13 之江实验室 一种基于忆阻器的神经网络处理连续图像的方法和装置
CN114463161A (zh) * 2022-04-12 2022-05-10 之江实验室 一种基于忆阻器的神经网络处理连续图像的方法和装置
CN115358383A (zh) * 2022-07-06 2022-11-18 华南理工大学 一种联合比特量化和比特权重搜索的模型压缩方法
CN115600647A (zh) * 2022-10-21 2023-01-13 中国科学技术大学（Cn） 一种面向稀疏神经网络加速的bit级计算的模型架构系统
CN115393174A (zh) * 2022-10-27 2022-11-25 之江实验室 一种粗粒度的图像神经网络加速器指令集架构方法及装置
CN115965067A (zh) * 2023-02-01 2023-04-14 苏州亿铸智能科技有限公司 一种针对ReRAM的神经网络加速器
CN115965067B (zh) * 2023-02-01 2023-08-25 苏州亿铸智能科技有限公司 一种针对ReRAM的神经网络加速器
CN118409872A (zh) * 2024-07-02 2024-07-30 山东海量信息技术研究院 一种基于内存算力感知的模型剪枝方法、设备及介质

Also Published As

Publication number Publication date
US20220012593A1 (en) 2022-01-13
CN110378468B (zh) 2020-11-20
CN110378468A (zh) 2019-10-25
US12271820B2 (en) 2025-04-08

Similar Documents

Publication Publication Date Title
WO2021004366A1 (zh) 基于结构化剪枝和低比特量化的神经网络加速器及方法
CN111062472B (zh) 一种基于结构化剪枝的稀疏神经网络加速器及其加速方法
Wu et al. Easyquant: Post-training quantization via scale optimization
CN112200300B (zh) 卷积神经网络运算方法及装置
CN113762493B (zh) 神经网络模型的压缩方法、装置、加速单元和计算系统
CN109791628B (zh) 神经网络模型分块压缩方法、训练方法、计算装置及系统
CN111445012A (zh) 一种基于fpga的分组卷积硬件加速器及其方法
CN106846235B (zh) 一种利用NVIDIA Kepler GPU汇编指令加速的卷积优化方法及系统
CN109886397A (zh) 一种针对卷积层的神经网络结构化剪枝压缩优化方法
CN112598129A (zh) 基于ReRAM神经网络加速器的可调硬件感知的剪枝和映射框架
CN112257844B (zh) 一种基于混合精度配置的卷积神经网络加速器及其实现方法
CN111931917A (zh) 前向计算的实现方法及装置、存储介质、电子装置
CN113158132B (zh) 一种基于非结构化稀疏的卷积神经网络加速系统
CN111507465B (zh) 一种可配置的卷积神经网络处理器电路
CN101576961A (zh) 高速的图像匹配方法及装置
CN115983350A (zh) 一种二值权重卷积神经网络加速器及risc-v片上系统
CN109948787A (zh) 用于神经网络卷积层的运算装置、芯片及方法
CN115482456A (zh) 一种yolo算法的高能效fpga加速架构
Wang et al. Balancing memory-accessing and computing over sparse DNN accelerator via efficient data packaging
CN110377874A (zh) 卷积运算方法及系统
CN113705784A (zh) 一种基于矩阵共享的神经网络权重编码方法及硬件系统
CN112988229A (zh) 基于异构计算的卷积神经网络资源优化配置方法
CN113592072B (zh) 一种面向访存优化的稀疏卷积神经网络加速器
TWI798591B (zh) 卷積神經網路運算方法及裝置
Li et al. Fpga-based object detection acceleration architecture design

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20836180

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 20836180

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 20836180

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 21.09.2022)

122 Ep: pct application non-entry in european phase

Ref document number: 20836180

Country of ref document: EP

Kind code of ref document: A1