WO2020199476A1 - 基于脉动阵列的神经网络加速方法、装置、计算机设备及存储介质 - Google Patents

基于脉动阵列的神经网络加速方法、装置、计算机设备及存储介质 Download PDF

Info

Publication number
WO2020199476A1
WO2020199476A1 PCT/CN2019/103137 CN2019103137W WO2020199476A1 WO 2020199476 A1 WO2020199476 A1 WO 2020199476A1 CN 2019103137 W CN2019103137 W CN 2019103137W WO 2020199476 A1 WO2020199476 A1 WO 2020199476A1
Authority
WO
WIPO (PCT)
Prior art keywords
convolution
filter
feature
feature map
convolved
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/103137
Other languages
English (en)
French (fr)
Inventor
郭跃超
高鹏
谢国彤
唐义君
张萌
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Technology Shenzhen Co Ltd
Original Assignee
Ping An Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Technology Shenzhen Co Ltd filed Critical Ping An Technology Shenzhen Co Ltd
Publication of WO2020199476A1 publication Critical patent/WO2020199476A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means

Definitions

  • This application relates to the technical field of neural networks, and in particular to a neural network acceleration method, device, computer equipment, and storage medium based on a systolic array.
  • the prior art generally calculates the convolution result when the convolution step length is 1, and then down-samples and discards the unnecessary convolution result to obtain the feature map of a specific convolution step length. This obviously wastes calculation and scheduling resources. It will also slow down the convolution calculation.
  • the embodiments of the application provide a systolic array-based neural network acceleration method, device, computer equipment, and storage medium, which can better solve the problem of systolic array calculation and scheduling resources being wasted in convolution calculation with a step size of not one.
  • this application provides a neural network acceleration method based on a systolic array, the method including:
  • the convolution step length is not 1 and the size of the convolution filter is greater than 1 ⁇ 1, a number of sub-filters are segmented from the convolution filter according to a preset filter segmentation rule, and each sub-filter is The size of the filter is smaller than the size of the convolution filter;
  • the convolution calculation results corresponding to each of the subfilters are superimposed, and the superimposed result is output as the convolution calculation result of the feature map to be convolved by the convolution filter.
  • the present application provides a neural network acceleration device based on a systolic array, the device including:
  • a convolution parameter acquisition module configured to acquire convolution parameters of a convolution filter, where the convolution parameters include a convolution step size and a size of the convolution filter;
  • the filter segmentation module is configured to: if the convolution step size is not 1 and the size of the convolution filter is greater than 1 ⁇ 1, segment the convolution filter according to a preset filter segmentation rule. Filter, the size of each of the sub-filters is smaller than the size of the convolution filter;
  • the feature map segmentation module is used to obtain the feature map to be convolved and segment the feature map to be convolved according to a preset feature map segmentation rule into several feature sub-images, the several feature sub-images and the several sub-filters One-to-one correspondence
  • the convolution module is configured to perform convolution calculation on the corresponding feature sub-maps according to each of the sub-filters based on the systolic array, and the step size of the convolution calculation is 1;
  • the superposition module is configured to superimpose the convolution calculation results corresponding to each of the sub-filters, and output the superposition result as the convolution calculation result of the feature map to be convolved by the convolution filter.
  • the present application provides a computer device that includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and when the computer is executed
  • the program realizes the above-mentioned neural network acceleration method based on systolic array.
  • the present application provides a computer-readable storage medium that stores a computer program. If the computer program is executed by a processor, the above-mentioned systolic array-based neural network acceleration method is implemented.
  • This application discloses a neural network acceleration method, device, equipment and storage medium based on a systolic array.
  • the convolution step length is not 1
  • a number of sub-filters are segmented from the convolution filter according to a preset filter segmentation rule.
  • the preset feature map segmentation rules According to the preset feature map segmentation rules, several feature submaps can be segmented from the feature map to be convolved.
  • the convolution calculation can be performed with the convolution step length of 1, and the convolution calculation results corresponding to each subfilter are superimposed.
  • the superposition result of is the same as the convolution calculation result of the convolution step size of the convolution feature map to be performed according to the original convolution filter is not 1, that is, the two convolution calculations before and after the segmentation operation are equivalent; but due to the segmentation operation
  • the post-convolution step size is 1, which can make fuller use of the computing power of the systolic array.
  • FIG. 1 is a schematic flowchart of a neural network acceleration method based on a systolic array according to an embodiment of the application;
  • FIG. 2 is a schematic flowchart of a neural network acceleration method based on a systolic array according to another embodiment of the application;
  • Fig. 3 is a schematic diagram of a partitioned convolution filter when the convolution step length is 2 and the convolution filter size is 2 ⁇ 2;
  • 4 is a schematic diagram of segmenting the feature map to be convolved when the convolution step size is 2 and the convolution filter size is 2 ⁇ 2;
  • FIG. 5 is a schematic diagram of a sub-process of an embodiment of dividing the feature map to be convolved in FIG. 1;
  • FIG. 6 is a schematic diagram of a feature map to be convolved after segmentation and zero padding
  • Figure 7 is a schematic diagram of the structure of a systolic array
  • Fig. 8 is a schematic diagram of convolution calculation performed by a systolic array
  • FIG. 9 is a schematic diagram of a sub-process of performing convolution calculation based on a systolic array in FIG. 1;
  • FIG. 10 is a schematic flowchart of a neural network acceleration method based on a systolic array according to another embodiment of the application;
  • FIG. 11 is a schematic diagram of a divided convolution filter when the convolution step size is 2 and the convolution filter size is 3 ⁇ 3;
  • FIG. 12 is a schematic diagram of a sub-process of a partitioned convolution filter when the convolution step size is 2 and the convolution filter size is 3 ⁇ 3;
  • FIG. 13 is a schematic diagram of segmenting the feature map to be convolved when the convolution step length is 2 and the convolution filter size is 3 ⁇ 3;
  • FIG. 14 is a schematic flowchart of a neural network acceleration method based on a systolic array according to another embodiment of the application.
  • Fig. 15 is a schematic diagram of a partitioned convolution filter when the convolution step length is 3 and the convolution filter size is 3 ⁇ 3;
  • 16 is a schematic diagram of a sub-process of segmenting the feature map to be convolved when the convolution step is 3 and the size of the convolution filter is 3 ⁇ 3;
  • FIG. 17 is a schematic diagram of segmenting the feature map to be convolved when the convolution step length is 3 and the convolution filter size is 3 ⁇ 3;
  • FIG. 18 is a schematic diagram of performing equivalent transformation of the topological structure of a deep convolutional neural network downsampling according to a neural network acceleration method
  • FIG. 19 is a schematic structural diagram of a neural network acceleration device based on a systolic array according to an embodiment of the application;
  • 20 is a schematic structural diagram of a neural network acceleration device based on a systolic array according to another embodiment of the application;
  • FIG. 21 is a schematic structural diagram of a computer device provided by an embodiment of this application.
  • the embodiments of the present application provide a neural network acceleration method, device, equipment and storage medium based on a systolic array.
  • the systolic array-based neural network acceleration method can be applied to a terminal or a server to accelerate the training or inference of the systolic array-based neural network.
  • FIG. 1 is a schematic flowchart of a neural network acceleration method based on a systolic array provided by an embodiment of the present application.
  • the neural network acceleration method based on systolic array includes the following steps:
  • Step S110 Obtain convolution parameters of the convolution filter.
  • the convolution parameter includes the convolution step size and the size of the convolution filter.
  • Filter also known as the kernel kernel, feature detector, sliding the filter on the input image or feature map and calculating the point product is the convolution operation, and the output matrix of the convolution operation is called the convolution feature ( Convolved Feature), activation map (Activation Map) or feature map (Feature Map).
  • Convolved Feature Convolved Feature
  • Activation Map activation map
  • Feature Map feature map
  • the pre-stored or initialized convolution parameters of the convolution filter are first obtained.
  • the convolution parameter includes the convolution step stride and the size of the convolution filter, that is, the height h and the width w of the convolution filter; in other embodiments, the convolution parameter also includes input The number of channels and/or the number of output channels; the number of input channels in depth is determined by the number of channels in the feature map to be convolved, and the number of output channels out depth is equal to the number of convolution filters, which can determine the output after convolution The number of channels of the feature map.
  • the systolic array-based neural network acceleration method can be used in scenarios where the number of input channels is equal to or greater than 1, and can also be used in scenarios where the number of output channels is equal to or greater than 1.
  • Step S120 If the convolution step size is not 1 and the size of the convolution filter is greater than 1 ⁇ 1, segment a number of sub-filters from the convolution filter according to a preset filter segmentation rule.
  • the size of each sub-filter is smaller than the size of the convolution filter.
  • Some deep learning accelerators such as FPGA, dedicated NPU, etc. are usually implemented with a systolic array structure in the convolution part, but this structure is very unfriendly to the case where the convolution step length is not equal to 1; this embodiment will be larger than 1 ⁇ 1
  • the convolution filter is divided into several sub-filters, so that each sub-filter performs a convolution operation with a convolution step length equal to 1, so as to make full use of the performance of the systolic array structure.
  • step S120 if the convolution step size is not 1 and the size of the convolution filter is greater than 1 ⁇ 1, according to a preset filter segmentation rule
  • the convolution filter divides into several sub-filters, which specifically include:
  • Step S121 If the convolution step size is 2 and the size of the convolution filter is 2 ⁇ 2, 4 subfilters are divided from the convolution filter, and the size of each subfilter is 1. ⁇ 1.
  • the convolution parameter corresponding to a certain convolution operation is [6 6 2 2], that is, the number of input channels in depth is 6, the number of output channels out depth is equal to 6, and the size of the 6 convolution filters KernelTenseor Both are 2 ⁇ 2.
  • each 2 ⁇ 2 convolution filter is divided into 4 1 ⁇ 1 sub-filters.
  • the first sub-filter includes the weight w1 of the odd-numbered rows and odd-numbered columns of the convolution filter
  • the second sub-filter includes the convolution
  • the third subfilter includes the weights w3 of the even rows and odd columns of the convolution filter
  • the fourth subfilter includes the weights of the even rows and even columns of the convolution filter.
  • Step S130 Obtain a feature map to be convolved, and segment a number of feature sub-maps from the feature map to be convolved according to a preset feature map segmentation rule.
  • the plurality of feature sub-images correspond to the plurality of sub-filters one to one.
  • the number of channels of the feature map to be convolved may be equal to or greater than 1, and the number of channels of the feature map to be convolved may determine the number of input channels in the convolution parameter of the corresponding convolution filter.
  • step S130 obtains the feature map to be convolved and performs the The feature map segmentation rules of to segment several feature sub-maps from the feature map to be convolved, which specifically include:
  • Step S1311 Assign the values of the odd rows and odd columns of the feature map to be convolved to the corresponding positions of the first feature submap.
  • Step S1312 Assign the values of the odd rows and even columns of the feature map to be convolved to the corresponding positions of the second feature submap.
  • Step S1313 Assign the values of the even rows and odd columns of the feature map to be convolved to the corresponding positions of the third feature submap.
  • Step S1314 Assign the values of the even rows and even columns of the feature map to be convolved to the corresponding positions of the fourth feature submap.
  • the values in the same row in the feature map to be convolved are also located in the same row in each feature submap, and the values in the same column in the feature map to be convolved are also located in the same column in each feature submap.
  • the acquired convolution feature map input Tensor is the convolution feature map of [1 6 4 4].
  • the number of channels of the feature map to be convolved is 6, and the width and height are both 4.
  • 4 feature submaps are segmented from the feature map to be convolved.
  • step S130 acquiring a feature map to be convolved and segmenting several feature sub-maps from the feature map to be convolved according to a preset feature map segmentation rule specifically includes:
  • Step S131 Obtain a feature map to be convolved.
  • the acquired convolution feature map is shown in FIG. 6.
  • Step S132 If the length or width of the acquired feature map to be convolved is not an integer multiple of the convolution step length, perform zero padding on the preset position of the feature map to be convolved to make the zero padding to be convolved
  • the length or width of the feature map is an integer multiple of the convolution step length.
  • the length and width of the acquired convolution feature map are both 3.
  • the padding is exemplarily added to the right and bottom of the convolution feature map, so that the length of the feature map to be convolved after zero padding is The width is 4.
  • Step S133 According to a preset feature map segmentation rule, a number of feature sub-maps are segmented from the feature map to be convolved after zero padding.
  • the feature maps to be convolved for different channels under the same number can be divided and convolved first, and then the feature maps to be convolved under the next number and different channels can be divided and convolved.
  • Step S140 based on the systolic array, perform convolution calculation on the corresponding feature submap according to each of the subfilters, and the step size of the convolution calculation is 1.
  • Systolic Array The core concept of Systolic Array is to allow data to flow in the array of arithmetic units, reduce the number of memory accesses, and make the structure more regular, the wiring more uniform, and the frequency.
  • the Systolic Array includes L ⁇ L processing units PE, and the systolic array is connected to the weight register filter buffer, the input register in buffer, and the output register out buffer.
  • the left side of each row processing unit PE and the upper side of each column processing unit PE are provided with a first-in first-out register FIFO.
  • the weight of the filter is stored in the first-in first-out register FIFO and transmitted to all processing units PE in the same row.
  • the processing units PE in the first row and the first column receive the data from the feature map to be convolved in the input register, and the first The processing unit PE in the row and the first column both transmit data from the feature map to be convolved to the processing unit PE in the lower right corner of each. This design maximizes data reuse.
  • the systolic array performs two-dimensional convolution on a 5 ⁇ 5 feature map X according to a 3 ⁇ 3 filter W.
  • wi and xj respectively represent a certain row of data of filter W and feature map X
  • the three processing units PE in the last row output three rows of convolution results:
  • step S140 is based on the systolic array, and performs convolution calculation on the corresponding feature sub-map according to each of the sub-filters, which specifically includes:
  • Step S141 Load the weight of the sub-filter into the weight register connected to the systolic array.
  • the weight of the sub-filter is loaded into the weight register filter buffer, and the weight of the sub-filter is stored in the first-in first-out register FIFO and transmitted to the processing unit PE in the same row.
  • Step S142 Load the characteristic sub-map corresponding to the sub-filter into the input register connected to the systolic array.
  • the feature submap corresponding to the subfilter is loaded into the input register in buffer, and the processing unit PE in the first row and the first column of the systolic array receives data from the feature submap in the input register in buffer.
  • Step S143 Obtain the output result of the systolic array convolution calculation.
  • the processing units PE in the first row and the first column of the systolic array both transmit data from the feature sub-map to the processing unit PE in the lower right corner of each; the processing unit PE in the last row outputs the feature corresponding to the sub-filter pair Convolution calculation is performed on the sub-image, and the convolution step length is 1 convolution result.
  • the weight of the first sub-filter is w1, and the convolution calculation is performed on the first feature sub-image corresponding to it in Figure 4;
  • the weight of the second sub-filter is w2, which is Convolution calculation is performed on the three feature sub-images;
  • the weight of the third sub-filter is w3, and the convolution calculation is performed on the third feature sub-image;
  • the weight of the fourth sub-filter is w4, and the weight of the fourth sub-filter is w4.
  • the results of the convolution calculation corresponding to the first to fourth sub-filters are as follows:
  • Step S150 Superimpose the convolution calculation results corresponding to each of the sub-filters, and output the superimposed result as the result of the convolution calculation of the feature map to be convolved by the convolution filter.
  • the neural network acceleration method based on the systolic array of this embodiment divides a number of sub-filters from the convolution filter according to a preset filter segmentation rule when the convolution step length is not 1, and according to the preset Set the feature map segmentation rule to segment a number of feature sub-maps from the feature map to be convolved.
  • the convolution calculation can be performed with a convolution step length of 1, and the convolution calculation results corresponding to each sub-filter are superimposed.
  • the convolution filter outputs the result of the convolution calculation of the feature map to be convolved for subsequent processing such as reconvolution, pooling, classification, etc.; but because the convolution step length is 1, after the segmentation operation, The computing power of the systolic array can be fully utilized.
  • step S120 if the convolution step size is not 1 and the size of the convolution filter is greater than 1 ⁇ 1, according to a preset filter segmentation rule from the The convolution filter divides into several sub-filters, including:
  • Step S122 If the convolution step size is 2 and the size of the convolution filter is 3 ⁇ 3, 4 sub-filters are divided from the convolution filter, and the size of each sub-filter is 2. ⁇ 2.
  • the first subfilter includes the weights of odd rows and odd columns of the convolution filter
  • the second subfilter includes the weights of odd rows and even columns of the convolution filter
  • the third subfilter includes the weights of the odd rows and even columns of the convolution filter.
  • the fourth sub-filter includes the weights of the even rows and even columns of the convolution filter.
  • the size of the convolution filter kernel cannot divide the convolution stride, and the zero-filled convolution filter can be made by performing zero padding at the preset position of the convolution filter
  • the length or width of is an integer multiple of the convolution step length.
  • the size of the convolution filter is 3 ⁇ 3, and the convolution step size is 2.
  • the size of the convolution filter cannot divide the convolution step size.
  • the zero padding operation can be used to make the zero pad convolution
  • the length or width of the filter is an integer multiple of the convolution step length, so that the convolution filter can divide a number of sub-filters according to a preset filter division rule.
  • each sub-filter is 2 ⁇ 2, and specifically includes:
  • Step S11 Assign weights of odd rows and odd columns of the convolution filter to the first subfilter.
  • Step S12 Assign the weights of the odd rows and even columns of the convolution filter to the first column of the second sub-filter, and fill the second column of the second sub-filter with 0.
  • Step S13 Assign the weights of the even rows and odd columns of the convolution filter to the first row of the third sub-filter, and fill the second row of the third sub-filter with 0.
  • Step S14 Assign the weights of the even rows and even columns of the convolution filter to the first row and first column of the fourth subfilter, and fill the remaining positions of the fourth subfilter with 0.
  • step S130 obtains the feature map to be convolved and performs The feature map segmentation rules of to segment several feature sub-maps from the feature map to be convolved, which specifically include:
  • Step S1321 Allocate the values of odd rows and odd columns of the feature map to be convolved to the corresponding positions of the first feature submap.
  • Step S1322 assign the values of odd rows and even columns of the feature map to be convolved to the corresponding positions of the second feature submap.
  • Step S1323 Assign the values of the even rows and odd columns of the feature map to be convolved to the corresponding positions of the third feature submap.
  • Step S1324 Assign the values of the even rows and even columns of the feature map to be convolved to the corresponding positions of the fourth feature submap.
  • step S120 if the convolution step size is not 1 and the size of the convolution filter is greater than 1 ⁇ 1, according to the preset filter division rule
  • the convolution filter divides into several sub-filters, which specifically include:
  • Step S123 If the convolution step size is 3 and the size of the convolution filter is 3 ⁇ 3, 9 sub-filters are divided from the convolution filter, and the size of each sub-filter is 1. ⁇ 1 and each includes one of the 9 weights of the convolution filter.
  • step S130 obtains the feature map to be convolved and performs The feature map segmentation rules of to segment several feature sub-maps from the feature map to be convolved, which specifically include:
  • Step S1331 Assign the value of the 3n+1th row and 3n+1th column of the feature map to be convolved to the corresponding position of the first feature submap.
  • n is a natural number.
  • Step S1332 assign the value in the 3n+1th row and 3n+2th column of the feature map to be convolved to the corresponding position of the second feature submap.
  • Step S1333 Assign the values in the 3n+1th row and 3n+3th column of the feature map to be convolved to the corresponding position of the third feature submap.
  • Step S1334 Assign the value in the 3n+2th row and 3n+1th column of the feature map to be convolved to the corresponding position of the fourth feature submap.
  • Step S1335 Assign the values in the 3n+2th row and 3n+2th column of the feature map to be convolved to the corresponding position of the fifth feature submap.
  • Step S1336 Assign the value in the 3n+2th row and 3n+3th column of the feature map to be convolved to the corresponding position of the sixth feature submap.
  • Step S1337 Assign the values in the 3n+3th row and 3n+1th column of the feature map to be convolved to the corresponding position of the seventh feature submap.
  • Step S1338 Assign the value in the 3n+3th row and 3n+2th column of the feature map to be convolved to the corresponding position of the eighth feature submap.
  • Step S1339 Assign the value in the 3n+3th row and 3n+3th column of the feature map to be convolved to the corresponding position of the ninth feature submap.
  • the length and width of the acquired feature map to be convolved are both 8. If it is not the convolution step size, that is, an integer multiple of 3, the pre-convolution feature map is Set the position to perform zero-padded so that the length or width of the zero-padded feature map to be convolution is an integer multiple of the convolution step; then according to the preset feature map segmentation rule, the zero-padded feature map to be convolved Segment 9 feature sub-maps.
  • the systolic array-based neural network acceleration method of the present application divides several sub-filters from the convolution filter according to the preset filter segmentation rule when the convolution step length is not 1, and according to the preset feature map segmentation rule Segmenting several feature sub-maps from the feature map to be convolved can achieve convolution calculation with a convolution step length of 1. It can be well adapted to some special dedicated deep network accelerators such as FPGA, NPU and other low-level systolic array (Systolic Array) structure, which can save computing resources, and this segmentation method itself is a special computing logic , Can be integrated into various deep learning frameworks.
  • the segmentation transformation method provided in this application does not affect the forward and backward paths of the deep network itself, and because it saves computing resources, it actually improves the speed of training and inference.
  • FIG. 18 is a schematic diagram of the down-sampling topology of the traditional deep convolutional neural network ResNet50 according to the systolic array-based neural network acceleration method of the present application; the left side of the arrow is the traditional A simplified model of the down-sampled topology structure of the deep convolutional neural network ResNet50. The left side of the arrow is the calculated topology after the equivalent transformation of the segmentation transformation.
  • the calculation topology after the equivalent transformation of the segmentation transformation has the following advantages: 1. Eliminates the two 1 ⁇ 1 mapping convolutions on the left side of the traditional ResNet50 , Reduce computing resources. 2.
  • the residual component part on the right side of the traditional ResNet50 can be converted into a direct identity mapping (Identity Mapping), which is conducive to the propagation of residuals.
  • the systolic array-based neural network acceleration method of this application can be applied to many network models For example, densNet, or shakeshake network, etc., as long as the network is stored in the down-sampling part, the neural network acceleration method provided in this application can be used for transformation, and then calculation training can be performed.
  • FIG. 19 is a schematic structural diagram of a systolic array-based neural network acceleration device provided by an embodiment of the present application.
  • the systolic array-based neural network acceleration device can be configured in a server to execute the aforementioned systolic array The neural network acceleration method.
  • the systolic array-based neural network acceleration device includes:
  • the convolution parameter obtaining module 110 is configured to obtain convolution parameters of a convolution filter, where the convolution parameters include a convolution step size and a size of the convolution filter.
  • the filter segmentation module 120 is configured to: if the convolution step length is not 1 and the size of the convolution filter is greater than 1 ⁇ 1, segment the convolution filter according to a preset filter segmentation rule. Sub-filters, the size of each sub-filter is smaller than the size of the convolution filter.
  • the feature map segmentation module 130 is configured to obtain a feature map to be convolved and segment a number of feature sub-maps from the feature map to be convolved according to a preset feature map segmentation rule, the number of feature sub-maps and the number of sub-filters One to one correspondence.
  • the convolution module 140 is configured to perform convolution calculation on the corresponding feature sub-maps according to each of the sub-filters based on the systolic array, and the step size of the convolution calculation is 1.
  • the superposition module 150 is configured to superimpose the convolution calculation results corresponding to each of the sub-filters, and output the superimposition result as the convolution calculation result of the feature map to be convolved by the convolution filter.
  • the feature map segmentation module 130 includes:
  • the feature map acquiring sub-module 131 is used to acquire the feature map to be convolved.
  • the zero padding sub-module 132 is configured to, if the length or width of the acquired feature map to be convolved is not an integer multiple of the convolution step length, perform zero padding on the preset position of the feature map to be convolved to make zero padding
  • the length or width of the subsequent feature map to be convolved is an integer multiple of the convolution step length.
  • the feature map segmentation submodule 133 is used to segment a number of feature submaps from the feature map to be convolved after zero padding according to preset feature map segmentation rules.
  • the convolution module 140 includes:
  • the weight loading sub-module 141 is used to load the weight of the sub-filter into the weight register connected to the systolic array;
  • the sub-picture loading sub-module 142 is used to load the characteristic sub-picture corresponding to the sub-filter to the input register connected to the systolic array.
  • the output sub-module 143 is used to obtain the output result of the systolic array convolution calculation.
  • the filter segmentation module 120 includes a first filter segmentation sub-module 121, which is used if the convolution step size is 2 and the size of the convolution filter is 2 ⁇ 2.
  • Divide 4 sub-filters from the convolution filter each of the sub-filters has a size of 1 ⁇ 1; wherein the first sub-filter includes the weights of odd rows and odd columns of the convolution filter , The second sub-filter includes the weights of the odd-numbered rows and even columns of the convolution filter, the third sub-filter includes the weights of the even-numbered rows and odd columns of the convolution filter, and the fourth sub-filter includes the weights of the convolution filter. The weights of the even rows and even columns of the product filter.
  • the feature map segmentation module 130 includes a first feature map segmentation sub-module 1301, which is configured to, if the convolution step size is 2 and the size of the convolution filter is 2 ⁇ 2, divide the odd rows of the feature map to be convolution
  • the values of the odd columns are assigned to the corresponding positions of the first feature submap
  • the values of the odd rows and the even columns of the feature map to be convolved are assigned to the corresponding positions of the second feature submap
  • the feature map to be convolved is assigned
  • the values of even rows and odd columns are allocated to the corresponding positions of the third feature submap
  • the values of even rows and even columns of the feature map to be convolved are allocated to the corresponding positions of the fourth feature submap.
  • the filter segmentation module 120 includes a second filter segmentation sub-module 122, which is used if the convolution step size is 2 and the size of the convolution filter is 3 ⁇ 3.
  • Divide 4 sub-filters from the convolution filter each of the sub-filters has a size of 2 ⁇ 2; wherein the first sub-filter includes the weights of odd rows and odd columns of the convolution filter , The second sub-filter includes the weights of the odd-numbered rows and even columns of the convolution filter, the third sub-filter includes the weights of the even-numbered rows and odd columns of the convolution filter, and the fourth sub-filter includes the weights of the convolution filter. The weights of the even rows and even columns of the product filter.
  • the feature map segmentation module 130 includes a second feature map segmentation sub-module 1302, which is used to divide the odd-numbered rows of the feature map to be convolved if the convolution step size is 2 and the size of the convolution filter is 3 ⁇ 3
  • the values of the odd columns are assigned to the corresponding positions of the first feature submap
  • the values of the odd rows and the even columns of the feature map to be convolved are assigned to the corresponding positions of the second feature submap
  • the feature map to be convolved is assigned
  • the values of even rows and odd columns are allocated to the corresponding positions of the third feature submap
  • the values of even rows and even columns of the feature map to be convolved are allocated to the corresponding positions of the fourth feature submap.
  • the filter segmentation module 120 includes a third filter segmentation sub-module 123, which is used if the convolution step size is 3 and the size of the convolution filter is 3 ⁇ 3. Separate 9 sub-filters from the convolution filter, each of the sub-filters has a size of 1 ⁇ 1 and each includes one of the 9 weights of the convolution filter.
  • the feature map segmentation module 130 includes a third feature map segmentation sub-module 1303, which is configured to: if the convolution step size is 3 and the size of the convolution filter is 3 ⁇ 3, the 3nth feature map to be convolved Assign the value in row 3n+1 and column 3n+1 to the corresponding position of the first feature submap, and assign the value in row 3n+1 and column 3n+2 of the feature map to be convolved to the second feature submap Assign the value in row 3n+1, column 3n+3 of the feature map to be convolved to the corresponding position of the third feature submap, and assign the feature map to be convolved to row 3n+2, row 3n+3.
  • the value in column 3n+1 is assigned to the corresponding position of the fourth feature submap, and the value in row 3n+2 and column 3n+2 of the feature map to be convolved is assigned to the corresponding position of the fifth feature submap, Assign the value in row 3n+2, column 3n+3 of the feature map to be convolved to the corresponding position of the sixth feature submap, and assign the feature map to be convolved in row 3n+3, column 3n+1
  • the value of is assigned to the corresponding position of the seventh feature submap, the value of the 3n+3 row 3n+2 column of the feature map to be convolved is assigned to the corresponding position of the eighth feature submap, and the The value in the 3n+3 row and 3n+3 column of the convolution feature map is assigned to the corresponding position of the ninth feature sub-map, where n is a natural number.
  • the method and device of this application can be used in many general or special computing system environments or configurations.
  • the foregoing method and apparatus may be implemented in the form of a computer program, and the computer program may run on the computer device as shown in FIG. 21.
  • FIG. 21 is a schematic structural diagram of a computer device according to an embodiment of the present application.
  • the computer equipment can be a server or a terminal.
  • the computer device includes a processor, a memory, and a network interface connected through a system bus, where the memory may include a non-volatile storage medium and an internal memory.
  • the non-volatile storage medium can store an operating system and a computer program.
  • the computer program includes program instructions.
  • the processor can execute any neural network acceleration method based on a systolic array.
  • the processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.
  • the internal memory provides an environment for the operation of the computer program in the non-volatile storage medium.
  • the processor can execute any neural network acceleration method based on the systolic array.
  • the network interface is used for network communication, such as sending assigned tasks.
  • the structure of the computer device is only a block diagram of a part of the structure related to the solution of the application, and does not constitute a limitation on the computer device to which the solution of the application is applied.
  • the specific computer device may include More or fewer components are shown in the figure, or some components are combined, or have different component arrangements.
  • the processor may be a central processing unit (Central Processing Unit, CPU), the processor may also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), and application specific integrated circuits (Application Specific Integrated Circuits). Circuit, ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor.
  • the processor is used to run a computer program stored in a memory to implement the steps of the aforementioned method for accelerating a neural network based on a systolic array.
  • the processor is configured to run a computer program stored in the memory to implement the following steps:
  • the convolution step length is not 1 and the size of the convolution filter is greater than 1 ⁇ 1, a number of sub-filters are segmented from the convolution filter according to a preset filter segmentation rule, and each sub-filter is The size of the filter is smaller than the size of the convolution filter;
  • the convolution calculation results corresponding to each of the subfilters are superimposed, and the superimposed result is output as the convolution calculation result of the feature map to be convolved by the convolution filter.
  • a computer-readable storage medium stores a computer program
  • the computer program includes program instructions
  • the processor executes the program instructions to implement any item provided in the embodiments of this application based on Neural network acceleration method of systolic array.
  • the computer-readable storage medium may be the internal storage unit of the computer device described in the foregoing embodiment, such as the hard disk or memory of the computer device.
  • the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SMC), or a secure digital (Secure Digital, SD) equipped on the computer device. ) Card, Flash Card, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Neurology (AREA)
  • Image Analysis (AREA)
  • Complex Calculations (AREA)

Abstract

一种基于脉动阵列的神经网络加速方法、装置、计算机设备及存储介质，该方法包括：获取卷积滤波器的参数；根据滤波器分割规则从卷积滤波器分割出若干子滤波器；根据特征图分割规则从待卷积特征图分割出若干特征子图；基于脉动阵列，根据各子滤波器对对应的特征子图进行卷积计算；将各子滤波器对应的卷积计算结果进行叠加。

Description

基于脉动阵列的神经网络加速方法、装置、计算机设备及存储介质
本申请要求于2019年4月4日提交中国专利局、申请号为201910268881.8、发明名称为“基于脉动阵列的神经网络加速方法、装置、计算机设备及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及神经网络技术领域，尤其涉及一种基于脉动阵列的神经网络加速方法、装置、计算机设备及存储介质。
背景技术
常用的神经网络最主要的部分就是卷积的计算，卷积计算中常常会碰到卷积滤波器不等于1的情况，在这种情况下，主流的一些神经网络计算库，例如CUDNN(NVIDIA的深度网络计算库)在计算这种卷积的时候会显著的变慢。一些深度学习加速器例如现场可编程门阵列(Field-Programmable Gate Array，FPGA)，专用网络处理器(network process units，NPU)等在卷积部分通常会用脉动阵列的结构实现，此种结构对于卷积滤波器的卷积滤波器不等于1的情况非常不友好。
现有技术一般通过先计算卷积步长为1时的卷积结果，然后下采样丢弃不需要的卷积结果以得到特定卷积步长的特征图，这样做显然会浪费计算和调度资源，同样会使得卷积计算变慢。
发明内容
本申请实施例提供一种基于脉动阵列的神经网络加速方法、装置、计算机设备及存储介质，能够较佳地解决步长不为1的卷积计算会浪费脉动阵列计算和调度资源的问题。
第一方面，本申请提供了一种基于脉动阵列的神经网络加速方法，所述方法包括：
获取卷积滤波器的卷积参数，所述卷积参数包括卷积步长和所述卷积滤波器的尺寸；
若所述卷积步长不为1且所述卷积滤波器的尺寸大于1×1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器，各所述子滤波器的尺寸小于所述卷积滤波器的尺寸；
获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，所述若干特征子图与所述若干子滤波器一一对应；
基于脉动阵列，根据各所述子滤波器对各自对应的特征子图进行卷积计算，卷积计算的步长为1；
将各所述子滤波器对应的卷积计算结果进行叠加，并将叠加的结果作为所述卷积滤波器对所述待卷积特征图卷积计算的结果进行输出。
第二方面，本申请提供了一种基于脉动阵列的神经网络加速装置，所述装置包括：
卷积参数获取模块，用于获取卷积滤波器的卷积参数，所述卷积参数包括卷积步长和所述卷积滤波器的尺寸；
滤波器分割模块，用于若所述卷积步长不为1且所述卷积滤波器的尺寸大于1×1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器，各所述子滤波器的尺寸小于所述卷积滤波器的尺寸；
特征图分割模块，用于获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，所述若干特征子图与所述若干子滤波器一一对应；
卷积模块，用于基于脉动阵列，根据各所述子滤波器对各自对应的特征子图进行卷积计算，卷积计算的步长为1；
叠加模块，用于将各所述子滤波器对应的卷积计算结果进行叠加，并将叠加的结果作为 所述卷积滤波器对所述待卷积特征图卷积计算的结果进行输出。
第三方面，本申请提供了一种计算机设备，所述计算机设备包括存储器和处理器；所述存储器用于存储计算机程序；所述处理器，用于执行所述计算机程序并在执行所述计算机程序时实现上述的基于脉动阵列的神经网络加速方法。
第四方面，本申请提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，若所述计算机程序被处理器执行，实现上述的基于脉动阵列的神经网络加速方法。
本申请公开了一种基于脉动阵列的神经网络加速方法、装置、设备及存储介质，通过在卷积步长不为1时根据预设的滤波器分割规则从卷积滤波器分割出若干子滤波器，以及根据预设的特征图分割规则从待卷积特征图分割出若干特征子图实现可以以卷积步长为1执行卷积计算，各子滤波器对应的卷积计算结果进行叠加后的叠加结果与根据原卷积滤波器对待卷积特征图执行的卷积步长不为1的卷积计算结果相同，即分割操作前后的两种卷积计算是等价的；但是由于分割操作后卷积步长为1，可以更充分的利用脉动阵列的计算能力。
附图说明
为了更清楚地说明本申请实施例的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本申请一实施例的基于脉动阵列的神经网络加速方法的流程示意图；
图2为本申请另一实施例的基于脉动阵列的神经网络加速方法的流程示意图；
图3为卷积步长为2且卷积滤波器尺寸为2×2时分割卷积滤波器的示意图；
图4为卷积步长为2且卷积滤波器尺寸为2×2时分割待卷积特征图的示意图；
图5为图1中分割待卷积特征图一实施方式的子流程示意图；
图6为分割补零后的待卷积特征图的示意图；
图7为脉动阵列的结构示意图；
图8为脉动阵列进行卷积计算的示意图；
图9为图1中基于脉动阵列进行卷积计算的子流程示意图；
图10为本申请再一实施例的基于脉动阵列的神经网络加速方法的流程示意图；
图11为卷积步长为2且卷积滤波器尺寸为3×3时分割卷积滤波器的示意图；
图12为卷积步长为2且卷积滤波器尺寸为3×3时分割卷积滤波器的子流程示意图；
图13为卷积步长为2且卷积滤波器尺寸为3×3时分割待卷积特征图的示意图；
图14为本申请又一实施例的基于脉动阵列的神经网络加速方法的流程示意图；
图15为卷积步长为3且卷积滤波器尺寸为3×3时分割卷积滤波器的示意图；
图16为卷积步长为3且卷积滤波器尺寸为3×3时分割待卷积特征图的子流程示意图；
图17为卷积步长为3且卷积滤波器尺寸为3×3时分割待卷积特征图的示意图；
图18为根据神经网络加速方法对深度卷积神经网络下采样的拓扑结构进行等价变换的示意图；
图19为本申请实施例的基于脉动阵列的神经网络加速装置的结构示意图；
图20为本申请另一实施例的基于脉动阵列的神经网络加速装置的结构示意图；
图21为本申请一实施例提供的一种计算机设备的结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
附图中所示的流程图仅是示例说明，不是必须包括所有的内容和操作/步骤，也不是必须按所描述的顺序执行。例如，有的操作/步骤还可以分解、组合或部分合并，因此实际执行的顺序有可能根据实际情况改变。另外，虽然在装置示意图中进行了功能模块的划分，但是在某些情况下，可以以不同于装置示意图中的模块划分。
本申请的实施例提供了一种基于脉动阵列的神经网络加速方法、装置、设备及存储介质。其中，该基于脉动阵列的神经网络加速方法可以应用于终端或服务器中，以实现加速基于脉动阵列的神经网络的训练或推理。
下面结合附图，对本申请的一些实施方式作详细说明。在不冲突的情况下，下述的实施例及实施例中的特征可以相互组合。
请参阅图1，图1是本申请的实施例提供的一种基于脉动阵列的神经网络加速方法的流程示意图。
如图1所示，基于脉动阵列的神经网络加速方法包括以下步骤：
步骤S110、获取卷积滤波器的卷积参数。
其中，所述卷积参数包括卷积步长和所述卷积滤波器的尺寸。
滤波器filter，又称内核kernel、特征检测器(feature detector)，在输入的图像或特征图上滑动滤波器并计算点乘即为卷积操作，卷积操作的输出的矩阵叫做卷积特征(Convolved Feature)、激活图(Activation Map)或者特征图(Feature Map)。
示例性的，在神经网络中的神经元进行卷积操作前，先获取预先存储的或者初始化的卷积滤波器的卷积参数。
在本实施例中，卷积参数包括卷积步长stride和所述卷积滤波器的尺寸，即卷积滤波器的高度h和宽度w；在另一些实施例中，卷积参数还包括输入通道数和/或输出通道数；其中输入通道数in depth是由待卷积特征图的通道数所决定的，输出通道数out depth等于卷积滤波器的个数，可以决定卷积结束后输出特征图的通道数。
基于脉动阵列的神经网络加速方法可以用于输入通道数等于1或大于1的场景，也可用于输出通道数等于1或大于1的场景。
步骤S120、若所述卷积步长不为1且所述卷积滤波器的尺寸大于1×1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器。
其中，各所述子滤波器的尺寸小于所述卷积滤波器的尺寸。
一些深度学习加速器例如FPGA，专用NPU等在卷积部分通常会用脉动阵列的结构实现，但是这种结构对于卷积步长不等于1的情况非常不友好；本实施例将尺寸大于1×1卷积滤波器分割为若干子滤波器，以使得各子滤波器分别以等于1的卷积步长执行卷积操作，从而充分利用脉动阵列结构的性能。
在一些实施例中，如图2和图3所示，步骤S120若所述卷积步长不为1且所述卷积滤波器的尺寸大于1×1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器，具体包括：
步骤S121、若所述卷积步长为2且所述卷积滤波器的尺寸为2×2，从所述卷积滤波器分割出4个子滤波器，各所述子滤波器的尺寸为1×1。
如图3所示，某卷积操作对应的卷积参数为[6 6 2 2]，即输入通道数in depth为6，输出通道数out depth等于6，6个卷积滤波器Kernel Tenseor的尺寸均为2×2。
如图3所示，将每个2×2的卷积滤波器分别分割为4个1×1的子滤波器。以第一个卷积滤波器分割的4个子滤波器为例，其中第一个子滤波器包括所述卷积滤波器奇数行奇数列的权值w1，第二个子滤波器包括所述卷积滤波器奇数行偶数列的权值w2，第三个子滤波器包括所述卷积滤波器偶数行奇数列的权值w3，第四个子滤波器包括所述卷积滤波器偶数行偶数列的权值w4。
具体的，将卷积滤波器第一行第一列的权值分配给第一个1×1的子滤波器，将卷积滤波 器第一行第二列的权值分配给第二个1×1的子滤波器，将卷积滤波器第二行第一列的权值分配给第三个1×1的子滤波器，将卷积滤波器第二行第二列的权值分配给第四个1×1的子滤波器。
步骤S130、获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图。
其中，所述若干特征子图与所述若干子滤波器一一对应。
示例性的，待卷积特征图的通道数可以等于1或大于1，待卷积特征图的通道数可以决定相应卷积滤波器卷积参数中的输入通道数。
在一些实施例中，如图2和图4所示，若所述卷积步长为2且所述卷积滤波器的尺寸为2×2，步骤S130获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，具体包括：
步骤S1311、将所述待卷积特征图奇数行奇数列的数值分配至第一个特征子图的相应位置。
步骤S1312、将所述待卷积特征图奇数行偶数列的数值分配至第二个特征子图的相应位置。
步骤S1313、将所述待卷积特征图偶数行奇数列的数值分配至第三个特征子图的相应位置。
步骤S1314、将所述待卷积特征图偶数行偶数列的数值分配至第四个特征子图的相应位置。
示例性的，待卷积特征图中同一行的数值在各特征子图中也位于同一行，待卷积特征图中同一列的数值在各特征子图中也位于同一列。
如图4所示，获取到的卷积特征图input Tensor是[1 6 4 4]的卷积特征图，该待卷积特征图的通道数为6，宽和高均为4。根据预设的分割特征从所述待卷积特征图分割出了4个特征子图。
在一些实施例中，如图5所示，步骤S130获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图具体包括：
步骤S131、获取待卷积特征图。
示例性的，获取的卷积特征图如图6所示。
步骤S132、若获取的待卷积特征图的长或宽不是所述卷积步长的整数倍，对所述待卷积特征图的预设位置进行补零以使补零后的待卷积特征图的长或宽是所述卷积步长的整数倍。
在本实施例中，获取的卷积特征图的长、宽均为3，示例性的在该卷积特征图的右侧和下方补零padding，使得补零后的待卷积特征图的长、宽为4。
步骤S133、根据预设的特征图分割规则从补零后的待卷积特征图分割出若干特征子图。
如图6所示，根据上述步骤S1311-步骤S1314从补零后的待卷积特征图分割出了4个特征子图。
在一些实施例中，待卷积特征图的结构为NCHW，如[1 6 4 4]，N代表数量，C代表通道数channel，H代表高度，W代表宽度；因此实例数batchsize＝1，通道数channel＝6，待卷积特征图的高度H＝4，待卷积特征图的宽度W＝4时，即待卷积张量的数量为1，有6个通道，各通道均为一个待卷积特征图。可以根据先分割、卷积同一数量编号下不同通道的待卷积特征图，然后分割、卷积下一数量编号下不同通道的待卷积特征图。
步骤S140、基于脉动阵列，根据各所述子滤波器对各自对应的特征子图进行卷积计算，卷积计算的步长为1。
脉动阵列(Systolic Array)核心概念就是让数据在运算单元的阵列中进行流动，减少访存的次数，并且使得结构更加规整，布线更加统一，提高频率。
在一些实施例中，如图7所示，脉动阵列Systolic Array包括L×L个处理单元PE，所述 脉动阵列连接于权值寄存器filter buffer、输入寄存器in buffer和输出寄存器out buffer。每行处理单元PE的左侧、每列处理单元PE的上侧都设有先进先出寄存器FIFO。滤波器的权值通过先进先出寄存器FIFO存储并传输给同一行的所有处理单元PE，第一行和第一列的处理单元PE接收来自输入寄存器中待卷积特征图的数据，并且第一行和第一列的处理单元PE均向各自右下角的处理单元PE传输来自待卷积特征图的数据。这样的设计最大化了数据的复用。
示例性的，如图8所示，脉动阵列根据一个3×3滤波器W对一个5×5的特征图X进行二维卷积。
假设滤波器W和特征图X有以下形式：
Figure PCTCN2019103137-appb-000001
其中，wi和xj分别代表滤波器W和特征图X的某一行数据，则最后一行的三个处理单元PE输出三行卷积结果：
Figure PCTCN2019103137-appb-000002
其中*表示一维卷积计算。
在一些实施例中，如图9所示，步骤S140基于脉动阵列，根据各所述子滤波器对各自对应的特征子图进行卷积计算，具体包括：
步骤S141、将所述子滤波器的权值加载至连接于所述脉动阵列的权值寄存器中。
示例性的，将子滤波器的权值加载至权值寄存器filter buffer中，子滤波器的权值通过先进先出寄存器FIFO存储并传输给同一行的处理单元PE。
步骤S142、将所述子滤波器对应的特征子图加载至连接于所述脉动阵列的输入寄存器中。
示例性的，将与子滤波器对应的特征子图加载至输入寄存器in buffer中，脉动阵列第一行和第一列的处理单元PE接收来自输入寄存器in buffer中特征子图的数据。
步骤S143、获取所述脉动阵列卷积计算后的输出结果。
示例性的，脉动阵列第一行和第一列的处理单元PE均向各自右下角的处理单元PE传输来自特征子图的数据；最后一行的处理单元PE输出所述子滤波器对对应的特征子图进行卷积计算，卷积步长为1的卷积结果。
如图3所示，第一个子滤波器的权值为w1，对图4中与其对应的第一个特征子图进行卷积计算；第二个子滤波器的权值为w2，对第二个特征子图进行卷积计算；第三个子滤波器的权值为w3，对第三个特征子图进行卷积计算；第四个子滤波器的权值为w4，对第四个特征子图进行卷积计算。第一至第四个子滤波器对应的卷积计算的结果如下：
Figure PCTCN2019103137-appb-000003
步骤S150、将各所述子滤波器对应的卷积计算结果进行叠加，并将叠加的结果作为所述卷积滤波器对所述待卷积特征图卷积计算的结果进行输出。
示例性的，将4个子滤波器对应的卷积计算结果进行叠加，得到：
Figure PCTCN2019103137-appb-000004
如果直接根据图3左侧的卷积滤波器对图4左侧的卷积特征图以卷积步长为2进行卷积计算，卷积计算的结果为：
Figure PCTCN2019103137-appb-000005
因此，本实施例的基于脉动阵列的神经网络加速方法，通过在卷积步长不为1时根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器，以及根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图实现可以以卷积步长为1执行卷积计算，各子滤波器对应的卷积计算结果进行叠加后的叠加结果与根据原卷积滤波器对待卷积特征图执行的卷积步长不为1的卷积计算结果相同，即分割操作前后的两种卷积计算是等价的；因此叠加的结果可以作为所述卷积滤波器对所述待卷积特征图卷积计算的结果进行输出，以进行后续的处理如再一次卷积、池化、分类等；但是由于分割操作后卷积步长为1，可以更充分的利用脉动阵列的计算能力。
示例性的，如图10和图11所示，步骤S120若所述卷积步长不为1且所述卷积滤波器的尺寸大于1×1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器，具体包括：
步骤S122、若所述卷积步长为2且所述卷积滤波器的尺寸为3×3，从所述卷积滤波器分割出4个子滤波器，各所述子滤波器的尺寸为2×2。
其中第一个子滤波器包括所述卷积滤波器奇数行奇数列的权值，第二个子滤波器包括所述卷积滤波器奇数行偶数列的权值，第三个子滤波器包括所述卷积滤波器偶数行奇数列的权值，第四个子滤波器包括所述卷积滤波器偶数行偶数列的权值。
在一些实施例中，卷积滤波器kernel(filter)的尺寸无法整除卷积步长(stride)，可以通过在卷积滤波器的预设位置进行补零以使补零后的卷积滤波器的长或宽是所述卷积步长的整数倍。在本实施例中，卷积滤波器的尺寸为3×3，卷积步长为2，卷积滤波器的尺寸无法整除卷积步长，可以通过补零操作以使补零后的卷积滤波器的长或宽是所述卷积步长的整数倍，从而使卷积滤波器可以根据预设的滤波器分割规则分割出若干子滤波器。
具体的，如图11和图12所示，所述若所述卷积步长为2且所述卷积滤波器的尺寸为3×3，从所述卷积滤波器分割出4个子滤波器，各所述子滤波器的尺寸为2×2，具体包括：
步骤S11、将所述卷积滤波器奇数行奇数列的权值分配至第一个子滤波器。
步骤S12、将所述卷积滤波器奇数行偶数列的权值分配至第二个子滤波器的第一列，并以0填充所述第二个子滤波器的第二列。
步骤S13、将所述卷积滤波器偶数行奇数列的权值分配至第三个子滤波器的第一行，并以0填充所述第三个子滤波器的第二行。
步骤S14、将所述卷积滤波器偶数行偶数列的权值分配至第四个子滤波器的第一行第一列，并以0填充所述第四个子滤波器的其余位置。
在本实施例中，如图10和图13所示，若所述卷积步长为2且所述卷积滤波器的尺寸为3×3，步骤S130获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，具体包括：
步骤S1321、将所述待卷积特征图奇数行奇数列的数值分配至第一个特征子图的相应位置。
步骤S1322、将所述待卷积特征图奇数行偶数列的数值分配至第二个特征子图的相应位置。
步骤S1323、将所述待卷积特征图偶数行奇数列的数值分配至第三个特征子图的相应位置。
步骤S1324、将所述待卷积特征图偶数行偶数列的数值分配至第四个特征子图的相应位置。
卷积步长为2且所述卷积滤波器的尺寸为3×3时，根据预设的特征图分割规则从所述待卷积特征图分割出4个特征子图；如果待卷积特征图的通道数为1，则分割后的通道数为4。
在一些实施例中，如图14和图15所示，步骤S120若所述卷积步长不为1且所述卷积滤波器的尺寸大于1×1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器，具体包括：
步骤S123、若所述卷积步长为3且所述卷积滤波器的尺寸为3×3，从所述卷积滤波器分割出9个子滤波器，各所述子滤波器的尺寸为1×1且分别包括所述卷积滤波器9个权值中的一个。
示例性的，将卷积滤波器第一行第一列的权值分配给第一个1×1的子滤波器，将卷积滤波器第一行第二列的权值分配给第二个1×1的子滤波器，将卷积滤波器第一行第三列的权值分配给第三个1×1的子滤波器，将卷积滤波器第二行第一列的权值分配给第四个1×1的子滤波器，将卷积滤波器第二行第二列的权值分配给第五个1×1的子滤波器，以此类推。
在本实施例中，如图16和图17所示，若所述卷积步长为3且所述卷积滤波器的尺寸为3×3，步骤S130获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，具体包括：
步骤S1331、将所述待卷积特征图第3n+1行第3n+1列的数值分配至第一个特征子图的相应位置。其中n为自然数。
步骤S1332、将所述待卷积特征图第3n+1行第3n+2列的数值分配至第二个特征子图的相应位置。
步骤S1333、将所述待卷积特征图第3n+1行第3n+3列的数值分配至第三个特征子图的相应位置。
步骤S1334、将所述待卷积特征图第3n+2行第3n+1列的数值分配至第四个特征子图的相应位置。
步骤S1335、将所述待卷积特征图第3n+2行第3n+2列的数值分配至第五个特征子图的相应位置。
步骤S1336、将所述待卷积特征图第3n+2行第3n+3列的数值分配至第六个特征子图的相应位置。
步骤S1337、将所述待卷积特征图第3n+3行第3n+1列的数值分配至第七个特征子图的相应位置。
步骤S1338、将所述待卷积特征图第3n+3行第3n+2列的数值分配至第八个特征子图的相应位置。
步骤S1339、将所述待卷积特征图第3n+3行第3n+3列的数值分配至第九个特征子图的相应位置。
示例性的，如图17所示，获取到的待卷积特征图的长宽均为8，不是所述卷积步长，即3的整数倍，则对所述待卷积特征图的预设位置进行补零以使补零后的待卷积特征图的长或宽是所述卷积步长的整数倍；然后根据预设的特征图分割规则从补零后的待卷积特征图分割出9个特征子图。
本申请的基于脉动阵列的神经网络加速方法通过在卷积步长不为1时根据预设的滤波器分割规则从卷积滤波器分割出若干子滤波器，以及根据预设的特征图分割规则从待卷积特征图分割出若干特征子图实现可以以卷积步长为1执行卷积计算。可以很好的与一些特殊的专用深度网络加速器例如FPGA，NPU等底层都会采用的脉动阵列(Systolic Array)的结构适 配，可以节省计算资源，而且这种分割方法本身就是一种特殊的计算逻辑，可以集成进入各种深度学习的框架中。本申请提供的分割变换方法，并不影响深度网络本身的前传和后传路径，而且因为节省了计算资源，实际上提升了训练和推理的速度。
在一些实施例中，如图18所示为根据本申请的基于脉动阵列的神经网络加速方法对传统的深度卷积神经网络ResNet50的下采样的拓扑结构进行分割变换的示意图；箭头左侧为传统的深度卷积神经网络ResNet50的下采样的拓扑结构部分的简化模型，箭头左侧为经过分割变换等价变换后的计算拓扑结构。
相较于传统的深度卷积神经网络的计算图结构，经过分割变换等价变换后的计算拓扑结构如下的优势：1.省去了传统ResNet50在左侧的两个1×1的映射卷积，减少计算资源。2.传统ResNet50右侧的残差分量部分，可以转换为直接恒等映射((Identity Mapping)，有利于残差的传播。本申请的基于脉动阵列的神经网络加速方法可以应用在很多网络模型中，例如densNet，或者shakeshake网络等等，只要存于在下采样部分的网络，都可以采用本申请提供的神经网络加速方法做变换后，再进行计算训练等。
请参阅图19，图19是本申请一实施例提供的基于脉动阵列的神经网络加速装置的结构示意图，该基于脉动阵列的神经网络加速装置可以配置于服务器中，用于执行前述的基于脉动阵列的神经网络加速方法。
如图19所示，该基于脉动阵列的神经网络加速装置，包括：
卷积参数获取模块110，用于获取卷积滤波器的卷积参数，所述卷积参数包括卷积步长和所述卷积滤波器的尺寸。
滤波器分割模块120，用于若所述卷积步长不为1且所述卷积滤波器的尺寸大于1×1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器，各所述子滤波器的尺寸小于所述卷积滤波器的尺寸。
特征图分割模块130，用于获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，所述若干特征子图与所述若干子滤波器一一对应。
卷积模块140，用于基于脉动阵列，根据各所述子滤波器对各自对应的特征子图进行卷积计算，卷积计算的步长为1。
叠加模块150，用于将各所述子滤波器对应的卷积计算结果进行叠加，并将叠加的结果作为所述卷积滤波器对所述待卷积特征图卷积计算的结果进行输出。
在一些实施例中，如图20所示，特征图分割模块130包括：
特征图获取子模块131，用于获取待卷积特征图。
补零子模块132，用于若获取的待卷积特征图的长或宽不是所述卷积步长的整数倍，对所述待卷积特征图的预设位置进行补零以使补零后的待卷积特征图的长或宽是所述卷积步长的整数倍。
特征图分割子模块133，用于根据预设的特征图分割规则从补零后的待卷积特征图分割出若干特征子图。
在一些实施例中，如图20所示，卷积模块140包括：
权值加载子模块141，用于将所述子滤波器的权值加载至连接于所述脉动阵列的权值寄存器中；
子图加载子模块142，用于将所述子滤波器对应的特征子图加载至连接于所述脉动阵列的输入寄存器中。
输出子模块143，用于获取所述脉动阵列卷积计算后的输出结果。
在一些实施例中，如图20所示，滤波器分割模块120包括第一滤波器分割子模块121，用于若所述卷积步长为2且所述卷积滤波器的尺寸为2×2，从所述卷积滤波器分割出4个子滤波器，各所述子滤波器的尺寸为1×1；其中第一个子滤波器包括所述卷积滤波器奇数行奇数列的权值，第二个子滤波器包括所述卷积滤波器奇数行偶数列的权值，第三个子滤波器包 括所述卷积滤波器偶数行奇数列的权值，第四个子滤波器包括所述卷积滤波器偶数行偶数列的权值。
特征图分割模块130包括第一特征图分割子模块1301，用于若所述卷积步长为2且所述卷积滤波器的尺寸为2×2，将所述待卷积特征图奇数行奇数列的数值分配至第一个特征子图的相应位置，将所述待卷积特征图奇数行偶数列的数值分配至第二个特征子图的相应位置，将所述待卷积特征图偶数行奇数列的数值分配至第三个特征子图的相应位置，将所述待卷积特征图偶数行偶数列的数值分配至第四个特征子图的相应位置。
在一些实施例中，如图20所示，滤波器分割模块120包括第二滤波器分割子模块122，用于若所述卷积步长为2且所述卷积滤波器的尺寸为3×3，从所述卷积滤波器分割出4个子滤波器，各所述子滤波器的尺寸为2×2；其中第一个子滤波器包括所述卷积滤波器奇数行奇数列的权值，第二个子滤波器包括所述卷积滤波器奇数行偶数列的权值，第三个子滤波器包括所述卷积滤波器偶数行奇数列的权值，第四个子滤波器包括所述卷积滤波器偶数行偶数列的权值。
特征图分割模块130包括第二特征图分割子模块1302，用于若所述卷积步长为2且所述卷积滤波器的尺寸为3×3，将所述待卷积特征图奇数行奇数列的数值分配至第一个特征子图的相应位置，将所述待卷积特征图奇数行偶数列的数值分配至第二个特征子图的相应位置，将所述待卷积特征图偶数行奇数列的数值分配至第三个特征子图的相应位置，将所述待卷积特征图偶数行偶数列的数值分配至第四个特征子图的相应位置。
在一些实施例中，如图20所示，滤波器分割模块120包括第三滤波器分割子模块123，用于若所述卷积步长为3且所述卷积滤波器的尺寸为3×3，从所述卷积滤波器分割出9个子滤波器，各所述子滤波器的尺寸为1×1且分别包括所述卷积滤波器9个权值中的一个。
特征图分割模块130包括第三特征图分割子模块1303，用于若所述卷积步长为3且所述卷积滤波器的尺寸为3×3，将所述待卷积特征图第3n+1行第3n+1列的数值分配至第一个特征子图的相应位置，将所述待卷积特征图第3n+1行第3n+2列的数值分配至第二个特征子图的相应位置，将所述待卷积特征图第3n+1行第3n+3列的数值分配至第三个特征子图的相应位置，将所述待卷积特征图第3n+2行第3n+1列的数值分配至第四个特征子图的相应位置，将所述待卷积特征图第3n+2行第3n+2列的数值分配至第五个特征子图的相应位置，将所述待卷积特征图第3n+2行第3n+3列的数值分配至第六个特征子图的相应位置，将所述待卷积特征图第3n+3行第3n+1列的数值分配至第七个特征子图的相应位置，将所述待卷积特征图第3n+3行第3n+2列的数值分配至第八个特征子图的相应位置，将所述待卷积特征图第3n+3行第3n+3列的数值分配至第九个特征子图的相应位置，其中n为自然数。
需要说明的是，所属领域的技术人员可以清楚地了解到，为了描述的方便和简洁，上述描述的装置和各模块、单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
本申请的方法、装置可用于众多通用或专用的计算系统环境或配置中。例如：个人计算机、服务器计算机、手持设备或便携式设备、平板型设备、多处理器系统、基于微处理器的系统、机顶盒、可编程的消费电子设备、网络PC、小型计算机、大型计算机、包括以上任何系统或设备的分布式计算环境等等。
示例性的，上述的方法、装置可以实现为一种计算机程序的形式，该计算机程序可以在如图21所示的计算机设备上运行。
请参阅图21，图21是本申请实施例提供的一种计算机设备的结构示意图。该计算机设备可以是服务器或终端。
参阅图21，该计算机设备包括通过系统总线连接的处理器、存储器和网络接口，其中，存储器可以包括非易失性存储介质和内存储器。
非易失性存储介质可存储操作系统和计算机程序。该计算机程序包括程序指令，该程序 指令被执行时，可使得处理器执行任意一种基于脉动阵列的神经网络加速方法。
处理器用于提供计算和控制能力，支撑整个计算机设备的运行。
内存储器为非易失性存储介质中的计算机程序的运行提供环境，该计算机程序被处理器执行时，可使得处理器执行任意一种基于脉动阵列的神经网络加速方法。
该网络接口用于进行网络通信，如发送分配的任务等。本领域技术人员可以理解，该计算机设备的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的计算机设备的限定，具体的计算机设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。
应当理解的是，处理器可以是中央处理单元(Central Processing Unit，CPU)，该处理器还可以是其他通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。其中，通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
其中，在一个实施例中，所述处理器用于运行存储在存储器中的计算机程序，以实现前述基于脉动阵列的神经网络加速方法的步骤。
示例性的，所述处理器用于运行存储在存储器中的计算机程序，以实现如下步骤：
获取卷积滤波器的卷积参数，所述卷积参数包括卷积步长和所述卷积滤波器的尺寸；
若所述卷积步长不为1且所述卷积滤波器的尺寸大于1×1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器，各所述子滤波器的尺寸小于所述卷积滤波器的尺寸；
获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，所述若干特征子图与所述若干子滤波器一一对应；
基于脉动阵列，根据各所述子滤波器对各自对应的特征子图进行卷积计算，卷积计算的步长为1；
将各所述子滤波器对应的卷积计算结果进行叠加，并将叠加的结果作为所述卷积滤波器对所述待卷积特征图卷积计算的结果进行输出。
通过以上的实施方式的描述可知，本领域的技术人员可以清楚地了解到本申请可借助软件加必需的通用硬件平台的方式来实现。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品可以存储在存储介质中，如ROM/RAM、磁碟、光盘等，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例或者实施例的某些部分所述的方法，如：
一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序中包括程序指令，所述处理器执行所述程序指令，实现本申请实施例提供的任一项基于脉动阵列的神经网络加速方法。
其中，所述计算机可读存储介质可以是前述实施例所述的计算机设备的内部存储单元，例如所述计算机设备的硬盘或内存。所述计算机可读存储介质也可以是所述计算机设备的外部存储设备，例如所述计算机设备上配备的插接式硬盘，智能存储卡(Smart Media Card，SMC)，安全数字(Secure Digital，SD)卡，闪存卡(Flash Card)等。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到各种等效的修改或替换，这些修改或替换都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以权利要求的保护范围为准。

Claims (20)

  1. 一种基于脉动阵列的神经网络加速方法，包括：
    获取卷积滤波器的卷积参数，所述卷积参数包括卷积步长和所述卷积滤波器的尺寸；
    若所述卷积步长不为1且所述卷积滤波器的尺寸大于1×1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器，各所述子滤波器的尺寸小于所述卷积滤波器的尺寸；
    获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，所述若干特征子图与所述若干子滤波器一一对应；
    基于脉动阵列，根据各所述子滤波器对各自对应的特征子图进行卷积计算，卷积计算的步长为1；
    将各所述子滤波器对应的卷积计算结果进行叠加，并将叠加的结果作为所述卷积滤波器对所述待卷积特征图卷积计算的结果进行输出。
  2. 如权利要求1所述的神经网络加速方法，其中，所述获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，具体包括：
    获取待卷积特征图；
    若获取的待卷积特征图的长或宽不是所述卷积步长的整数倍，对所述待卷积特征图的预设位置进行补零以使补零后的待卷积特征图的长或宽是所述卷积步长的整数倍；
    根据预设的特征图分割规则从补零后的待卷积特征图分割出若干特征子图。
  3. 如权利要求2所述的神经网络加速方法，其中，所述若所述卷积步长不为1且所述卷积滤波器的尺寸大于1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器，具体包括：
    若所述卷积步长为2且所述卷积滤波器的尺寸为2×2，从所述卷积滤波器分割出4个子滤波器，各所述子滤波器的尺寸为1×1；其中第一个子滤波器包括所述卷积滤波器奇数行奇数列的权值，第二个子滤波器包括所述卷积滤波器奇数行偶数列的权值，第三个子滤波器包括所述卷积滤波器偶数行奇数列的权值，第四个子滤波器包括所述卷积滤波器偶数行偶数列的权值；
    若所述卷积步长为2且所述卷积滤波器的尺寸为2×2，所述获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，具体包括：
    将所述待卷积特征图奇数行奇数列的数值分配至第一个特征子图的相应位置，将所述待卷积特征图奇数行偶数列的数值分配至第二个特征子图的相应位置，将所述待卷积特征图偶数行奇数列的数值分配至第三个特征子图的相应位置，将所述待卷积特征图偶数行偶数列的数值分配至第四个特征子图的相应位置。
  4. 如权利要求2所述的神经网络加速方法，其中，所述若所述卷积步长不为1且所述卷积滤波器的尺寸大于1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器，具体包括：
    若所述卷积步长为2且所述卷积滤波器的尺寸为3×3，从所述卷积滤波器分割出4个子滤波器，各所述子滤波器的尺寸为2×2；其中第一个子滤波器包括所述卷积滤波器奇数行奇数列的权值，第二个子滤波器包括所述卷积滤波器奇数行偶数列的权值，第三个子滤波器包括所述卷积滤波器偶数行奇数列的权值，第四个子滤波器包括所述卷积滤波器偶数行偶数列的权值；
    若所述卷积步长为2且所述卷积滤波器的尺寸为3×3，所述获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，具体包括：
    将所述待卷积特征图奇数行奇数列的数值分配至第一个特征子图的相应位置，将所述待卷积特征图奇数行偶数列的数值分配至第二个特征子图的相应位置，将所述待卷积特征图偶数行奇数列的数值分配至第三个特征子图的相应位置，将所述待卷积特征图偶数行偶数列的数值分配至第四个特征子图的相应位置。
  5. 如权利要求4所述的神经网络加速方法，其中，所述若所述卷积步长为2且所述卷积 滤波器的尺寸为3×3，从所述卷积滤波器分割出4个子滤波器，各所述子滤波器的尺寸为2×2，具体包括：
    将所述卷积滤波器奇数行奇数列的权值分配至第一个子滤波器；
    将所述卷积滤波器奇数行偶数列的权值分配至第二个子滤波器的第一列，并以0填充所述第二个子滤波器的第二列；
    将所述卷积滤波器偶数行奇数列的权值分配至第三个子滤波器的第一行，并以0填充所述第三个子滤波器的第二行；
    将所述卷积滤波器偶数行偶数列的权值分配至第四个子滤波器的第一行第一列，并以0填充所述第四个子滤波器的其余位置。
  6. 如权利要求2所述的神经网络加速方法，其中，所述若所述卷积步长不为1且所述卷积滤波器的尺寸大于1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器，具体包括：
    若所述卷积步长为3且所述卷积滤波器的尺寸为3×3，从所述卷积滤波器分割出9个子滤波器，各所述子滤波器的尺寸为1×1且分别包括所述卷积滤波器9个权值中的一个；
    若所述卷积步长为3且所述卷积滤波器的尺寸为3×3，所述获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，具体包括：
    将所述待卷积特征图第3n+1行第3n+1列的数值分配至第一个特征子图的相应位置，将所述待卷积特征图第3n+1行第3n+2列的数值分配至第二个特征子图的相应位置，将所述待卷积特征图第3n+1行第3n+3列的数值分配至第三个特征子图的相应位置，将所述待卷积特征图第3n+2行第3n+1列的数值分配至第四个特征子图的相应位置，将所述待卷积特征图第3n+2行第3n+2列的数值分配至第五个特征子图的相应位置，将所述待卷积特征图第3n+2行第3n+3列的数值分配至第六个特征子图的相应位置，将所述待卷积特征图第3n+3行第3n+1列的数值分配至第七个特征子图的相应位置，将所述待卷积特征图第3n+3行第3n+2列的数值分配至第八个特征子图的相应位置，将所述待卷积特征图第3n+3行第3n+3列的数值分配至第九个特征子图的相应位置，其中n为自然数。
  7. 如权利要求1-6中任一项所述的神经网络加速方法，其中：所述基于脉动阵列，根据各所述子滤波器对各自对应的特征子图进行卷积计算，具体包括：
    将所述子滤波器的权值加载至连接于所述脉动阵列的权值寄存器中；
    将所述子滤波器对应的特征子图加载至连接于所述脉动阵列的输入寄存器中；
    获取所述脉动阵列卷积计算后的输出结果。
  8. 一种基于脉动阵列的神经网络加速装置，包括：
    卷积参数获取模块，用于获取卷积滤波器的卷积参数，所述卷积参数包括卷积步长和所述卷积滤波器的尺寸；
    滤波器分割模块，用于若所述卷积步长不为1且所述卷积滤波器的尺寸大于1×1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器，各所述子滤波器的尺寸小于所述卷积滤波器的尺寸；
    特征图分割模块，用于获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，所述若干特征子图与所述若干子滤波器一一对应；
    卷积模块，用于基于脉动阵列，根据各所述子滤波器对各自对应的特征子图进行卷积计算，卷积计算的步长为1；
    叠加模块，用于将各所述子滤波器对应的卷积计算结果进行叠加，并将叠加的结果作为所述卷积滤波器对所述待卷积特征图卷积计算的结果进行输出。
  9. 一种计算机设备，所述计算机设备包括存储器和处理器；
    所述存储器用于存储计算机程序；
    所述处理器，用于执行所述计算机程序并在执行所述计算机程序时实现如下步骤：
    获取卷积滤波器的卷积参数，所述卷积参数包括卷积步长和所述卷积滤波器的尺寸；
    若所述卷积步长不为1且所述卷积滤波器的尺寸大于1×1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器，各所述子滤波器的尺寸小于所述卷积滤波器的尺寸；
    获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，所述若干特征子图与所述若干子滤波器一一对应；
    基于脉动阵列，根据各所述子滤波器对各自对应的特征子图进行卷积计算，卷积计算的步长为1；
    将各所述子滤波器对应的卷积计算结果进行叠加，并将叠加的结果作为所述卷积滤波器对所述待卷积特征图卷积计算的结果进行输出。
  10. 如权利要求9所述的计算机设备，其中，所述处理器在实现所述获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图时，用于实现如下步骤：
    获取待卷积特征图；
    若获取的待卷积特征图的长或宽不是所述卷积步长的整数倍，对所述待卷积特征图的预设位置进行补零以使补零后的待卷积特征图的长或宽是所述卷积步长的整数倍；
    根据预设的特征图分割规则从补零后的待卷积特征图分割出若干特征子图。
  11. 如权利要求10所述的计算机设备，其中，所述处理器在实现所述若所述卷积步长不为1且所述卷积滤波器的尺寸大于1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器时，用于实现如下步骤：
    若所述卷积步长为2且所述卷积滤波器的尺寸为2×2，从所述卷积滤波器分割出4个子滤波器，各所述子滤波器的尺寸为1×1；其中第一个子滤波器包括所述卷积滤波器奇数行奇数列的权值，第二个子滤波器包括所述卷积滤波器奇数行偶数列的权值，第三个子滤波器包括所述卷积滤波器偶数行奇数列的权值，第四个子滤波器包括所述卷积滤波器偶数行偶数列的权值；
    若所述卷积步长为2且所述卷积滤波器的尺寸为2×2，所述获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，具体包括：
    将所述待卷积特征图奇数行奇数列的数值分配至第一个特征子图的相应位置，将所述待卷积特征图奇数行偶数列的数值分配至第二个特征子图的相应位置，将所述待卷积特征图偶数行奇数列的数值分配至第三个特征子图的相应位置，将所述待卷积特征图偶数行偶数列的数值分配至第四个特征子图的相应位置。
  12. 如权利要求10所述的计算机设备，其中，所述处理器在实现所述若所述卷积步长不为1且所述卷积滤波器的尺寸大于1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器时，用于实现如下步骤：
    若所述卷积步长为2且所述卷积滤波器的尺寸为3×3，从所述卷积滤波器分割出4个子滤波器，各所述子滤波器的尺寸为2×2；其中第一个子滤波器包括所述卷积滤波器奇数行奇数列的权值，第二个子滤波器包括所述卷积滤波器奇数行偶数列的权值，第三个子滤波器包括所述卷积滤波器偶数行奇数列的权值，第四个子滤波器包括所述卷积滤波器偶数行偶数列的权值；
    若所述卷积步长为2且所述卷积滤波器的尺寸为3×3，所述获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，具体包括：
    将所述待卷积特征图奇数行奇数列的数值分配至第一个特征子图的相应位置，将所述待卷积特征图奇数行偶数列的数值分配至第二个特征子图的相应位置，将所述待卷积特征图偶数行奇数列的数值分配至第三个特征子图的相应位置，将所述待卷积特征图偶数行偶数列的数值分配至第四个特征子图的相应位置。
  13. 如权利要求10所述的计算机设备，其中，所述处理器在实现所述若所述卷积步长不 为1且所述卷积滤波器的尺寸大于1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器时，用于实现如下步骤：
    若所述卷积步长为3且所述卷积滤波器的尺寸为3×3，从所述卷积滤波器分割出9个子滤波器，各所述子滤波器的尺寸为1×1且分别包括所述卷积滤波器9个权值中的一个；
    若所述卷积步长为3且所述卷积滤波器的尺寸为3×3，所述获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，具体包括：
    将所述待卷积特征图第3n+1行第3n+1列的数值分配至第一个特征子图的相应位置，将所述待卷积特征图第3n+1行第3n+2列的数值分配至第二个特征子图的相应位置，将所述待卷积特征图第3n+1行第3n+3列的数值分配至第三个特征子图的相应位置，将所述待卷积特征图第3n+2行第3n+1列的数值分配至第四个特征子图的相应位置，将所述待卷积特征图第3n+2行第3n+2列的数值分配至第五个特征子图的相应位置，将所述待卷积特征图第3n+2行第3n+3列的数值分配至第六个特征子图的相应位置，将所述待卷积特征图第3n+3行第3n+1列的数值分配至第七个特征子图的相应位置，将所述待卷积特征图第3n+3行第3n+2列的数值分配至第八个特征子图的相应位置，将所述待卷积特征图第3n+3行第3n+3列的数值分配至第九个特征子图的相应位置，其中n为自然数。
  14. 如权利要求9-13中任一项所述的计算机设备，其中：所述处理器在实现所述基于脉动阵列，根据各所述子滤波器对各自对应的特征子图进行卷积计算时，用于实现如下步骤：
    将所述子滤波器的权值加载至连接于所述脉动阵列的权值寄存器中；
    将所述子滤波器对应的特征子图加载至连接于所述脉动阵列的输入寄存器中；
    获取所述脉动阵列卷积计算后的输出结果。
  15. 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，若所述计算机程序被处理器执行，实现如下步骤：
    获取卷积滤波器的卷积参数，所述卷积参数包括卷积步长和所述卷积滤波器的尺寸；
    若所述卷积步长不为1且所述卷积滤波器的尺寸大于1×1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器，各所述子滤波器的尺寸小于所述卷积滤波器的尺寸；
    获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，所述若干特征子图与所述若干子滤波器一一对应；
    基于脉动阵列，根据各所述子滤波器对各自对应的特征子图进行卷积计算，卷积计算的步长为1；
    将各所述子滤波器对应的卷积计算结果进行叠加，并将叠加的结果作为所述卷积滤波器对所述待卷积特征图卷积计算的结果进行输出。
  16. 如权利要求15所述的存储介质，其中，所述处理器在实现所述获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图时，用于实现如下步骤：
    获取待卷积特征图；
    若获取的待卷积特征图的长或宽不是所述卷积步长的整数倍，对所述待卷积特征图的预设位置进行补零以使补零后的待卷积特征图的长或宽是所述卷积步长的整数倍；
    根据预设的特征图分割规则从补零后的待卷积特征图分割出若干特征子图。
  17. 如权利要求16所述的存储介质，其中，所述处理器在实现所述若所述卷积步长不为1且所述卷积滤波器的尺寸大于1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器时，用于实现如下步骤：
    若所述卷积步长为2且所述卷积滤波器的尺寸为2×2，从所述卷积滤波器分割出4个子滤波器，各所述子滤波器的尺寸为1×1；其中第一个子滤波器包括所述卷积滤波器奇数行奇数列的权值，第二个子滤波器包括所述卷积滤波器奇数行偶数列的权值，第三个子滤波器包括所述卷积滤波器偶数行奇数列的权值，第四个子滤波器包括所述卷积滤波器偶数行偶数列的权值；
    若所述卷积步长为2且所述卷积滤波器的尺寸为2×2，所述获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，具体包括：
    将所述待卷积特征图奇数行奇数列的数值分配至第一个特征子图的相应位置，将所述待卷积特征图奇数行偶数列的数值分配至第二个特征子图的相应位置，将所述待卷积特征图偶数行奇数列的数值分配至第三个特征子图的相应位置，将所述待卷积特征图偶数行偶数列的数值分配至第四个特征子图的相应位置。
  18. 如权利要求16所述的存储介质，其中，所述处理器在实现所述若所述卷积步长不为1且所述卷积滤波器的尺寸大于1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器时，用于实现如下步骤：
    若所述卷积步长为2且所述卷积滤波器的尺寸为3×3，从所述卷积滤波器分割出4个子滤波器，各所述子滤波器的尺寸为2×2；其中第一个子滤波器包括所述卷积滤波器奇数行奇数列的权值，第二个子滤波器包括所述卷积滤波器奇数行偶数列的权值，第三个子滤波器包括所述卷积滤波器偶数行奇数列的权值，第四个子滤波器包括所述卷积滤波器偶数行偶数列的权值；
    若所述卷积步长为2且所述卷积滤波器的尺寸为3×3，所述获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，具体包括：
    将所述待卷积特征图奇数行奇数列的数值分配至第一个特征子图的相应位置，将所述待卷积特征图奇数行偶数列的数值分配至第二个特征子图的相应位置，将所述待卷积特征图偶数行奇数列的数值分配至第三个特征子图的相应位置，将所述待卷积特征图偶数行偶数列的数值分配至第四个特征子图的相应位置。
  19. 如权利要求16所述的存储介质，其中，所述处理器在实现所述若所述卷积步长不为1且所述卷积滤波器的尺寸大于1，根据预设的滤波器分割规则从所述卷积滤波器分割出若干子滤波器时，用于实现如下步骤：
    若所述卷积步长为3且所述卷积滤波器的尺寸为3×3，从所述卷积滤波器分割出9个子滤波器，各所述子滤波器的尺寸为1×1且分别包括所述卷积滤波器9个权值中的一个；
    若所述卷积步长为3且所述卷积滤波器的尺寸为3×3，所述获取待卷积特征图并根据预设的特征图分割规则从所述待卷积特征图分割出若干特征子图，具体包括：
    将所述待卷积特征图第3n+1行第3n+1列的数值分配至第一个特征子图的相应位置，将所述待卷积特征图第3n+1行第3n+2列的数值分配至第二个特征子图的相应位置，将所述待卷积特征图第3n+1行第3n+3列的数值分配至第三个特征子图的相应位置，将所述待卷积特征图第3n+2行第3n+1列的数值分配至第四个特征子图的相应位置，将所述待卷积特征图第3n+2行第3n+2列的数值分配至第五个特征子图的相应位置，将所述待卷积特征图第3n+2行第3n+3列的数值分配至第六个特征子图的相应位置，将所述待卷积特征图第3n+3行第3n+1列的数值分配至第七个特征子图的相应位置，将所述待卷积特征图第3n+3行第3n+2列的数值分配至第八个特征子图的相应位置，将所述待卷积特征图第3n+3行第3n+3列的数值分配至第九个特征子图的相应位置，其中n为自然数。
  20. 如权利要求15-19中任一项所述的存储介质，其中：所述处理器在实现所述基于脉动阵列，根据各所述子滤波器对各自对应的特征子图进行卷积计算时，用于实现如下步骤：
    将所述子滤波器的权值加载至连接于所述脉动阵列的权值寄存器中；
    将所述子滤波器对应的特征子图加载至连接于所述脉动阵列的输入寄存器中；
    获取所述脉动阵列卷积计算后的输出结果。
PCT/CN2019/103137 2019-04-04 2019-08-28 基于脉动阵列的神经网络加速方法、装置、计算机设备及存储介质 Ceased WO2020199476A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910268881.8A CN110135556B (zh) 2019-04-04 2019-04-04 基于脉动阵列的神经网络加速方法、装置、计算机设备及存储介质
CN201910268881.8 2019-04-04

Publications (1)

Publication Number Publication Date
WO2020199476A1 true WO2020199476A1 (zh) 2020-10-08

Family

ID=67569234

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/103137 Ceased WO2020199476A1 (zh) 2019-04-04 2019-08-28 基于脉动阵列的神经网络加速方法、装置、计算机设备及存储介质

Country Status (2)

Country Link
CN (1) CN110135556B (zh)
WO (1) WO2020199476A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114781619A (zh) * 2022-04-29 2022-07-22 吉林大学 软错误检测方法和装置
US20230009202A1 (en) * 2020-10-30 2023-01-12 Boe Technology Group Co., Ltd. Image processing method and device, electronic apparatus and readable storage medium
CN119558366A (zh) * 2024-11-28 2025-03-04 中国科学技术大学 可配置深度感知卷积算子硬件架构

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110135556B (zh) * 2019-04-04 2024-08-23 平安科技（深圳）有限公司 基于脉动阵列的神经网络加速方法、装置、计算机设备及存储介质
CN110826710B (zh) * 2019-10-18 2021-04-23 南京大学 基于横向脉动阵列的rnn前向传播模型的硬件加速实现方法
CN112766474B (zh) * 2019-11-04 2024-03-22 北京地平线机器人技术研发有限公司 用于实现卷积运算的方法、装置、介质以及电子设备
US11861485B2 (en) * 2019-11-22 2024-01-02 Baidu Usa Llc Data format transform method to improve AI engine MAC utilization
CN113033761B (zh) * 2019-12-09 2024-05-14 中科寒武纪科技股份有限公司 数据处理方法、装置、计算机设备和存储介质
CN112862667A (zh) * 2021-01-29 2021-05-28 成都商汤科技有限公司 一种池化方法、芯片、设备和存储介质
CN113870273B (zh) * 2021-12-02 2022-03-25 之江实验室 一种基于脉动阵列的神经网络加速器特征图分割方法
CN116167425B (zh) * 2023-04-26 2023-08-04 浪潮电子信息产业股份有限公司 一种神经网络加速方法、装置、设备及介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107885700A (zh) * 2017-12-29 2018-04-06 中国人民解放军国防科技大学 一种大规模矩阵卷积的多核实现方法
CN108268931A (zh) * 2016-12-30 2018-07-10 华为技术有限公司 数据处理的方法、装置和系统
CN108875908A (zh) * 2017-05-16 2018-11-23 三星电子株式会社 优化的神经网络输入步长方法及设备
KR20190035445A (ko) * 2017-09-26 2019-04-03 삼성전자주식회사 전자 장치 및 그 제어 방법
CN109934339A (zh) * 2019-03-06 2019-06-25 东南大学 一种基于一维脉动阵列的通用卷积神经网络加速器
CN110135556A (zh) * 2019-04-04 2019-08-16 平安科技（深圳）有限公司 基于脉动阵列的神经网络加速方法、装置、计算机设备及存储介质

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10990648B2 (en) * 2017-08-07 2021-04-27 Intel Corporation System and method for an optimized winograd convolution accelerator
CN107564025B (zh) * 2017-08-09 2020-05-29 浙江大学 一种基于深度神经网络的电力设备红外图像语义分割方法
CN108491926B (zh) * 2018-03-05 2022-04-12 东南大学 一种基于对数量化的低比特高效深度卷积神经网络硬件加速设计方法、模块及系统
CN108875904A (zh) * 2018-04-04 2018-11-23 北京迈格威科技有限公司 图像处理方法、图像处理装置和计算机可读存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108268931A (zh) * 2016-12-30 2018-07-10 华为技术有限公司 数据处理的方法、装置和系统
CN108875908A (zh) * 2017-05-16 2018-11-23 三星电子株式会社 优化的神经网络输入步长方法及设备
KR20190035445A (ko) * 2017-09-26 2019-04-03 삼성전자주식회사 전자 장치 및 그 제어 방법
CN107885700A (zh) * 2017-12-29 2018-04-06 中国人民解放军国防科技大学 一种大规模矩阵卷积的多核实现方法
CN109934339A (zh) * 2019-03-06 2019-06-25 东南大学 一种基于一维脉动阵列的通用卷积神经网络加速器
CN110135556A (zh) * 2019-04-04 2019-08-16 平安科技（深圳）有限公司 基于脉动阵列的神经网络加速方法、装置、计算机设备及存储介质

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230009202A1 (en) * 2020-10-30 2023-01-12 Boe Technology Group Co., Ltd. Image processing method and device, electronic apparatus and readable storage medium
US12277703B2 (en) * 2020-10-30 2025-04-15 Boe Technology Group Co., Ltd. Image processing method and device, electronic apparatus and readable storage medium
CN114781619A (zh) * 2022-04-29 2022-07-22 吉林大学 软错误检测方法和装置
CN119558366A (zh) * 2024-11-28 2025-03-04 中国科学技术大学 可配置深度感知卷积算子硬件架构

Also Published As

Publication number Publication date
CN110135556A (zh) 2019-08-16
CN110135556B (zh) 2024-08-23

Similar Documents

Publication Publication Date Title
WO2020199476A1 (zh) 基于脉动阵列的神经网络加速方法、装置、计算机设备及存储介质
TWI811291B (zh) 深度學習加速器及加快深度學習操作的方法
US11960566B1 (en) Reducing computations for data including padding
JP6857286B2 (ja) ニューラルネットワークアレイの性能の改善
US10445638B1 (en) Restructuring a multi-dimensional array
CN107862650B (zh) 加速计算二维图像cnn卷积的方法
CN114491399B (zh) 数据处理方法、装置、终端设备及计算机可读存储介质
US20190138567A1 (en) Hardware Implementation of Convolutional Layer of Deep Neural Network
JP7710507B2 (ja) 表畳み込みおよびアクセラレーション
US11315344B2 (en) Reconfigurable 3D convolution engine
WO2019136762A1 (zh) 人工智能处理器、及其所应用的处理方法
CN109635630B (zh) 手部关节点检测方法、装置及存储介质
CN112598673A (zh) 全景分割方法、装置、电子设备和计算机可读介质
WO2019085709A1 (zh) 一种应用于卷积神经网络的池化处理的方法及系统
WO2019215907A1 (ja) 演算処理装置
CN110147252A (zh) 一种卷积神经网络的并行计算方法及装置
CN113297973A (zh) 关键点检测方法、装置、设备和计算机可读介质
JP7108702B2 (ja) 複数の入力データセットのための処理
CN110738317A (zh) 基于fpga的可变形卷积网络运算方法、装置和系统
JP2020135871A (ja) 物体認識方法、装置及びシングルステップの物体認識ニューラルネットワーク
CN118262316A (zh) 一种夜间车道线检测方法、系统、设备及介质
CN109815789A (zh) 在cpu上实时多尺度人脸检测方法与系统及相关设备
CN113642510A (zh) 目标检测方法、装置、设备和计算机可读介质
CN111178513A (zh) 神经网络的卷积实现方法、卷积实现装置及终端设备
CN114419051B (zh) 一种适应含有像素级分割的多任务场景的方法及系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19923089

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19923089

Country of ref document: EP

Kind code of ref document: A1