TWI798591B - Convolutional neural network operation method and device - Google Patents

Abstract

A convolutional neural network device includes a scheduling mode unit, a first data processor unit, a second data processor unit, and a multiply-accumulation operation array. The scheduling mode unit determines a target scheduling mode according to a number of target convolutional kernel(s) and size information of the target convolutional kernel(s), in which the target scheduling mode corresponds to a size of a convolutional computing block. The first data processor unit recombines weighted data in the target convolutional kernel(s) according to the target scheduling mode. The second data processor unit recombines input data of a target convolutional kernel layer. The multiply-accumulation operation array includes multiply-accumulation cells which performs a multiply-accumulation operation based on the recombined weighted data and the recombined input data. A number of the multiply-accumulation cells being utilized by the multiply-accumulation array in each operation corresponds to a size of the convolutional computing block.

Description

Convolutional neural network computing method and device

The present application relates to the technical field of data processing, in particular to a convolutional neural network computing method and device.

Deep learning (Deep learning) is one of the important application technologies for the development of AI (Artificial intelligence), which is widely used in computer vision, speech recognition and other fields. Among them, CNN (Convolutional Neural Network, Convolutional Neural Network) is a deep learning and efficient recognition technology that has attracted attention in recent years. Layer convolution operations and vector operations to produce high-accuracy results in image and speech recognition.

However, with the development and wide application of convolutional neural network, it faces more and more challenges. For example, the parameter scale of CNN model is getting larger and larger, and the network structure is complex and changeable. A CNN model often Contains multiple convolutional layers, and the depth of each convolutional layer and the size of the convolution kernel are different. Among them, in the shallower layers of the CNN network, the plane size of the input data to be processed may be larger, and the size in the channel direction may be smaller, and as the number of network layers deepens, some convolution kernels in the channel The depth in the direction will be larger, or the number of convolution kernels in the convolutional layer will be larger. then, yes For a multiplication-accumulation operation array composed of multiple MACs (Multiply Accumulation Cells, multiplication-accumulation operation units) in an electronic device, it will face a huge amount of data to be calculated. However, the processing capability provided by electronic devices is often limited, that is to say, the maximum amount of data that can be input in one round of the multiply-accumulate array is fixed. For example, if the multiply-accumulate operation array of the electronic device includes a plurality of multiply-accumulate operation units with a processing capacity of 256, then the multiply-accumulate operation array has 256 multipliers, that is, at the same time, up to 256 weight values and corresponding 256 input Data values are multiplied. The general input data is much larger than 256. Therefore, it is necessary to split the convolution kernel and input data into multiple data blocks and perform calculations one by one. However, in the prior art, the convolution kernel and the input data are split in the same way for different convolution layers, which cannot effectively utilize the hardware resources in electronic devices. Based on this, how to improve the performance of hardware accelerators in the calculation process The utilization of resources in the system has become an urgent problem to be solved.

The present application provides a convolutional neural network computing method and device, aiming at improving resource utilization of hardware accelerators.

The present application provides a convolutional neural network computing method, which is applied to a convolutional neural network computing device including a multiply-accumulate array, and the multiply-accumulate array includes a plurality of multiply-accumulate computing units. The convolutional neural network operation method of the present application includes: determining the number of target convolution kernels in a target convolution layer and a first size information of the target convolution kernel; according to the number of the target convolution kernels and the set The first size information determines a target scheduling mode, wherein the target scheduling mode corresponds to the size of a convolution calculation block; according to the target scheduling mode, the weight data in the target convolution kernel is reorganized, and After recombination, the weight data is output to the multiply-accumulate operation array; according to the purpose A standard scheduling mode, reorganizing the input data in the target convolutional layer, and outputting the reorganized input data to the multiply-accumulate operation array; and using the multiply-accumulate operation array based on the reorganized weight data and After recombining, the input data is subjected to a multiply-accumulate operation, wherein the number of multiply-accumulate units used in each round of the multiply-accumulate array corresponds to the size of the convolution calculation block.

The present application also provides a convolutional neural network computing device, which includes a scheduling mode unit, a first data processing unit, a second data processing unit, and a multiply-accumulate array. The scheduling mode unit determines a target scheduling mode according to the number of target convolution kernels and a first size information, wherein the target scheduling mode corresponds to the size of a convolution calculation block. The first data processing unit reorganizes the weight data in the target convolution kernel according to the target scheduling mode. The second data processing unit reorganizes the input data in the target convolutional layer according to the target scheduling mode. The multiply-accumulate operation array includes a plurality of multiply-accumulate operation units, which perform multiply-accumulate operations based on the reorganized weight data and the reorganized input data, wherein the product used in each round of the multiply-accumulate operation array The number of accumulation operation units corresponds to the size of the convolution calculation block.

The convolutional neural network calculation scheme provided by the embodiment of the present application can dynamically adjust the target scheduling mode for each convolutional layer with different network structures in the convolutional neural network, so that each convolutional layer can adopt a different The scheduling mode of the multiplication and accumulation operation array structure matching is used to split the data block between the input data to be processed and the target convolution kernel, so that the weight value contained in the split weight data block and the input data contained in the input data block The number can realize the maximum utilization of the operation resources of the multiply-accumulate operation array, which improves the resource utilization rate of the hardware accelerator as a whole, and then improves the operation speed of the convolutional neural network.

101, 102, 103, 104, 105: steps

00~09,10~17: weight data block

0~41,47,53,59,65,71: input data block

40: Convolutional neural network computing device

401: Mode Scheduling Unit

402: Weight data processing unit

403: Feature data processing unit

404: multiply-accumulate operation array

405: scratchpad

406: Cache

407: memory

408: Direct Memory Access Controller

409: Cache

K ₁ , K ₂ , K ₃ , K _M : convolution kernel

R, W, w: width

C,D,d: Depth

S, H, h: Height

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings on the premise of not paying progressive efforts.

[Fig. 1] is a schematic flow diagram of the operation method of the convolutional neural network provided by the embodiment of the present application; [Fig. 2] is a schematic diagram of the input data of a convolution layer and a data structure of a convolution kernel; [Fig. 3] is an implementation In the example, a schematic diagram of the input data of the convolutional layer and data splitting of the convolution kernel; and [FIG. 4] is a schematic block diagram of the application of the convolutional neural network computing device provided by the embodiment of the present application to electronic equipment.

The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Where the same reference numerals represent the same components, the principle of the present invention is illustrated by being implemented in a suitable application environment. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making progressive efforts belong to the scope of protection of this application.

Reference herein to "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The appearances of this term in various places in the specification are not necessarily all referring to the same embodiment, nor are they mutually exclusive from other embodiments. or alternative embodiments. It may be apparent or implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

The embodiment of the present application provides a convolutional neural network computing method. The execution subject of the convolutional neural network computing method may be the convolutional neural network computing device provided in the embodiment of the present application, or integrate the convolutional neural network The electronic equipment of the computing device of the neural network, in terms of implementation, the computing device of the convolutional neural network can be realized by hardware, software, or a combination of hardware and software.

The convolutional neural network operation scheme provided by the embodiment of the present application can be applied to any structured convolutional neural network (hereinafter referred to as CNN), for example, it can be applied to a CNN with only one convolutional layer, and it can also be applied to some complex CNNs, such as CNNs that include up to hundreds or more convolutional layers. In addition, the CNN in the embodiment of the present application may also have a pooling layer, a fully connected layer, and the like. That is to say, the scheme of the embodiment of this application is not limited to a specific convolutional neural network, as long as it is a neural network that includes a convolutional layer, it can be considered as a "convolutional neural network" in this application , the convolutional layer part can be operated according to the embodiment of the present application.

It should be noted that the convolutional neural network of the embodiment of the present application can be applied to various scenarios, for example, image recognition fields such as face recognition and license plate recognition, feature fields such as image feature extraction and speech feature extraction, In the field of speech recognition, natural language processing, etc., the feature data converted from images or other forms of data can be registered into the pre-trained convolutional neural network, and the convolutional neural network can be used to perform calculations to achieve Or classification or recognition or feature extraction purposes.

Please refer to FIG. 1 . FIG. 1 is a schematic flowchart of a convolutional neural network operation method provided by an embodiment of the present application. FIG. 4 is a schematic block diagram of a convolutional neural network computing device provided by an embodiment of the present application applied to an electronic device. The convolutional neural network operation device 40 can be used to realize the convolutional neural network in Fig. 1 Road operation method. The specific steps of the convolutional neural network computing method and the operation of the convolutional neural network computing device 40 are described as follows.

In step 101, the number of target convolution kernels in a target convolution layer and a first size information of the target convolution kernels are determined.

For an electronic device integrated with a convolutional neural network computing device, the convolution layer performs convolution operations on input data and convolution kernel data to obtain output data. The input data can be the original image, voice data, or the output data of the previous convolutional layer or pooling layer, and the input data in the convolutional neural network computing device is generally characteristic data. Therefore, the convolutional neural network computing device The input data of 40 may be feature data of the target convolutional layer.

The input data can have multiple channels. The input data on each channel can be understood as a two-dimensional data. When the number of channels of the input data is greater than 1, the input data can be understood as a two-dimensional data stack of multiple channels. Together the volume data has a depth equal to the number of channels. The target convolutional layer (that is, the convolutional layer to be convolutional at present) can contain one or more convolution kernels, which are also called filters, and the number of channels of each convolution kernel is equal to the input of the layer The number of channels of the data, the number of convolution kernel data is equal to the number of channels of the output data of the target convolution layer. That is to say, after the input data is convoluted with a convolution kernel data, a two-dimensional data is obtained. When the target convolution layer has multiple convolution kernels, the two-bit data output by each convolution kernel is superimposed to obtain A 3D output data.

When performing calculations based on convolutional neural networks, the mode scheduling unit 401 determines the target convolutional layer from the convolutional neural network currently used for calculations. In practice, the mode scheduling unit 401 can obtain the target volume from the configuration register 405 Layer-related information, such as learning from the configuration register 405 which is the target The convolutional layer and its number of convolution kernels, the plane size of the convolution kernel, and the depth in the direction of the channel.

Please refer to FIG. 2 . FIG. 2 is a schematic diagram of the input data of a convolution layer and the data structure of the convolution kernel. The convolution layer in Fig. 2 includes M convolution kernels, namely K ₁ , K ₂ , K ₃ . . . K _M . The M convolution kernels have the same size, all of which are D×R×S. As shown in the figure, D represents the depth of the convolution kernel in the channel direction, and R×S represents the size of the convolution kernel in the plane direction. . The size of the input data is C×W×H, where C represents the depth of the input data in the channel direction. In practice, C=D. And W×H represents the size of the input data in the plane direction.

Since the size and number of convolution kernels in each convolution layer may be different, in the embodiment of the present application, when starting to operate on the target convolution layer, first determine the number of target convolution kernels in the target convolution layer, the target Size and/or depth information of the convolution kernel.

In step 102, a target scheduling mode is determined according to the number of target convolution kernels and the first size information, wherein the target scheduling mode corresponds to the size of a convolution calculation block.

In the case where the number of multiply-accumulate operation units in the multiply-accumulate operation array is limited, and the amount of parameters in the convolution layer (such as the amount of data of the convolution kernel) is huge, for a convolution layer, it may be necessary to multiply-accumulate. Multiple rounds of calculations can complete all calculations. Therefore, it is necessary to split the convolution kernel and input data into multiple data blocks, and input a certain number of weight data blocks and corresponding number of input data blocks into the multiplication-accumulation operation array each time. , and then perform a multiply-accumulate operation. In order to effectively utilize the computing resources of the multiply-accumulate array 404, the mode scheduling unit 401 can determine a target scheduling mode according to the number of target convolution kernels and related size information. In practice, the mode scheduling unit 401 can determine a target scheduling mode according to the target convolution kernel The number and its related size information select the target scheduling mode from multiple preset scheduling modes, and each scheduling mode corresponds to a specific convolution calculation block size, convolution calculation A block is the smallest unit for performing convolution operations. The target scheduling mode selected by the mode scheduling unit 401 can make the most effective use of the multiply-accumulate array 404. For example, when the convolutional neural network computing device 40 operates in the target scheduling mode, the product The accumulation operation array can complete the multiplication and accumulation operation of the input data and the target convolution kernel with the minimum number of operation rounds.

In an embodiment, the size of the convolution calculation block corresponding to the scheduling mode can be obtained from m target convolution kernels in one round of operation of the multiply-accumulate operation array. m weight data blocks with a size of d×w×h The size of the data block, where d represents the depth of the weight data block in the channel direction, and w×h represents the size of the weight data block in the plane direction, where m, d, w, and h are all positive integers. When setting the preset scheduling mode, it can be regarded as setting the specific values of m, d, w, and h, and various factors need to be considered comprehensively.

256。 First, the processing capability of the multiply-accumulate array 404 of the electronic device needs to be considered, that is, the number of multiply-accumulate units in the multiply-accumulate array 404 needs to be considered. For example, there are 256 multiply-accumulate units in the multiply-accumulate array 404, and one round of operation A maximum of 256 multiply-accumulate unit operations can be performed simultaneously in the system. Therefore, when m weight data blocks of size d×w×h are respectively obtained from m target convolution kernels in a round of operation of the multiply-accumulate array, the sizes of m, d, w, and h need to satisfy: m× d×w×h

256.

Secondly, the needs of the actual network must also be considered, for example, the size and number of convolution kernels in the convolution layer, for example, the size of some convolution kernels is 1×1×64, and the size of some convolution kernels is 11×11 ×3, some convolutional layers may have 8 convolutional kernels, and some convolutional layers may have 2048 convolutional kernels. After comprehensive consideration of the above parameters, different sizes of convolution calculation blocks are set to adapt to different network layers.

256的條件下，當卷積層的卷積核數量較多、深度較小時，可以將m 的值設置的大一些，d的值設置的小一些，比如，m=64，d=4，w=1，h=1，或者m=16，d=16，w=1，h=1。反之，當卷積層的卷積核數量較少、深度較大時，可以將m的值設置的較小一些，d的值設置的較大一些，比如，m=4，d=64，w=1，h=1。或者，對於一些特別尺寸的卷積核，還可以做一些特別的設置，比如，對於3×3的卷積核，可以設置m=1，d=32，w=3，h=3，這種情況下，雖然不能百分之百的利用乘積累加運算單元的計算資源，但是也達到了較高的利用率。請再參閱圖2，圖2的卷積層中包含M個卷積核，此例中m較佳為的M的正因數，也就是說，M為m的整數倍。 For example, under the premise that the processing capability of the multiply-accumulate operation unit is fixed, that is, in m×d×w×h

Under the condition of 256, when the number of convolution kernels of the convolutional layer is large and the depth is small, the value of m can be set larger, and the value of d can be set smaller, for example, m=64, d=4, w =1, h=1, or m=16, d=16, w=1, h=1. Conversely, when the number of convolution kernels of the convolutional layer is small and the depth is large, the value of m can be set smaller, and the value of d can be set larger, for example, m=4, d=64, w= 1, h=1. Or, for some special-sized convolution kernels, you can also make some special settings. For example, for a 3×3 convolution kernel, you can set m=1, d=32, w=3, h=3, this kind of In this case, although the computing resources of the multiply-accumulate operation unit cannot be utilized 100%, a relatively high utilization rate is achieved. Please refer to FIG. 2 again. The convolution layer in FIG. 2 includes M convolution kernels. In this example, m is preferably a positive factor of M, that is, M is an integer multiple of m.

In practice, it is possible to pre-evaluate the size of the most suitable convolution calculation block for the number and size of convolution kernels of various convolution layers, and then pre-set a variety of preset scheduling modes based on this, and create a look-up data table in a memory, This lookup data table contains the mapping relationship between the parameters of the convolution kernel and the default scheduling mode. The mode scheduling unit 401 can look up the target scheduling mode from the lookup data table according to the number of target convolution kernels and their related size information. In one embodiment, the mode scheduling unit 401 can be implemented by a processor executing the program code, and the data volume information of the input data of the target convolution layer and the number and size of convolution kernels are stored in the configuration register 405. , the mode scheduling unit 401 obtains the number of target convolution kernels and related size information from the configuration register 405 .

As shown in FIG. 4 , during the convolution operation, the convolutional neural network operation device 40 obtains the weight value and input data required for each round of operation from the memory 407 via the cache 409, and the multiply-accumulate operation The intermediate results generated during the operation of the array 404 are temporarily stored in the cache 406 . For electronic devices, the storage space allocated for convolution operations is limited. For the above reasons, when pre-setting the scheduling mode, in addition to considering the network structure, it is also necessary to consider when using this scheduling mode operation. Occupancy of storage space to set an appropriate scheduling mode. Therefore, in a In an embodiment, the mode scheduling unit 401 also determines the target scheduling mode according to the size of the cache 409 and/or the cache 406 .

In step 103, according to the target scheduling mode, reorganize the weight data in the target convolution kernel, and output the reorganized weight data to the multiply-accumulate array.

After the target scheduling mode is determined, the weight data processing unit 402 splits and properly reorganizes the weight data in the target convolution kernel according to the target scheduling mode, so that the reorganized weight data can be input into the multiply-accumulate operation array 404 in an appropriate order , and the multiply-accumulate array 404 can complete the required convolution operation.

In practice, the weight data in the target convolution kernel can be stored in the memory 407, and the weight data processing unit 402 can automatically store the weight data under the control of the direct memory access (Direct Memory Access, DMA) controller 408 via the cache 409. The memory 407 reads weight data.

In one embodiment, for each scheduling mode, the weight data processing unit 402 is configured with a corresponding operation setting for reading and reorganizing the weight data. When the target scheduling mode is determined, the weight data processing unit 402 operates according to the target scheduling mode. Set to read and reorganize the weight data in the target convolution kernel. In practice, the weight data processing unit 402 can write the weight data in the target convolution kernel into a cache in the original order, and then read the weight data from the cache in the required order to realize the weight data Purposes of reorganization and reordering.

Please refer to FIG. 3 . FIG. 3 is a schematic diagram of splitting the input data of the convolution layer and the data of the convolution kernel in an embodiment. After the target scheduling mode is determined, it is assumed that the size of the convolution calculation block corresponding to the target scheduling mode is m×d×w×h. The target convolution kernel reads m weight data blocks with a size of d×w×h, and inputs the reorganized weight data to the multiply-accumulate array 404 . In detail, the weight data processing unit 402 obtains m weight data blocks from the m convolution kernels K ₁ , K ₂ , ..., K _m respectively, wherein the depth of each weight data block in the channel direction is d, the dimension on the plane is w×h.

Step 104, according to the target scheduling mode, reorganize the input data in the target convolutional layer, and output the reorganized input data to the multiply-accumulate array.

After the target scheduling mode is determined, the characteristic data processing unit 403 splits the input data in the target convolutional layer according to the target scheduling mode and appropriately reorganizes the data, so that the reorganized input data can match the order of the corresponding weight data blocks and input them to the product Accumulate the operation array 404 to complete the required convolution operation.

In practice, the input data of the target convolutional layer can be stored in the memory 407 , and the feature data processing unit 403 can read the input data from the memory 407 through the cache 409 under the control of the direct memory access controller.

Similarly, for each scheduling mode, the feature data processing unit 403 is configured with a corresponding operation setting for reading and reorganizing input data. When the target scheduling mode is determined, the feature data processing unit 403 reads the corresponding operation settings according to the target scheduling mode. Fetch and reorganize the input data in the target convolutional layer. In practice, the characteristic data processing unit 403 can also write the input data in the target convolutional layer into a cache in the original order, and then read the input data from the cache in the required order, for example, to match the corresponding The data sequence of the weighted data blocks is read from the cache to achieve the purpose of reorganizing and reordering the input data.

Please refer to FIG. 3 again. In the embodiment of FIG. 3, the feature data processing unit 403 splits the input data in the target convolutional layer into a plurality of input data blocks whose size is d×w×h, and for each input data block Data reorganization is performed so that the data order of the input data blocks after reorganization can match the corresponding weight data blocks, so that the multiply-accumulate operation array 404 can complete the correct multiply-accumulate operation accordingly.

Step 105, perform multiply-accumulate calculation based on the reorganized weight data and the reorganized input data.

The multiply-accumulate operation array 404 performs multiply-accumulate operations based on the reorganized weight data and the reorganized input data, wherein the multiply-accumulate operation array 404 uses the number of multiply-accumulate operation units used in each round of operation corresponding to the size of the convolution calculation block.

After the multiply-accumulate operation array 404 performs one round of operation, the calculated result is stored in the cache 406 as intermediate data. During the multiply-accumulate operation, the multiply-accumulate operation array 404 will add the products in the channel direction of the same convolution kernel and store them as intermediate data. Then, the weight data processing unit 402 continues to read weight data blocks from the cache 409 according to the order in which the convolution kernel performs convolution operations on the input data, while the feature data processing unit 403 reads and reorganizes the input data from the cache 409. data to output an input data block that matches the weight data block, and the multiplication and accumulation operation array 404 performs another round of calculations, and so on, until the calculation of each data block in the input data and the weight data block is completed.

Next, take a specific application scenario as an example to illustrate the present invention, please continue to refer to Figure 3, assuming C=32, W=6, H=6; D=32, R=3, S=3, M=16; d =16, m=16, w=1, h=1. As shown in the figure, the input data can be split into 72 input data blocks, and one convolution kernel can be split into 18 weight data blocks. Assuming that the step size corresponding to the convolutional layer is 1, and zero padding with a length of 2 is performed in both the length and width directions, the 36 input data blocks of 0~35 on the input data need to be combined with each convolution kernel. Inner product operation is performed on the weight data block corresponding to 00.

For example, the feature data block 0 with a size of 16×1×1 (the gray square of the input data in Figure 3) on the input data needs to be matched with the size of 16×1×1 in each convolution kernel The weight data block 00 (the gray square of the convolution kernel in Fig. 3 ) performs the inner product operation. Therefore, the weight data According to the size of the convolution calculation block corresponding to the target scheduling mode (d=16, m=16, w=1, h=1), the processing unit 402 reads the first weight data blocks to obtain 16 weight data blocks 00 with a size of 16×1×1. The feature data processing unit 403 reads an input data block 0 with a size of 16×1×1 from the input data, and matches the input data block 0 to 16 weight data blocks 00 respectively (that is to say, the input data block 0 is in In one round of operation of the multiply-accumulate array 404, 16 times are repeatedly used, which is equivalent to 256 data). The input data block 0 and 16 weight data blocks 00 are input to the multiply-accumulate array 404 for calculation, and 16 values are obtained (the products in the channel direction are added), which are stored as intermediate results. This process is the multiply-accumulate array 404 round of operations. Then, the data is read for the second time, so as to perform the second round of operation of the multiply-accumulate operation array 404 . As mentioned above, the 36 input data blocks of 0-35 on the input data need to be inner producted with the weight data block 00 in each convolution kernel, so there is no need to read the weight data repeatedly when reading the data for the second time Block 00, just read the input data block 1 with a size of 16×1×1 from the input data, match the input data block 1 to the 16 weight data 00, and use the multiply-accumulate operation array 404 to perform calculations to obtain 16 values are also stored as intermediate results. Then read the data block 2 that is 16 * 1 * 1 according to the same data reading mode as the second round operation, read a size and carry out the third round operation of the multiply-multiply-accumulate operation array 404, ..., then according to the second round operation The same way of reading data as the round operation, reads an input data block 35 with a size of 16×1×1, and performs the 36th round operation of the multiply-accumulate array 404 . So far, the convolution operation of the input data and the weight data block 00 is completed, and at the same time, 36 sets of intermediate results are stored, and each set of intermediate results has 16 values.

Next, in the 37th round of calculation, read 16 weight data blocks 01 with a size of 16×1×1 from the cache, and read an input data block 0 with a size of 16×1×1 from the input data , match the input data block 0 to the 16 weight data blocks 01 respectively, and use the multiply-accumulate operation array 404 to perform calculations to obtain 16 values, because these 16 values and the 16 values obtained in the first round of calculation are all corresponding to the input The input data block 0 on the output data, so these 16 values need to be added to the 16 values obtained in the first round of calculations to obtain new 16 values, and stored as new intermediate results, overwriting the first 16 intermediate results stored in a round of calculation. According to the same data fetching method as the previous 36 times, the multiply-accumulate operation array 404 performs the 37th to 72nd rounds of operations to complete the convolution operation of the input data and the weight data block 01 .

Repeat the above operation process until all the convolution operations of the target convolution kernel on the input data are completed, and 16 two-dimensional output data are obtained. These 16 two-dimensional output data are superimposed to obtain the three-dimensional output data of the target convolution layer. Wherein, if the next layer is also a convolutional layer, the output data can be read into the cache and used as the input data for the next layer operation to continue the convolution operation.

It should be noted that what is mentioned above is a specific embodiment for the convenience of readers to understand the solution of this application. In this embodiment, since the number of weight values read at one time is more than the number of input data, when the weight values used in two adjacent rounds of calculations are repeated, in order to improve data processing efficiency, the weight data blocks are not repeatedly read. However, this is not a limitation to the solution of the present application. In other embodiments, data may also be read in other orders. When reading data in other orders, data blocks may be read repeatedly or not.

During specific implementation, the present application is not limited by the execution order of the described steps, and some steps may be performed in other orders or simultaneously in the case of no conflict.

In summary, the convolutional neural network computing method proposed in the embodiment of this application can dynamically adjust the target scheduling mode for each convolutional layer with different network structures in the convolutional neural network, so that each The convolutional layer can use a scheduling mode that matches its structure to split and reorganize the input data and the target convolution kernel data blocks, so that the weight values contained in the split weight data blocks and the features contained in the feature data blocks The number of values enables the operation of an array of multiply-accumulate operations The maximum utilization of computing resources improves the resource utilization of hardware accelerators as a whole, thereby improving the computing speed of convolutional neural networks.

The above is a detailed introduction to a convolutional neural network computing method and device provided by the embodiment of the present application. In this paper, a specific example is used to illustrate the principle and implementation of the present application. The description of the above embodiment is only used to help Understand the method of this application and its core idea; at the same time, for technical personnel in this field, according to the idea of this application, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be understood For the limitation of this application.

101, 102, 103, 104, 105: steps

Claims (15)

A convolutional neural network computing method, applied to a convolutional neural network computing device, the convolutional neural network computing device includes a multiply-accumulate array, the multiply-accumulate array includes a plurality of multiply-accumulate computing units , and the operation method of the convolutional neural network includes: determining the number of target convolution kernels in a target convolution layer and a first size information of the target convolution kernels; determining a target scheduling mode according to the number of target convolution kernels and the first size information, wherein the target scheduling mode corresponds to a size of a convolution calculation block; According to the target scheduling mode, reorganize the weight data in the target convolution kernel, and output the reorganized weight data to the multiply-accumulate array; According to the target scheduling mode, reorganize the input data in the target convolutional layer, and output the reorganized input data to the multiply-accumulate operation array; and Using the multiply-accumulate array to perform multiply-accumulate operations based on the reorganized weight data and the reorganized input data, wherein the number of multiply-accumulate units used in each round of the multiply-accumulate array corresponds to the The size of the convolution computation block described above. The computing method of the convolutional neural network according to claim 1, wherein the target scheduling mode corresponds to the minimum number of rounds of computing for the input data and the target convolution kernel to be completed by the multiply-accumulate computing array. The convolutional neural network computing method according to claim 1, wherein the first size information includes depth information of the target convolution kernel in a channel direction. The operation method of the convolutional neural network according to claim 1, wherein the target scheduling mode is selected from a plurality of preset scheduling modes. The operation method of the convolutional neural network according to claim 1, wherein the multiply-accumulate operation array stores intermediate data in a cache, and the step of determining the target scheduling mode is further based on the size of the cache to determine the target scheduling mode. The operation method of the convolutional neural network as claimed in item 1, wherein the number of the target convolution kernel is M, the size of the convolution calculation block is an integer multiple of m, and M is an integer multiple of m, M and m are all positive integers. The convolutional neural network computing method according to claim 1, wherein in the step of reorganizing the input data in the target convolution layer, the reorganized input data matches the reorganized weight data. A convolutional neural network computing device is used to perform convolution operations on a target convolution kernel and input data in a target convolutional layer, and the convolutional neural network computing device includes: A scheduling mode unit, which determines a target scheduling mode according to the number of target convolution kernels and a first size information, wherein the target scheduling mode corresponds to the size of a convolution calculation block; A first data processing unit, reorganizing the weight data in the target convolution kernel according to the target scheduling mode; a second data processing unit, reorganizing the input data in the target convolutional layer according to the target scheduling mode; and A multiply-accumulate operation array, including a plurality of multiply-accumulate operation units, which perform multiply-accumulate operations based on the reorganized weight data and the reorganized input data, wherein, the multiply-accumulate operation array used in each round of calculation The number of the multiply-accumulate operation units corresponds to the size of the convolution calculation block. The convolutional neural network computing device according to claim 8, wherein the target scheduling mode corresponds to the minimum number of computing rounds for the multiply-accumulate computing array to complete the input data and the target convolution kernel. The convolutional neural network computing device according to claim 8, wherein the first size information includes depth information of the target convolution kernel in a channel direction. The convolutional neural network computing device according to claim 8, wherein the target scheduling mode is selected from a plurality of preset scheduling modes. The convolutional neural network computing device according to claim 11, wherein the plurality of preset scheduling modes are stored in a memory. The convolutional neural network computing device according to claim 8, wherein the multiply-accumulate array stores intermediate data in a cache, and the scheduling mode unit determines the target scheduling mode according to the size of the cache . The convolutional neural network computing device according to claim 8, wherein the number of the target convolution kernels is M, the size of the convolution calculation block is an integer multiple of m, and M is an integer multiple of m, M and m All are positive integers. The convolutional neural network computing device according to claim 8, wherein the first data processing unit performs data reorganization by writing and reading weight data in the target convolution kernel from a cache.

Images