TW202230229A - Computing circuit and data processing method based on convolution neural network and computer readable storage medium - Google Patents

Abstract

A computing circuit and a data processing method based on convolution neural network and a computer readable storage medium are provided. An input data is obtained from the memory. The first computation is performed on the first part data of the input data, to obtain a first output data. The first output data is buffered in a first buffer area. When the buffered first output feature map is larger than a first predetermined data amount, the second computation is performed on the first output data. The second output data is buffered in a second buffer area. The third output data obtained from the third computation on the second output data is outputted to the memory. When the second computation is performed on the first output data, the first computation is performed on the input data continuously. Accordingly, the accessing time on the main memory could be reduced.

Description

Operation circuit, data processing method and computer readable storage medium based on convolutional neural network

The present invention relates to a machine learning (ML) technology, and more particularly, to a computing circuit, a data processing method and a computer-readable storage medium based on a convolutional neural network (Conventional Neural Network, CNN).

Machine learning is an important topic in artificial intelligence (AI), and can analyze training samples to obtain patterns from them, so as to predict unknown data through patterns. The machine learning model constructed after learning is used to infer the evaluation data.

There are many algorithms for machine learning. For example, neural networks can make decisions by simulating the workings of human brain cells. Among them, the convolutional neural network provides better results in image and speech recognition, and has gradually become one of the widely used and main research and development machine learning architectures.

Notably, in the convolutional layer of a convolutional neural network architecture, the processing element slides a kernel or filter on the input data and performs specific operations. The processing element needs to repeatedly read the input data and the weight value from the memory and output the operation result to the memory. Even, if different convolution layers use different convolution kernels or different convolution operations, the number of memory accesses will be greatly increased. For example, the MobileNet model combines convolution operations with Depthwise Separable Convolution operations. Therefore, these operations require separate memory accesses.

In view of this, embodiments of the present invention provide a convolutional neural network-based computing circuit, a data processing method, and a computer-readable storage medium, which integrates multiple convolutional layers and reduces the number of memory accesses accordingly.

The data processing method based on the convolutional neural network according to the embodiment of the present invention includes (but is not limited to) the following steps: reading input data from a memory. A first operation is performed on the first part of the input data to obtain the first output data. The first operation is provided with a first filter, and the size of the first output data is related to the size of the first filter of the first operation and the size of the first part of the data. The first output data is temporarily stored in the first temporary storage area. When the first output data temporarily stored in the first temporary storage area is larger than the first predetermined amount of data, a second operation is performed on the first output data to obtain the second output data. The second operation is provided with a second filter, and the size of the second output data is related to the second filter size of the second operation. The second output data is temporarily stored in the second register. The third output data obtained by performing the third operation on the second output data is output to the memory. When the second operation is performed on the first output data, the first operation is continuously performed on the input data.

The computing circuit based on the convolutional neural network according to the embodiment of the present invention includes (but is not limited to) a memory and a processing element. Memory is used to store input data. The processing element is coupled to the memory and includes first, second and third arithmetic units, a first temporary memory and a second temporary memory. The first operator is used for performing a first operation on the first part of the input data to obtain the first output data, and temporarily storing the first output data in the first temporary storage memory of the processing element. The size of the first output data is related to the size of the first filter of the first operation and the size of the first part of the data. The second arithmetic unit is used for performing a second operation on the second input data when the first output data temporarily stored in the first temporary storage memory meets the size required by the second operation, so as to obtain the second output data and temporarily store it The second output data is to the third memory of the processing element. The second operation is provided with a second filter, and the size of the second output data is related to the size of the second filter of the second operation. The third operator is used for outputting the third output data obtained by performing the third operation on the second output data to the memory. When the second arithmetic unit performs the second arithmetic operation, the first arithmetic unit continues to perform the first arithmetic operation.

The computer-readable storage medium of the embodiment of the present invention is used for storing the code, and the processor loads the code to execute the aforementioned data processing method based on the convolutional neural network.

Based on the above, the computing circuit, data processing method and computer-readable storage medium based on convolutional neural network according to the embodiments of the present invention temporarily store the output data in the memory of the processing element, and according to the next computing unit (ie , the start condition of the next operation layer) triggers its operation. In this way, the next operation layer can trigger the operation in advance without waiting for the previous operation layer to operate on all the input data. In addition, the embodiments of the present invention can reduce the number of times of accessing input data from the memory.

In order to make the above-mentioned features and advantages of the present invention more obvious and easy to understand, the following embodiments are given and described in detail with the accompanying drawings as follows.

FIG. 1 is a block diagram of components of an operation circuit 100 based on a convolutional neural network according to an embodiment of the present invention. Referring to FIG. 1 , the arithmetic circuit 100 includes (but is not limited to) a memory 110 and one or more processing elements (PE) 120 .

The memory 110 may be a dynamic random access memory (DRAM), a flash memory (Flash Memory), a register (Register), a combinational logic circuit (Combinational Circuit), or a combination of the above elements.

The processing element 120 is coupled to the memory 110 . The processing element 120 includes (but is not limited to) a feature temporary memory 131, a first-in, first-out (First Input First Output, FIFO) unit 132, a first weight temporary memory 133, a first operator 135, and a first temporary memory The bank 151 , the second weight temporary storage memory 153 , the second arithmetic unit 155 , the second temporary storage memory 171 , the second FIFO unit 172 , the third weight temporary storage memory 173 , and the third arithmetic unit 175 .

In one embodiment, the feature temporary storage 131 , the FIFO unit 132 , the first weight temporary storage 133 , and the first operator 135 correspond to one convolution layer/operation layer. In addition, the first operator 135 is provided with a first filter for the first operation.

In one embodiment, the feature temporary storage memory 131 is used for storing part or all of the input data from the memory 110, and the first FIFO unit 132 is used for inputting and/or outputting the feature temporary storage according to the FIFO rule For the data in the memory 131, the first weight temporary storage memory 133 is used to store the weights from the memory 110 (forming the first convolution kernel/filter), and the first operator 135 is used to perform the first operation. In one embodiment, the first operation is a convolution operation, which will be described in detail in subsequent embodiments. In another embodiment, the first operation may also be a depthwise separable convolution operation or other types of convolution operations.

In one embodiment, the first temporary memory 151 , the second weight temporary memory 153 , and the second operator 155 correspond to one convolution layer/operation layer. In addition, the second operator 155 is provided with a second filter for the second operation.

In one embodiment, the first temporary memory 151 is used to store part or all of the input data output from the first arithmetic unit 135 , and the second weight temporary memory 153 is used to store the weights from the memory 110 (forming the second convolution kernel/filter), and the second operator 155 is used to perform the second operation. In one embodiment, the second operation is a depthwise convolution operation, which will be described in detail in subsequent embodiments. In another embodiment, the second operation may also be a convolution operation or other types of convolution operations.

In one embodiment, the second temporary memory 171 , the second FIFO unit 172 , the third weighted temporary memory 173 and the third operator 175 correspond to one convolution layer/operation layer. In addition, the third operator 175 is provided with a third filter for the third operation.

In one embodiment, the second temporary memory 171 is used for storing part or all of the input data output from the second arithmetic unit 155, and the second FIFO unit 172 is used for inputting and/ Or output the data in the second temporary memory 171, the third weight temporary memory 173 is used to store the weights from the memory 110 (forming the third convolution kernel/filter), and the third operator 175 is used for Perform the third operation. In one embodiment, the third operation is a pointwise convolution operation, which will be described in detail in subsequent embodiments. In another embodiment, the third operation may also be a convolution operation or other types of convolution operations.

In one embodiment, the aforementioned feature temporary storage 131, first temporary storage, second temporary storage, first weight temporary storage 133, second weight temporary storage 153, and third weight temporary storage The memory 173 may be a static random access memory (SRAM), a flash memory, a register, various types of buffers, or a combination of the above elements.

In one embodiment, some or all of the elements in the arithmetic circuit 100 may form a neural network processing unit (NPU), a system on chip (SoC), or an integrated circuit (IC).

In one embodiment, the first arithmetic unit 135 has a first maximum arithmetic amount within a unit time, the second arithmetic unit 155 has a second maximum arithmetic amount within the same unit time, and the third arithmetic unit 175 has the same unit time Has the third largest amount of operations. The first maximum computation amount is greater than the second maximum computation amount, and the first maximum computation amount is greater than the third maximum computation amount.

Hereinafter, the method according to the embodiment of the present invention will be described in conjunction with various devices, components and modules in the computing circuit 100 . Each process of the method can be adjusted according to the implementation situation, and is not limited to this.

FIG. 2 is a flowchart of a data processing method based on a convolutional neural network according to an embodiment of the present invention. Referring to FIG. 2, the processing element 120 reads the input data from the memory 110 (step S210). Specifically, the input data may be data of some or all of the pixels in the image (eg, level, brightness, or gray level). Alternatively, the input data can also be a set of data related to speech, text, patterns or other aspects.

There are many ways to read input data. In one embodiment, the processing element 120 reads all the input data as the first part of the data. In another embodiment, the processing element 120 reads a part of the input data each time according to the amount of data required for the first operation or the capacity of the feature temporary storage memory 131 and uses it as the first part of the data.

FIG. 3 is a schematic diagram of input data F according to an embodiment of the present invention. Referring to FIG. 3 , it is assumed that the size of the input data F is (height H, width W, the number of channels C), and the size of the first part of the data F _fi1 read by the processing element 120 is (height H _fi1 , width W _fi1 , channel number C). The height H _fi1 may be less than or equal to the height H, and the width W _fi1 may be less than or equal to the width W.

It should be noted that the input data may be stored in a specific block or location in the memory 110 , but the embodiment of the present invention does not limit the storage location of each element in the input data in the memory 110 .

The feature temporary memory 131 stores some or all of the input data from the memory 110 . That is, the feature temporary storage memory 131 stores the first part of the data. The first operation unit 135 performs a first operation on the first part of the input data to obtain the first output data (step S230). Specifically, the first operation is to perform a first operation (eg, a convolution operation) on the first part of the data and the corresponding weight. The size of the first output data is related to the size of the first filter of the first operation and the size of the first part of the data.

For example, FIG. 4 is a schematic diagram of a first operation according to an embodiment of the present invention. Referring to FIG. 4 , the first operation is a convolution operation as an example. The first part of data F _fi1 (size is height H _fi1 , width W _fi1 , number of channels C _fi1 ) and the first filter K _n (size is height H _kd , width W _kd ). If the height H _fi1 is greater than or equal to the height H _kd and the width W _fi1 is greater than or equal to the width W _kd , the first operator 135 may trigger the first operation. The height H _f1o of the result of the first operation (ie, the first output data F _fo1 ) is the height H _fi1 -H _kd +1, the width W _f1o is the width W _fi1 -W _kd +1, and the number of channels C _{f1o is} the same as The number of channels C _fi1 .

For another example, if the size (height, width, number of channels) of the first part of the data F _fi1 is (3, 32, 16), and the size (height, width) of the first filter K _n is (3, 3), then The size (height, width, number of channels) of the first output data F _fo1 is (1, 30, 16).

In one embodiment, the first operator 135 adopts a systolic array structure. The first operator 135 divides the first partial data area into a plurality of first systolic array inputs, and performs a first operation on the first systolic array inputs respectively to obtain a plurality of first systolic array outputs. The size of each first systolic array output will be limited by the size of the systolic array. For example, the amount of elements output by the first systolic array is less than or equal to the capacity of the systolic array. Furthermore, these first systolic array outputs based on the same first partial data form the first output data.

For example, FIGS. 5A and 5B are schematic diagrams of input and output of a systolic array according to an embodiment of the present invention. Referring to FIG. 5A and FIG. 5B , it is assumed that the size of the systolic array is (number M _sa ×number N _sa ). The height H _a1o of the systolic array output SA _1o is 1, its width W _a1o can be the number M _sa , and its channel number C _a1o can be the number N _sa . Therefore, the first operator 135 divides the first part of the data into a systolic array input SA _1i (whose size is height H _a1i , width W _a1i , and the number of channels C _a1i ) and systolic array input SA _2i (whose size is height H _a2i , width W _a2i , number of channels C _a2i ). The two systolic array inputs SA _1i , SA _2i are respectively convolved with the weights of each channel of the filter K _n , and the systolic array outputs SA _1o , SA _2o are obtained accordingly. The height H _a2o of the systolic array output SA _2o is 1, the width W _a2o may be less than or equal to the number M _sa , and the number of channels C _a2o may be less than or equal to the number N _sa .

For another example, the size (height, width, number of channels) of the first part of the data is (3, 32, 16), the size of the systolic array is 16×16, and the size of the filter K _n is 3×3. The height H _a1o of the systolic array output SA _1o may be 1, the width W _a1o may be 16, and the number of channels C _a1o may be 16. In addition, the height H _a1i of the systolic array input SA _1i is 3, its width W _a1i is 18, and its channel number C _a1i is 16. On the other hand, after the first operator 135 distinguishes the systolic array input SA _1i from the first part of the data, the systolic array input SA _2i can be obtained. The height H _a2i of the systolic array input SA _2i is 3, its width W _a2i is 16, and its channel number C _a2i is 16. In addition, the height H _a2o of the systolic array output SA _2o is 1, its width W _a2o is 14 (ie, the width W _a2i - the width of the filter K _n +1), and its number of channels C _a2o is 16.

For another example, Tables (1) to (3) are the data of the first, second and fifteenth first part data stored in the feature temporary memory 131 (the rest of the data can be deduced by analogy): Table (1) I(0,2,15)~I(0,2,0) I(0,1,15)~I(0,1,0) I(0,0,15)~I(0,0,0) I(1,2,15)~I(1,2,0) I(1,1,15)~I(1,1,0) I(1,0,15)~I(1,0,0) I(2,2,15)~I(2,2,0) I(2,1,15)~I(2,1,0) I(2,0,15)~I(2,0,0) Table 2) I(0,3,15)~I(0,2,0) I(0,2,15)~I(0,2,0) I(0,1,15)~I(0,1,0) I(1,3,15)~I(1,2,0) I(1,2,15)~I(1,2,0) I(1,1,15)~I(1,1,0) I(2,3,15)~I(2,2,0) I(2,2,15)~I(2,2,0) I(2,1,15)~I(2,1,0) table 3) I(0,17,15)~I(0,17,0) I(0,16,15)~I(0,16,0) I(0,15,15)~I(0,15,0) I(1,17,15)~I(1,17,0) I(1,16,15)~I(1,16,0) I(1,15,15)~I(1,15,0) I(2, 17,15)~I(2,17,0) I(2,16,15)~I(2,16,0) I(2,15,15)~I(2,15,0) I(i1,j1,n1) represents the value of the read input data at the position (high position i1, wide position j1, channel position n1). The first FIFO unit 132 sequentially inputs the data to the first operator 135 from right to left and from top to bottom.

Table (4) is the data of the 16-channel 3×3 filter used in the convolution operation: Table (4) channel 0 channel 1 … Channel 14 Channel 15 F _d0 (2,2,15) F _d1 (2,2,15) … F _d14 (2,2,15) F _d15 (2,2,15) … … … … … F _d0 (2,2,0) F _d1 (2,2,0) … F _d14 (2,2,0) F _d15 (2,2,0) … … … … … F _d0 (0,0,0) F _d1 (0,0,0) … F _d14 (0,0,0) F _d15 (0,0,0) … … … … … F _d0 (0,0,0) F _d1 (2,2,15) … F _d14 (2,2,15) F _d15 (2,2,15) F _dn (i2,j2,n2) represents the read value of the nth filter at the position (high position i2, wide position j2, channel position n2).

Table (5) is the systolic array output: Table (5) A(0,0,0) A(0,0,1) … A(0,0,14) A(0,0,15) A(0,1,0) A(0,1,1) … A(0,1,14) A(0,1,15) … … … … … A(0,15,0) A(0,15,1) … A(0,15,14) A(0,15,15) A(i3,j3,n3) represents the value of the systolic array output at the position (high position i3, wide position j3, channel position n3), and its mathematical expression is: A(i3,j3,n3) = I(i3, j3,0)×F _dn3 (0,0,0)+I(i3,j3,1)×F _dn3 (0,0,1)+…+I(i3,j3,15) ×F _dn3 (0, 0,15) +I(i3,j3+1,0)×F _dn3 (0,1,0)+I(i3,j3+1,1) ×F _dn3 (0,1,1)+…+I (i3,j3+1,15) × F _dn3 (0,1,15) +I(i3,j3+2,0)×F _dn3 (0,2,0)+I(i3,j3+2,1 ) ×F _dn3 (0,2,1)+…+I(i3,j3+2,15) ×F _dn3 (0,2,15) +I(i3+1,j3,0)×F _dn3 (1 ,0,0)+I(i3+1,j3,1) ×F _dn3 (1,0,1)+…+I(i3+1,j3,15) ×F _dn3 (1,0,15) + I(i3+1,j3+1,0)×F _dn3 (1,1,0)+I(i3+1,j3+1,1) ×F _dn3 (1,1,1)+…+I( i3+1,j3+1,15) ×F _dn3 (1,1,15) +I(i3+1,j3+2,0)×F _dn3 (1,2,0)+I(i3+1, j3+2,1) ×F _dn3 (1,2,1)+…+I(i3+1,j3+2,15) ×F _dn3 (1,2,15) +I(i3+2,j3, 0)×F _dn3 (2,0,0)+I(i3+2,j3,1) ×F _dn3 (2,0,1)+…+I(i3+2,j3,15) ×F _dn3 ( 2,0,15) +I(i3+2,j3+1,0)×F _dn3 (2,1,0)+I(i3+2,j3+1,1) ×F _dn3 (2,1, 1)+…+I(i3+2,j3+1,15) ×F _dn3 (2,1,15) +I(i3+2,j3+2,0)×F _dn3 (2,2,0) +I(i3+2,j3+2,1) ×F _dn3 (2,2,1)+…+I(i3+2,j3+2,15)×F _dn3 (2,2,15)…( 1).

FIG. 6A is a schematic diagram of a systolic array output SA _3o according to an embodiment of the present invention. Referring to FIG. 6A , the height H _a3o of the systolic array output SA _3o is 1, the width W _a3o is 16, and the number of channels C _a3o is 16. And in the mathematical expression (1), i3 ∈ 0 representing the height, j3 ∈ 0~15 representing the width and n3 ∈ 0~15 representing the filter output channel.

FIG. 6B is a schematic diagram of a systolic array output SA _4o according to an embodiment of the present invention. Referring to FIG. 6B , the height H _a3o of the systolic array output SA _4o is 1, the width W _a4o is 14, and the number of channels C _a3o is 16. And in the mathematical expression (1), i3 ∈ 0 representing the height, j3 ∈ 16~29 representing the width and n3 ∈ 0~15 representing the filter output channel. Additionally, the completed output 601 is the systolic array output SA _3o of FIG. 6A , and the current processing output 602 is the systolic array output SA _4o .

By analogy, FIG. 6C is a schematic diagram of the systolic array output SA _4o according to an embodiment of the present invention. Referring to FIG. 6C , the height H _a3o of the systolic array output SA _5o is 1, the width W _a5o is 14, and the number of channels C _a3o is 16. In the mathematical expression (1), i3 ∈ 4 representing the height, j3 ∈ 16~29 representing the width and n3 ∈ 0~15 representing the filter output channel. Additionally, the current processing output 602 is the systolic array output SA _5o . These systolic array outputs SA _3o to SA _5o can form one or more first output data.

In one embodiment, the first operation is a convolution operation, and the first operation unit 135 reads the first part of the data in the first sliding direction for the input data stored in the memory 110 . The first arithmetic unit 135 divides the input data into a plurality of segments, and successively reads the next segment in a first sliding direction parallel to the height of the input data, and uses them as the first part of the data.

For example, FIG. 7A is a schematic diagram illustrating reading input data F _fi1 , F _fi2 according to an embodiment of the present invention. Referring to FIG. 7A , if the first operation of the first part of the data F _fi1 has been completed, the first operator 135 regards the first part of the data F _fi1 as having completed the input 701 , and goes further to the direction D1 (eg, the lower part of the drawing) The first part of the data F _fi2 of the next section in the input data F is read and input 702 as the current process.

FIG. 7B is a schematic diagram of the first output data F _fo1 , F _fo2 according to an embodiment of the present invention. Referring to FIG. 7B , the first output data F _fo1 is the output of the convolution operation performed on the first part of the data F _fi1 in FIG. 7A and is regarded as the completed output 703 . In addition, the first output data F _fo2 is the output of the convolution operation performed on the first partial data F _fi2 of FIG. 7A and is used as the current processing output 704 . The first output data F _fo2 is also arranged below the first output data F _fo1 according to the direction D1 of FIG. 7A .

FIG. 7C is a schematic diagram illustrating reading the input data F _fi3 according to an embodiment of the present invention. Referring to FIG. 7C , if the input 701 has been completed and the bottom of the input data F is reached, the first arithmetic unit 135 reads in the direction D2 (for example, the right side of the drawing) and from top to bottom (corresponding to the direction D1 in FIG. 7A ). The first part of the data F _fi3 of the next section in the data F is entered and entered 702 as the current process.

FIG. 7D is a schematic diagram of the first output data F _fo3 according to an embodiment of the present invention. Referring to FIG. 7D , the first output data F _fo3 is the output of the convolution operation performed on the first part of the data F _fi3 in FIG. 7C and is used as the current processing output 704 . Similarly, the current processing output 704 is arranged to the right of the completed output 703 .

FIG. 7E is a schematic diagram illustrating reading the input data F _fi4 according to an embodiment of the present invention. Referring to FIG. 7E, the first part of the data F _fi4 as the current processing input 702 is the last section in the input data.

FIG. 7F is a schematic diagram of the first output data F _fo4 according to an embodiment of the present invention. Referring to FIG. 7F , the first output data F _fo4 is the output of the convolution operation performed on the first part of the data F _fi4 in FIG. 7E and is used as the current processing output 704 . Similarly, the current processing output 704 is arranged below the completed output 703, and the convolution operation on the input data F is performed accordingly.

In another embodiment, the first arithmetic unit 135 reads the first part of the data in the second sliding direction (different from the first sliding direction) to the input data stored in the memory 110 . Similarly, the first arithmetic unit 135 divides the input data into a plurality of sections, and successively reads the next section in a second sliding direction parallel to the width of the input data, and uses them as the first part of the data.

For example, FIG. 8 is a schematic diagram illustrating reading input data according to an embodiment of the present invention. Referring to FIG. 8 , if the first operation of the first part of the data F _fi1 has been completed, the first calculator 135 regards the first part of the data F _fi1 as having completed the input 701 , and goes further to the direction D2 (for example, the right side of the figure). ) to read the first part of the data F _fi6 of the next section in the input data F and input 702 as the current process. Similarly, if the last segment of the row has been read in the direction D2, the first arithmetic unit 135 will read the segment below the first partial data F _fi1 . In addition, the arrangement of the first part of the data F _fi1 and other first part of the data (not shown) can refer to the above description, and will not be repeated here.

Referring to FIG. 2 , the first arithmetic unit 135 temporarily stores one or more first output data in the first temporary storage area of the first temporary storage memory 151 (step S250 ). Specifically, unlike the prior art that outputs the first output data to the memory 110 , the first output data of the embodiment of the present invention is output to the first temporary memory 151 of the second arithmetic unit 155 for temporary storage, thereby reducing the amount of memory 110 access times.

When the first output data temporarily stored in the first temporary storage memory 151 (or the first temporary storage area) is larger than the first predetermined amount of data, the second operation unit 155 performs a second operation on the first output data to obtain the first output data. 2. Output data (step S270). Specifically, in the existing multi-convolutional layer architecture, the next convolutional layer needs to wait until the previous convolutional layer operates on all its input data and outputs it to the main memory, and then reads the output from the previous convolutional layer from the autonomous memory. Enter data. Different from the prior art, in addition to the temporary storage in the storage medium other than the memory 110 (for example, the first temporary memory 151 or the second temporary memory 171 ), the embodiment of the present invention further satisfies the requirements of the next convolutional layer. In the case of the required size of the input data (ie, the first predetermined amount of data), the convolution operation of the next convolution layer can be triggered. At the same time, if the operation of the previous convolutional layer on all input data has not been completed, the operations of the two convolutional layers can be performed simultaneously. In other words, when the second operation unit 155 performs the second operation on the first output data, the first operation unit 135 continues to perform the first operation on the input data.

It is worth noting that the second part of the data input by the second operation includes the first output data temporarily stored in the first temporary storage memory 151, and the size of the second output data is related to the second filter of the second operation size. Assume that the second operation is a depthwise convolution operation. Each filter of the depthwise convolution operation corresponds to only one channel of data in the second part of the data, respectively. That is, any filter of the depthwise convolution operation only convolves the data of one channel. Therefore, the number of filters for the depthwise convolution operation is usually equal to the number of channels of the second part of the data. However, each filter of the convolution operation performs a convolution operation on the data of all channels. In addition, as long as the height of the temporarily stored first output data increases to the height of the filter and the width of the first output data increases to the width of the filter, the filter can be combined with these temporarily stored first output data (as the second output data) Part of the data) to perform depthwise convolution operations.

In one embodiment, it is assumed that the height of each filter used in the depthwise convolution operation is H _kd , and the width of the filter is W _kd . The height of the first output data of each section is H _f1o , and the width of the first output data is W _f1o . When the first output data temporarily stored in the first temporary storage memory 151 or the first temporary storage area is greater than W _kd ×H _kd , the second arithmetic unit 155 can perform the second operation. When the first output data temporarily stored in the first temporary storage memory 151 or the first temporary storage area is larger than the first predetermined amount of data, the first output data temporarily stored in the first temporary storage memory 151 or the first temporary storage area The height of the composition of the material is M _H × H _f1o and the width of the composition is M _W ×W _f1o . MH and MW are multiples and are positive integers, _MH × _{H f1o} is not less than H _kd , and MW × _{W f1o} is not less than _{W kd} . In other words, the height M _H × H _f1o of the temporarily stored first output data is smaller than the height H _kd of the filter and the width M _W ×W _f1o of the temporarily stored first output data is smaller than the width W of the filter In the case of _kd , the second operator 155 will continue to wait for the next first output data or systolic array output until the temporarily stored high M _H ×H _f1o of the first output data is greater than or equal to the filter high H _kd And the width M _W ×W _f1o of the temporarily stored first output data is greater than or equal to the width W _kd of the filter.

For example, FIG. 9A is a schematic diagram illustrating a trigger condition of the second operation according to an embodiment of the present invention, and FIG. 9B is a schematic diagram of temporarily stored first output data according to an embodiment of the present invention. Referring to FIG. 9A , the completed input 901 in the input data corresponds to the completed output 903 of the first output data. If the size of the current processing output 904 corresponding to the current processing input 902 and the completed output 903 conform to the size required by the second operation, the second operation can be triggered.

Referring to FIG. 9B , it is assumed that the completed output 903 and the current processing output 904 of FIG. 9A form temporarily stored first output data F _tfo . The size of the systolic array used by the first operator 135 is 16×16, wherein the width W _tfo1 of the output of the systolic array may be 16 or the width W _tfo2 may be 14. Assume that the height of each filter used in the depthwise convolution operation is 3, and the width of its filter is 3. The widths W _tfo1 and W _tfo2 are all greater than 3. If the fifth systolic array output is temporarily stored in the first temporary memory 151, the size (height, width, channel number) of the first to fifth systolic array outputs is (1, 16, 16) or (1) , 14, 16), that is, the number of channels C _tfo is 16) The size formed by the output has satisfied the size of 3 × 3. That is, a systolic array output with a height of 1 is stacked three layers, so that the stacked height is 3. At this time, these systolic array outputs can be used as the second part of the data and can be used for the second operation.

It should be noted that in FIGS. 9A and 9B , the second operation is triggered as long as the number of stacked layers is equal to the high level of the filter. However, in other embodiments, the number of stacked layers may be greater than the height of the filter.

For the depthwise convolution operation, FIG. 10A is a schematic diagram of a second operation according to an embodiment of the present invention. Referring to FIG. 10A , it is assumed that the size (height, width, number of channels) of the second part of data F _si1 is (5, 30, 16), and the size of the filter F _d used in the depthwise convolution operation is 3×3. I(i4,j4,n4) represents the value of the second part of the data at the position (high position i4, wide position j4, channel position n4). F _dn4 (i5, j5, n5) represents the read value of the n4th filter at the position (high position i5, wide position j5). A(i4,j4,n4) represents the value of the second output data or systolic array output at the position (high position i4, wide position j4, channel position n4), and its mathematical expression is: A(i4,j4,n4) =I(i4,j4,n4)×F _dn4 (0,0)+I(i4,j4+1,n4)×F _dn4 (0,1)+ I(i4,j4+2,n4)×F _dn4 (0,2) +I(i4+1,j4,n4)×F _dn4 (1,0)+I(i4+1,j4+1,n4)×F _dn4 (1,1)+ I(i4+ 1,j4+2,n4)×F _dn4 (1,2) +I(i4+2,j4,n)×F _dn4 (2,0)+I(i4+2,j4+1,n4)×F _dn4 (2,1)+I(i4+2,j4+2,n4)×F _dn4 (2,2)…(2).

FIG. 10B is a schematic diagram of the second output data F _so1 according to an embodiment of the present invention. Referring to FIG. 10B , it is assumed that the size (height H _so1 , width W _so1 , and number of channels C _so1 ) of the currently processed second output data F _so1 is (1, 28, 16). Among them, each value in the second output data F _so1 is: A(0,0,n4)=I(0,0,n4)×F _dn4 (0,0)+I(0,1,n4)×F _dn4 (0,1)+ I(0,2,n4)×F _dn4 (0,2) +I(1,0,n4)×F _dn4 (1,0)+I(1,1,n4)× F _dn4 (1,1)+ I(1,2,n4)×F _dn4 (1,2) +I(2,0,n)×F _dn4 (2,0)+I(2,1,n4) ×F _dn4 (2,1)+ I(2,2,n4)×F _dn4 (2,2)…(3) A(0,1,n4)=I(0,1,n4)×F _dn4 ( 0,0)+I(0,2,n4)×F _dn4 (0,1)+ I(0,3,n4)×F _dn4 (0,2) +I(1,1,n4)×F _dn4 (1,0)+I(1,2,n4)×F _dn4 (1,1)+ I(1,3,n4)×F _dn4 (1,2) +I(2,1,n)×F _dn4 (2,0)+I(2,2,n4)×F _dn4 (2,1)+ I(2,3,n4)×F _dn4 (2,2)…(4) A(0,27, n4)=I(0,27,n4)×F _dn4 (0,0)+I(0,28,n4)×F _dn4 (0,1)+ I(0,29,n4)×F _dn4 (0 ,2) +I(1, 27,n4)×F _dn4 (1,0)+I(1,28,n4)×F _dn4 (1,1)+ I(1,29,n4)×F _dn4 ( 1,2) +I(2, 27,n)×F _dn4 (2,0)+I(2,28,n4)×F _dn4 (2,1)+ I(2,29,n4)×F _dn4 (2,2)…(5) and so on and so on, so I won’t repeat them here.

FIG. 10C is a schematic diagram of the second output data F _so2 according to an embodiment of the present invention. Referring to FIG. 10C , the completed output 101 is the second output data F _so1 of FIG. 10B . The second output data F _so2 is the current processing output 102 , and its size may be the same as the second output data F _so1 of FIG. 10B . Among them, each value in the second output data F _so2 is: A(1,0,n4)=I(1,0,n4)×F _dn4 (0,0)+I(1,1,n4)×F _dn4 (0,1)+ I(1,2,n4)×F _dn4 (0,2) +I(2,0,n4)×F _dn4 (1,0)+I(2,1,n4)× F _dn4 (1,1)+ I(2,2,n4)×F _dn4 (1,2) +I(3,0,n)×F _dn4 (2,0)+I(3,1,n4) ×F _dn4 (2,1)+ I(3,2,n4)×F _dn4 (2,2)…(6) A(1,1,n4)=I(1,1,n4)×F _dn4 ( 0,0)+I(1,2,n4)×F _dn4 (0,1)+ I(1,3,n4)×F _dn4 (0,2) +I(2,1,n4)×F _dn4 (1,0)+I(2,2,n4)×F _dn4 (1,1)+ I(2,3,n4)×F _dn4 (1,2) +I(3,1,n)×F _dn4 (2,0)+I(3,2,n4)×F _dn4 (2,1)+ I(3,3,n4)×F _dn4 (2,2)…(7) A(1,27, n4)=I(1,27,n4)×F _dn4 (0,0)+I(1,28,n4)×F _dn4 (0,1)+ I(1,29,n4)×F _dn4 (0 ,2) +I(2, 27,n4)×F _dn4 (1,0)+I(2,28,n4)×F _dn4 (1,1)+ I(2,29,n4)×F _dn4 ( 1,2) +I(3, 27,n)×F _dn4 (2,0)+I(3,28,n4)×F _dn4 (2,1)+ I(3,29,n4)×F _dn4 (2,2)…(8) and so on and so on, so I won’t repeat them here.

FIG. 10D is a schematic diagram of the second output data F _so3 according to an embodiment of the present invention. Referring to FIG. 10C , the second output data F _so3 is the current processing output 102 , and its size may be the same as the second output data F _so1 of FIG. 10B . Among them, each value in the second output data F _so3 is: A(2,0,n4)=I(2,0,n4)×F _dn4 (0,0)+I(2,1,n4)×F _dn4 (0,1)+ I(2,2,n4)×F _dn4 (0,2) +I(3,0,n4)×F _dn4 (1,0)+I(3,1,n4)× F _dn4 (1,1)+ I(3,2,n4)×F _dn4 (1,2) +I(4,0,n)×F _dn4 (2,0)+I(4,1,n4) ×F _dn4 (2,1)+ I(4,2,n4)×F _dn4 (2,2)…(9) A(2,1,n4)=I(2,1,n4)×F _dn4 ( 0,0)+I(2,2,n4)×F _dn4 (0,1)+ I(2,3,n4)×F _dn4 (0,2) +I(3,1,n4)×F _dn4 (1,0)+I(3,2,n4)×F _dn4 (1,1)+ I(3,3,n4)×F _dn4 (1,2) +I(4,1,n)×F _dn4 (2,0)+I(4,2,n4)×F _dn4 (2,1)+ I(4,3,n4)×F _dn4 (2,2)…(10) A(2,27, n4)=I(2,27,n4)×F _dn4 (0,0)+I(2,28,n4)×F _dn4 (0,1)+ I(2,29,n4)×F _dn4 (0 ,2) +I(3, 27,n4)×F _dn4 (1,0)+I(3,28,n4)×F _dn4 (1,1)+ I(3,29,n4)×F _dn4 ( 1,2) +I(4, 27,n)×F _dn4 (2,0)+I(4,28,n4)×F _dn4 (2,1)+ I(4,29,n4)×F _dn4 (2,2)…(11) The rest are analogous and will not be repeated here.

In one embodiment, the second operator 155 adopts a systolic array structure. The second arithmetic unit 155 divides the second partial data (ie, the partially temporarily stored first output data) into a plurality of second systolic array inputs, and performs a second operation on the second systolic array inputs respectively, so as to obtain a plurality of second systolic array inputs. A second systolic array output. The size of each second systolic array output will be limited by the size of the systolic array. For example, the amount of elements output by the second systolic array is less than or equal to the capacity of the systolic array. Furthermore, these second systolic array outputs based on the same second partial data constitute the second output data. Taking FIG. 10B as an example, if the size of the systolic array is 16×16, the second output data F _so1 includes 1×16×16 and 1×12×16 outputs of the second systolic array.

For the next convolutional layer, FIG. 11 is a flowchart of a data processing method based on a convolutional neural network according to an embodiment of the present invention. Referring to FIG. 11 , in one embodiment, the second arithmetic unit 155 may temporarily store one or more second output data in the second temporary storage area of the second temporary storage memory 171 (step S111 ) (step S280 ) . Specifically, similarly, the embodiment of the present invention temporarily stores the output of the previous convolutional layer in the buffer of the next convolutional layer instead of directly outputting the output data to the memory 110 .

When the second output data temporarily stored in the first temporary storage memory 171 or the second temporary storage area is larger than the second predetermined amount of data, the third operator 175 may perform a third operation on the second output data to obtain the third output data. Data is output (step S113). Specifically, the third part of the data input by the third operation includes the second output data temporarily stored in the second temporary storage memory 171 , and the size of the third part of the data is related to the filter size of the third operation. Assume that the third operation is a pointwise convolution operation. The size of each filter of the point-by-point convolution operation is only 1×1. Similar to the convolution operation, each filter of the point-by-point convolution operation also performs a convolution operation on the data of all channels. In addition, as long as the height of the temporarily stored second output data is increased to the height of the filter (1) and the width of the second output data is increased to the width of the filter (1), the filter can The second output data (as the third part of data) is subjected to a depthwise convolution operation.

In one embodiment, according to FIG. 10B to FIG. 10D , the second output data of each stroke can satisfy the size required for the point-by-point convolution operation. Therefore, the second FIFO unit 172 can sequentially input each second output data to the third arithmetic unit 175 . The third operator 175 may perform a third operation on each of the temporarily stored second output data.

For example, FIG. 12A is a schematic diagram of the temporarily stored first output data F _tfo according to an embodiment of the present invention, and FIG. 12B is a schematic diagram of the temporarily stored second output data F _tso according to an embodiment of the present invention. Referring to FIGS. 12A and 12B , if the second operator 155 has completed the convolution operation on a part of the temporarily stored first output data, the first temporary storage memory 151 can temporarily store the size (high, width, number of channels) is the second systolic array output of (1, W _so21 , C _so2 ) or the second output data of (1, W _so21+ W _so21 , C _so2 ), and becomes the second output data temporarily stored accordingly _Ftso . The number of channels C _so2 is the same as the number of channels C _tf2 .

FIG. 13A is a schematic diagram of a third operation according to an embodiment of the present invention. Referring to FIG. 13A , the third arithmetic unit 175 uses the temporarily stored second output data F _tso in FIG. 12B as the third partial data F _ti (its width W _so3 is W _so21+ W _so21 ), and the third partial data F _ti performs a third operation with the filter F _p used for the pointwise convolution operation.

FIG. 13B is a schematic diagram of the third output data F _to according to an embodiment of the present invention. Referring to FIG. 13A and FIG. 13B , the size of the third output data F _to is equal to the size of the third partial data F _ti . That is, the width W _{to1 is} the same as the width W _so3 , and the number of channels C _{to1 is} the same as the number of channels C _so2 .

In one embodiment, the third operator 175 adopts a systolic array structure. The third operator 175 divides the third partial data area into a plurality of third systolic array inputs, and performs a third operation on the third systolic array inputs respectively to obtain a plurality of third systolic array outputs. The size of each third systolic array output will be limited by the size of the systolic array. For example, the amount of elements output by the third systolic array is less than or equal to the capacity of the systolic array. Furthermore, these third systolic array outputs based on the same third partial data (ie, the partially temporarily stored second output data) constitute the third output data. For example, if the size of the third part of the data is 1×28×16 and the size of the systolic array is 16×16, the third output data includes the third systolic array outputs of 1×16×16 and 1×12×16 .

For example, FIG. 14A is a schematic diagram of a systolic array output SA _6o according to an embodiment of the present invention. Please refer to FIG. 14A , Table (6) is the data of the second output data stored in the second temporary memory 171: Table (6) I(0,0,15) I(0,0,14) … I(0,0,1) I(0,0,0) I(0,1,15) I(0,1,14) … I(0,1,1) I(0,1,0) … … … … … I(0,15,15) I(0,15,14) … I(0,15,1) I(0,15,0) I(i6,j6,n6) represents the value of the read input data at the position (high position i6, wide position j6, channel position n6). The second FIFO unit 172 sequentially inputs the data to the third operator 175 from right to left and from top to bottom.

Table (7) is the data of the 16-channel 1×1 filter used in the point-by-point convolution operation: Table (8) channel 0 channel 1 … Channel 14 Channel 15 F _p0 (0,0,15) F _p1 (0,0,15) … F _p14 (0,0,15) F _p15 (0,0,15) … … … … … F _p0 (0,0,1) F _p1 (0,0,1) … F _p14 (0,0,1) F _p15 (0,0,1) F _p0 (0,0,0) F _p1 (0,0,0) … F _p14 (0,0,0) F _p15 (0,0,0) F _dn (i7, j7, n7) represents the read value of the nth filter at position (high position i7, wide position j7, channel position n7).

Table (9) is the systolic array output: Table (9) A(0,0,0) A(0,0,1) … A(0,0,14) A(0,0,15) A(0,1,0) A(0,1,1) … A(0,1,14) A(0,1,15) … … … … … A(0,15,0) A(0,15,1) … A(0,15,14) A(0,15,15) A(i6,j6,n6) represents the value of the systolic array output at the position (high position i6, wide position j6, channel position n6), and its mathematical expression is: A(i6,j3,n6) = I(i6, j6,0)×F _dn6 (0,0,0)+I(i6,j6,1)×F _dn6 (0,0,1)+ …+I(i6,j6,15) ×F _dn6 (0, 0,15)…(12).

Therefore, each value of the systolic array output SA _6o is (𝑛6∈0~15): A(0,0,n6) = I(0,0,0)×F _dn6 (0,0,0)+I(0 ,0,1)×F _dn6 (0,0,1)+ …+I(0,0,15) ×F _dn6 (0,0,15)…(13); A(0,1,n6) = I(0,1,0)×F _dn6 (0,0,0)+I(0,1,1)×F _dn6 (0,0,1)+ …+I(0,1,15)×F _dn6 (0,0,15)…(14). A(0,15,n6) = I(0,15,0)×F _dn6 (0,0,0)+I(0,15,1)×F _dn6 (0,0,1)+ …+I (0,15,15) ×F _dn6 (0,0,15)…(15), and the rest are analogous and will not be repeated.

For another example, FIG. 14B is a schematic diagram of a systolic array output SA _7o according to an embodiment of the present invention. Please refer to FIG. 14A, the table (10) is the data of the second output data stored in the second temporary memory 171: Table (10) I(0,16,15) I(0,16,14) … I(0,16,1) I(0,17,0) I(0,17,15) I(0,17,14) … I(0,17,1) I(0,17,0) … … … … … I(0,27,15) I(0,27,14) … I(0,27,1) I(0,27,0)

Table (11) is the systolic array output: Table (11) A(0,16,0) A(0,16,1) … A(0,16,14) A(0,16,15) A(0,17,0) A(0,17,1) … A(0,17,14) A(0,17,15) … … … … … A(0,27,0) A(0,27,1) … A(0,27,14) A(0,27,15)

Therefore, each value of the systolic array output SA _7o is (𝑛6∈0~15): A(0,16,n6) = I(0,16,0)×F _dn6 (0,0,0)+I(0 ,16,1)×F _dn6 (0,0,1)+ …+I(0,16,15) ×F _dn6 (0,0,15)…(16); A(0,17,n6) = I(0,17,0)×F _dn6 (0,0,0)+I(0,17,1)×F _dn6 (0,0,1)+ …+I(0,17,15)×F _dn6 (0,0,15)…(17). A(0,27,n6) = I(0,27,0)×F _dn6 (0,0,0)+I(0,27,1)×F _dn6 (0,0,1)+ …+I (0,27,15) ×F _dn6 (0,0,15)…(18), and the rest are analogous and will not be repeated. Furthermore, the systolic array output SA _6o of FIG. 14A is the completed output 141 , and the systolic array output SA _7o is the current processing output 142 .

For another example, FIG. 14C is a schematic diagram of a systolic array output SA _8o according to an embodiment of the present invention. Please refer to FIG. 14A, the table (12) is the data of the second output data stored in the second temporary memory 171: Table (12) I(2,16,15) I(2,16,14) … I(2,16,1) I(2,17,0) I(2,17,15) I(2,17,14) … I(2,17,1) I(2,17,0) … … … … … I(2,27,15) I(2,27,14) … I(2,27,1) I(2,27,0)

Table (13) is the systolic array output: Table (13) A(2,16,0) A(2,16,1) … A(2,16,14) A(2,16,15) A(2,17,0) A(2,17,1) … A(2,17,14) A(2,17,15) … … … … … A(2,27,0) A(2,27,1) … A(2,27,14) A(2,27,15)

Therefore, each value of the systolic array output SA _8o of the last current processing output 142 is (𝑛6∈0~15): A(2,16,n6) = I(2,16,0)×F _dn6 (0,0 ,0)+I(2,16,1)×F _dn6 (0,0,1)+ …+I(2,16,15) ×F _dn6 (0,0,15)…(19); A( 2,17,n6) = I(2,17,0)×F _dn6 (0,0,0)+I(2,17,1)×F _dn6 (0,0,1)+ …+I(2 ,17,15) ×F _dn6 (0,0,15)…(20). A(2,27,n6) = I(2,27,0)×F _dn6 (0,0,0)+I(2,27,1)×F _dn6 (0,0,1)+ …+I (2, 27, 15) ×F _dn6 (0, 0, 15)…(21), and the rest are analogous and will not be repeated.

In one embodiment, when the third operator 175 is running the third calculation, the first operator 135 and the second operator 155 continue to run the first calculation and the second calculation, respectively. That is, if the operations of the first operator 135 and the second operator 155 on all the input data have not been completed, the operations of the first operator 135, the second operator 155 and the third operator 175 can be performed together.

Referring to FIG. 2 , finally, the third operation unit 175 outputs the third output data obtained by the third operation to the memory 110 (step S290 ).

In order to facilitate the understanding of the complete process, another embodiment is described below. FIG. 15 is a flowchart of a data processing method of the MobileNet architecture according to an embodiment of the present invention. Referring to FIG. 15 , the first arithmetic unit 135 reads the data of the width of the defined section from the input data in the memory 110 as the first part of data (step S1501 ). The first operator 135 determines whether the number of pieces currently processed is greater than or equal to (the size of the first filter - 1) and whether the remainder obtained by dividing this number with the first stride used in the first operation is 1 (step S1503). If the conditions of step S1503 are met, the first FIFO unit 132 sequentially outputs the first part of the data to the first arithmetic unit 135 (step S1505 ). The first operator 135 reads the weights in the first filter used in the convolution operation from the memory 110 (step S1511 ), performs the convolution operation (step S1513 ), and outputs the obtained first output data to the first temporary filter. stored in the memory 151 (step S1515). The first operator 135 determines whether all the data of the current bar (the size of which is the same as the filter size) in the current segment has been subjected to the convolution operation (step S1517 ). If the data has not completed the convolution operation, the first FIFO unit 132 continues to output the first part of the data to the first operator 135 (step S1505). If the convolution operation has been performed on all the data of this piece, the first operator 135 determines whether all the data of all the pieces in the current segment have been subjected to the convolution operation (step S1518 ). If one or more pieces of data in the current segment have not completed the convolution operation or the condition of step S1503 is satisfied, the first operator 135 continues to process the next piece of data from the input data in the memory 110 (step S1507 ) . If the convolution operation has been performed on all the data in each of the current segments, the first operator 135 determines whether all the data in all the segments have been subjected to the convolution operation (step S1519 ). If there is still data in the segment that has not completed the convolution operation, the first operator 135 continues to process the data in the next segment (step S1509 ). In addition, the first operator 135 resets the number of bars currently processed to zero, and sets the number of widths currently processed as: the number of original widths + the width of the segment - (the first stride used in the first operation - 1 + the second The second stride used by the operation - 1). If all data of all sections have been subjected to convolution operations, the first operator 135 completes all convolution operations on the input data (step S1520 ).

The second calculator 155 determines whether the first output data temporarily stored in the first temporary storage memory 151 is larger than the first predetermined data amount (step S1521 ). The size of the second filter is 3×3, for example, the second arithmetic unit 155 determines whether there are three pieces of the temporarily stored first output data. The second operator 155 determines whether the remainder obtained by the number of strips processed in the first operation and the second stride used in the second operation is equal to zero (step S1523). If the number is zero, the second arithmetic unit 155 reads the temporarily stored first output data as the second partial data (step S1525).

If the conditions in steps S1521 and S1523 are not met, the second arithmetic unit 155 determines whether the currently processed data is the first part of the first data (step S1531 ). If step S1531 does not meet the conditions, all the second operations are ended (step S1540). On the other hand, the second operator 155 reads the weights in the second filter used for the depthwise convolution operation from the memory 110 (step S1533 ), performs the depthwise convolution operation (step S1535 ), and converts the obtained first depth The second output data is output to the second temporary memory 171 (step S1537). The second operator 155 determines whether all data of all stripes in the current segment have been subjected to depthwise convolution operations (step S1538 ). If one or more pieces of data in the current segment have not completed the convolution operation, the second operator 155 moves to the next point index (eg, a second step away) (step S1527 ) and processes the next data . Until all the data of all the strips in the first temporary memory 151 have been subjected to the depthwise convolution operation, the second operator 155 sets the number of currently processed strips as: the original number of strips+1, and the currently processed The width number is reset to zero (step S1539). Next, the second operator 155 completes all depthwise convolution operations on the second input data (step S1540).

The third arithmetic unit 175 determines whether the second output data temporarily stored in the second temporary storage memory 171 has reached one second output data (step S1541 ). The third arithmetic unit 175 reads the temporarily stored second output data as the third partial data (step S1543). The third operator 175 reads the weights in the third filter used in the point-by-point convolution operation from the memory 110 (step S1551 ), performs the point-by-point convolution operation (step S1553 ), and outputs the obtained third output data to the memory 110 (step S1555). The third operator 175 determines whether all the data of all the strips in the second temporary memory 171 have been subjected to the point-by-point convolution operation. If these data have not completed the point-by-point convolution operation, the second FIFO unit 172 continues to output the third part of the data to the third operator 175 . If all the data in the second temporary memory 171 have completed the point-by-point convolution operation, the third operator 175 completes all the point-by-point convolution operations on the third part of the data (step S1560 ).

Embodiments of the present invention further provide a computer-readable storage medium (eg, a hard disk, an optical disk, a flash memory, a solid state disk (SSD), etc.) for storing program codes. The arithmetic circuit 100 or other processors can load the program code, and execute the corresponding flow of one or more data processing methods according to the embodiments of the present invention. These processes can be referred to the above description, and will not be repeated here.

To sum up, in the computing circuit, data processing method and computer-readable storage medium based on the convolutional neural network according to the embodiments of the present invention, the first output data and/or the second output data are temporarily stored and not output to memory, and can start the second operation and/or the third operation when the temporarily stored data meets the size required by the second operation and/or the third operation. Thereby, the access times of the memory can be reduced, and the operation efficiency can be improved.

Although the present invention has been disclosed above by the embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the technical field can make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, The protection scope of the present invention shall be determined by the scope of the appended patent application.

100: arithmetic circuit 110: memory 120: processing element 131: feature temporary memory 151: first temporary memory 171: second temporary memory 132: first FIFO unit 172: second FIFO Unit 133: first weight buffer 153: second weight buffer 173: third weight buffer 135: first operator 155: second operator 175: third operator S210~S290, S111~S113, S1501~ S1560: Step F: Input data H, H _fi1 , H _kd , H _f1o , H _a1o , H _a2o , H _a1i , H _a2i , H _a3o , H _s1o : high W, W _fi1 , W _kd , W _f1o , W _a1o , W _a2o , W _a1i , W _a2i , W _a3o , W _a4o , W _a5o , W _tfo1 , W _tfo2 , W _s1o , W _so21 , W _so22 , W _so3 , W _to1 : Wide C, C _fi1 , C _f1o , C _a1o , C _a2o , C _a1i , C _a2i , C _a3o , C _tfo , C _s1o , C _so2 , C _to1 : the number of channels F _fi1 , F _fi2 , F _fi3 , F _fi4 , F _fi6 : the first part of the data K _n , F _d , F _p : filters F _fo1 , F _fo2 , F _fo3 , F _fo4 , F _tfo : first output data SA _1i , SA _2i : systolic array inputs SA _1o , SA _2o , SA _3o , SA _4o , SA _5o , SA _6o , SA _7o , SA _8o : Systolic Array Output 601, 703, 903, 141: Completed Output 602, 704, 904, 142: Current Processing Output D1, D2: Direction 701, 901: Completed Input 702, 902: Current processing input i4: high position j4: wide position n4: channel position I(,,,), A(,,,): value F _dn4 (,,): weight F _si1 : second part of data F _so1 , F _so2 , F _so3 , F _tso : the second output data F _ti : the third part data F _to : the third output data

FIG. 1 is a block diagram of components of an operation circuit based on a convolutional neural network according to an embodiment of the present invention. FIG. 2 is a flowchart of a data processing method based on a convolutional neural network according to an embodiment of the present invention. FIG. 3 is a schematic diagram of input data according to an embodiment of the present invention. FIG. 4 is a schematic diagram of a first operation according to an embodiment of the present invention. 5A and 5B are schematic diagrams of input and output of a systolic array according to an embodiment of the present invention. 6A-6C are schematic diagrams of systolic array output according to an embodiment of the present invention. FIG. 7A is a schematic diagram illustrating reading input data according to an embodiment of the present invention. FIG. 7B is a schematic diagram of the first output data according to an embodiment of the present invention. 7C is a schematic diagram illustrating reading input data according to an embodiment of the present invention. FIG. 7D is a schematic diagram of the first output data according to an embodiment of the present invention. FIG. 7E is a schematic diagram illustrating reading input data according to an embodiment of the present invention. FIG. 7F is a schematic diagram of the first output data according to an embodiment of the present invention. FIG. 8 is a schematic diagram illustrating reading input data according to an embodiment of the present invention. FIG. 9A is a schematic diagram illustrating a trigger condition of the second operation according to an embodiment of the present invention. FIG. 9B is a schematic diagram of temporarily stored first output data according to an embodiment of the present invention. FIG. 10A is a schematic diagram of a second operation according to an embodiment of the present invention. 10B to 10D are schematic diagrams of second output data according to an embodiment of the present invention. FIG. 11 is a flowchart of a data processing method based on a convolutional neural network according to an embodiment of the present invention. 12A is a schematic diagram of temporarily stored first output data according to an embodiment of the present invention. FIG. 12B is a schematic diagram of temporarily stored second output data according to an embodiment of the present invention. FIG. 13A is a schematic diagram of a third operation according to an embodiment of the present invention. FIG. 13B is a schematic diagram of a third output data according to an embodiment of the present invention. 14A-14C are schematic diagrams of systolic array output according to an embodiment of the present invention. FIG. 15 is a flowchart of a data processing method of the MobileNet architecture according to an embodiment of the present invention.

S210~S290: Steps

Claims (13)

A data processing method based on a convolutional neural network (Conventional Neural Network, CNN), comprising: read an input data from a memory; A first operation is performed on a first part of the input data to obtain a first output data, wherein the first operation is provided with a first filter, and the size of the first output data is related to the first output data the size of the first filter of the operation and the size of the first part of the data; temporarily storing the first output data in a first temporary storage area; When the first output data temporarily stored in the first temporary storage area is larger than a first predetermined amount of data, a second operation is performed on the first output data to obtain a second output data, wherein the second operation a second filter is provided, and the size of the second output data is related to the size of the second filter of the second operation; temporarily storing the second output data in a second temporary storage area; and A third output data obtained by performing a third operation on the second output data is output to the memory, Wherein, when the second operation is performed on the first output data, the first operation is continuously performed on the input data. The data processing method based on a convolutional neural network as claimed in claim 1, wherein the second operation is different from the third operation, and the step of temporarily storing the second output data in the second temporary storage area memory includes the following steps: : When the second output data temporarily stored in the second temporary storage area is larger than a second predetermined data amount, the third operation is performed on the second output data to obtain the third output data, wherein the third operation A third filter is provided, and the size of the third output data is related to the size of the third filter. The data processing method based on a convolutional neural network according to claim 2, wherein the step of outputting the third output data obtained by performing the third operation on the second output data to the memory comprises: When the third operation is performed on the second output data, the first operation and the second operation are continuously performed. The data processing method based on a convolutional neural network as claimed in claim 1, wherein the second operation is a depthwise convolution operation, the height of the second filter is H _kd , and the first filter is H kd . The width of the two filters is W _kd , the height of the first output data is H _f1o , the width of the first output data is W _f1o , H _kd , W _kd , H _f1o and W _f1o are positive integers, and The step of performing the second operation on the first output data includes: when the first output data temporarily stored in the first temporary storage area is greater than W _kd ×H _kd , performing the second operation, wherein the first temporary storage area The height of the maximum amount of data temporarily stored in the storage area is M _H ×H _f1o and the width is M _W ×W _f1o , M _H and M _W are multiples and are positive integers, and M _H ×H _f1o is not less than H _kd , and M _W ×W _f1o is not smaller than W _kd . The data processing method based on a convolutional neural network as claimed in claim 2, wherein the third operation is a pointwise convolution operation, the height and width of the third filter are both 1, and the third filter is The step of performing the third operation on the third input data includes: The third operation is performed on each of the temporarily stored second output data. The data processing method based on a convolutional neural network according to claim 4, wherein the first operation is a convolution operation, and the step of reading the input data from the memory comprises: The first part of the data is read in a first sliding direction of the input data, wherein the first sliding direction is parallel to the height of the input data. The data processing method based on a convolutional neural network according to claim 4, wherein the first operation is a convolution operation, and the step of reading the input data from the memory comprises: The first part of the data is read in a second sliding direction of the input data, wherein the second sliding direction is parallel to the width of the input data. The data processing method based on a convolutional neural network according to claim 2, wherein the first operation is performed on the first part of the input data, the second operation is performed on the first output data, or The step of performing the third operation on the second output data includes: dividing the first portion of data into a plurality of first systolic array inputs; and The first operation is respectively performed on the first systolic array inputs to obtain a plurality of first systolic array outputs, wherein the first systolic array outputs constitute the first output data. An arithmetic circuit based on a convolutional neural network, comprising: a memory for storing an input data; A processing element, coupled to the memory, includes: a first operator for performing a first operation on a first part of the input data to obtain a first output data, and temporarily storing the first output data in a first temporary part of the processing element storage memory, wherein the first operation is provided with a first filter, and the size of the first output data is related to the size of the first filter of the first operation and the size of the first part of the data; a second arithmetic unit for performing the second operation on the first output data to obtain a second output data when the first output data temporarily stored in the first temporary storage memory is greater than a first predetermined amount of data output data, and temporarily store the second output data to a second temporary storage memory, wherein the second operation is provided with a second filter, and the size of the second output data is related to the second output data of the second operation the size of the filter; the second temporary memory for storing the second output data; and a third operator for outputting a third output data obtained by performing a third operation on the second output data to the memory, Wherein, when the second arithmetic unit performs the second arithmetic operation, the first arithmetic unit continues to perform the first arithmetic operation. The operation circuit based on a convolutional neural network as claimed in claim 9, wherein the first operation unit has a first maximum operation amount within a unit time, and the second operation unit has a first maximum operation amount within the unit time. Two maximum computations, the third operator has a third maximum computation within the unit time, the first maximum computation is greater than the second maximum computation, the first maximum computation is greater than the third maximum computation . The operation circuit based on a convolutional neural network as claimed in claim 9, wherein when the third operation unit runs the third operation, the first operation unit and the second operation unit continue to run the first operation and the second operation unit. second operation. The computing circuit based on a convolutional neural network according to claim 9, wherein the first temporary memory and the second temporary memory are a static random access memory. A computer-readable storage medium is used for storing a program code, which is loaded by a processor to execute the data processing method based on a convolutional neural network as described in any one of claim items 1 to 8.

Images