TWI840715B - Computing circuit and data processing method based on convolution neural network and computer readable storage medium - Google Patents

Abstract

A computing circuit and a data processing method based on convolution neural network and a computer readable storage medium are provided. An input data is obtained from the memory. The first computation is performed on the first part data of the input data, to obtain a first output data. The first output data is buffered in a first buffer area. When the buffered first output feature map is larger than a first predetermined data amount, the second computation is performed on the first output data. The second output data is buffered in a second buffer area. The third output data obtained from the third computation on the second output data is outputted to the memory. When the second computation is performed on the first output data, the first computation is performed on the input data continuously. Accordingly, the accessing time on the main memory could be reduced.

Description

Computational circuit, data processing method and computer-readable storage medium based on convolutional neural network

The present invention relates to a machine learning (ML) technology, and in particular to an operation circuit, a data processing method and a computer-readable storage medium based on a convolutional neural network (CNN).

Machine learning is an important topic in artificial intelligence (AI). It can analyze training samples to obtain patterns and predict unknown data through patterns. The machine learning model constructed after learning is used to evaluate data inference.

There are many machine learning algorithms. For example, neural networks can make decisions by simulating the operation of human brain cells. Among them, convolutional neural networks provide better results in image and speech recognition, and have gradually become one of the most widely used and researched machine learning architectures.

It is worth noting that in the convolution layer of the convolutional neural network architecture, the processing element slides the convolution kernel or filter on the input data and performs specific operations. The processing element needs to repeatedly read the input data and weight values from the memory and output the operation results to the memory. Even if different convolution layers use convolution kernels of different sizes or different convolution operations, the number of memory accesses will be greatly increased. For example, the MobileNet model combines convolution operations and depthwise separable convolution operations. Therefore, these operations need to access the memory separately.

In view of this, the present invention provides an operation circuit, a data processing method and a computer-readable storage medium based on a convolutional neural network, which integrates multiple convolutional layers and reduces the number of memory accesses.

The data processing method based on the convolutional neural network of the embodiment of the present invention includes (but is not limited to) the following steps: reading input data from the memory. Performing a first operation on the first part of the input data to obtain first output data. The first operation is provided with a first filter, and the size of the first output data is related to the size of the first filter of the first operation and the size of the first part of the data. The first output data is temporarily stored in a first buffer area. When the first output data temporarily stored in the first buffer area is greater than the first predetermined data amount, performing a second operation on the first output data to obtain second output data. The second operation is provided with a second filter, and the size of the second output data is related to the size of the second filter of the second operation. The second output data is temporarily stored in a second buffer. The third output data obtained by performing the third operation on the second output data is output to the memory. When performing the second operation on the first output data, the first operation on the input data continues.

The convolution neural network-based computing circuit of the embodiment of the present invention includes (but is not limited to) a memory and a processing element. The memory is used to store input data. The processing element is coupled to the memory and includes a first, a second and a third operator, a first temporary memory and a second temporary memory. The first operator is used to perform a first operation on a first portion of the input data to obtain first output data, and temporarily store the first output data in the first temporary memory of the processing element. The size of the first output data is related to the size of the first filter of the first operation and the size of the first portion of the data. The second operator is used to perform a second operation on the second input data to obtain the second output data when the first output data temporarily stored in the first temporary memory meets the size required by the second operation, and temporarily store the second output data in the third memory of the processing element. The second operation is provided with a second filter, and the size of the second output data is related to the size of the second filter of the second operation. The third operator is used to output the third output data obtained by performing the third operation on the second output data to the memory. When the second operator performs the second operation, the first operator continues to perform the first operation.

The computer of the embodiment of the present invention can read the storage medium for storing program code, and the processor loads the program code to execute the aforementioned data processing method based on the convolutional neural network.

Based on the above, according to the convolutional neural network-based computing circuit, data processing method and computer-readable storage medium of the embodiment of the present invention, the output data is temporarily stored in the memory of the processing element, and the operation of the next operator (i.e., the next computing layer) is triggered according to the activation condition. In this way, the next computing layer can trigger the operation in advance without waiting for the previous computing layer to operate on all input data. In addition, the embodiment of the present invention can reduce the number of times the input data is accessed from the memory.

In order to make the above features and advantages of the present invention more clearly understood, the following is a detailed description of the embodiments with the accompanying drawings.

100: Operational circuit

110: Memory

120: Processing element

131: Feature temporary memory

151: First temporary memory

171: Second temporary memory

132: First FIFO unit

172: Second first-in-first-out unit

133: First weight buffer

153: Second weight buffer

173: Third weight buffer

135: First calculator

155: Second calculator

175: The third operator

S210~S290, S111~S113, S1501~S1560: Steps

F: Input data

H, H _fi1 , H _kd , H _f1o , H _a1o , H _a2o , H _a1i , H _a2i , H _a3o , H _s1o : high

W、W _fi1 、W _kd 、W _f1o 、W _a1o 、W _a2o 、W _a1i 、W _a2i 、W _a3o 、W _a4o 、W _a5o 、W _tfo1 、W _tfo2 、W _s1o 、W _so21 、W _so22 、W _so3 、W _to1 : width

C, C _fi1 , C _f1o , _Ca1o , _Ca2o , _Ca1i , _Ca2i , _Ca3o , C _tfo , C _s1o , C _so2 , C _to1 : Number of channels

F _fi1 , F _fi2 , F _fi3 , F _fi4 , F _fi6 : The first part of the data

_Kn , _Fd , _Fp : Filter

F _fo1 , F _fo2 , F _fo3 , F _fo4 , F _tfo : first output data

SA _1i , SA _2i : Pulse array input

SA _1o , SA _2o , SA _3o , SA _4o , SA _5o , SA _6o , SA _7o , SA _8o : Pulse array output

601, 703, 903, 141: Output completed

602, 704, 904, 142: Current processing output

D1, D2: Direction

701, 901: Input completed

702, 902: Currently processing input

i4: high position

j4: wide position

n4: Channel location

I(,,,), A(,,,): value

F _dn4 (,,):weight

F _si1 : The second part of the data

F _so1 , F _so2 , F _so3 , F _tso : Second output data

F _ti : The third part of the data

F _to : The third output data

FIG1 is a block diagram of components of an operation circuit based on a convolutional neural network according to an embodiment of the present invention.

Figure 2 is a flow chart of a data processing method based on a convolutional neural network according to an embodiment of the present invention.

Figure 3 is a schematic diagram of input data according to an embodiment of the present invention.

Figure 4 is a schematic diagram of the first operation according to an embodiment of the present invention.

Figures 5A and 5B are schematic diagrams of the pulse array input and output according to an embodiment of the present invention.

Figures 6A to 6C are schematic diagrams of the pulse array output according to an embodiment of the present invention.

FIG. 7A is a schematic diagram illustrating reading input data according to an embodiment of the present invention.

Figure 7B is a schematic diagram of the first output data according to an embodiment of the present invention.

FIG. 7C is a schematic diagram illustrating reading input data according to an embodiment of the present invention.

Figure 7D is a schematic diagram of the first output data according to an embodiment of the present invention.

FIG. 7E is a schematic diagram illustrating reading input data according to an embodiment of the present invention.

Figure 7F is a schematic diagram of the first output data according to an embodiment of the present invention.

FIG8 is a schematic diagram illustrating reading input data according to an embodiment of the present invention.

FIG9A is a schematic diagram illustrating the triggering conditions of the second operation according to an embodiment of the present invention.

FIG9B is a schematic diagram of the temporarily stored first output data according to an embodiment of the present invention.

Figure 10A is a schematic diagram of the second operation according to an embodiment of the present invention.

Figures 10B to 10D are schematic diagrams of the second output data according to an embodiment of the present invention.

Figure 11 is a flow chart of a data processing method based on a convolutional neural network according to an embodiment of the present invention.

FIG12A is a schematic diagram of the temporarily stored first output data according to an embodiment of the present invention.

FIG12B is a schematic diagram of the temporarily stored second output data according to an embodiment of the present invention.

Figure 13A is a schematic diagram of the third operation according to an embodiment of the present invention.

Figure 13B is a schematic diagram of the third output data according to an embodiment of the present invention.

Figures 14A to 14C are schematic diagrams of the pulse array output according to an embodiment of the present invention.

Figure 15 is a flow chart of a data processing method of a MobileNet architecture according to an embodiment of the present invention.

FIG1 is a block diagram of a convolutional neural network-based computing circuit 100 according to an embodiment of the present invention. Referring to FIG1 , the computing circuit 100 includes (but is not limited to) a memory 110 and one or more processing elements (PE) 120.

The memory 110 can be a dynamic random access memory (DRAM), a flash memory, a register, a combinatorial circuit, or a combination of the above elements.

The processing element 120 is coupled to the memory 110. The processing element 120 includes (but is not limited to) a feature temporary memory 131, a first input first output (FIFO) unit 132, a first weight temporary memory 133, a first operator 135, a first temporary memory 151, a second weight temporary memory 153, a second operator 155, a second temporary memory 171, a second first input first output unit 172, a third weight temporary memory 173 and a third operator 175.

In one embodiment, the feature temporary memory 131, the first-in-first-out unit 132, the first weight temporary memory 133, and the first operator 135 correspond to a convolution layer/operation layer. In addition, the first operator 135 is provided with a first filter used for the first operation.

In one embodiment, the feature temporary memory 131 is used to store part or all of the input data from the memory 110, the first first-in first-out unit 132 is used to input and/or output the data in the feature temporary memory 131 according to the first-in first-out rule, the first weight temporary memory 133 is used to store the weight from the memory 110 (forming the first convolution kernel/filter), and the first operator 135 is used to perform the first operation. In one embodiment, the first operation is a convolution operation, which will be described in detail in the subsequent embodiments. In another embodiment, the first operation may also be a depth-separable convolution operation or other types of convolution operations.

In one embodiment, the first temporary memory 151, the second weight temporary memory 153, and the second operator 155 correspond to a convolution layer/operation layer. In addition, the second operator 155 is provided with a second filter used for the second operation.

In one embodiment, the first temporary memory 151 is used to store part or all of the input data output from the first operator 135, the second weight temporary memory 153 is used to store the weights from the memory 110 (forming a second convolution kernel/filter), and the second operator 155 is used to perform a second operation. In one embodiment, the second operation is a depthwise convolution operation, which will be described in detail in the subsequent embodiments. In another embodiment, the second operation may also be a convolution operation or other types of convolution operations.

In one embodiment, the second temporary memory 171, the second first-in first-out unit 172, the third weight temporary memory 173 and the third operator 175 correspond to a convolution layer/operation layer. In addition, the third operator 175 is provided with a third filter used for the third operation.

In one embodiment, the second temporary memory 171 is used to store part or all of the input data output from the second operator 155, the second first-in first-out unit 172 is used to input and/or output the data in the second temporary memory 171 according to the first-in first-out rule, the third weight temporary memory 173 is used to store the weights from the memory 110 (forming a third convolution kernel/filter), and the third operator 175 is used to perform a third operation. In one embodiment, the third operation is a pointwise convolution operation, which will be described in detail in the subsequent embodiments. In another embodiment, the third operation may also be a convolution operation or other types of convolution operations.

In one embodiment, the aforementioned feature temporary memory 131, the first temporary memory, the second temporary memory, the first weight temporary memory 133, the second weight temporary memory 153 and the third weight temporary memory 173 can be static random access memory (SRAM), flash memory, register, various types of buffers or a combination of the above elements.

In one embodiment, some or all of the components in the computing circuit 100 may form a neural network processing unit (Network Processing Unit, NPU), a system on chip (System on Chip, SoC) or an integrated circuit (Integrated Circuit, IC).

In one embodiment, the first operator 135 has a first maximum amount of operations per unit time, the second operator 155 has a second maximum amount of operations per unit time, and the third operator 175 has a third maximum amount of operations per unit time. The first maximum amount of operations is greater than the second maximum amount of operations, and the first maximum amount of operations is greater than the third maximum amount of operations.

In the following, the method described in the embodiment of the present invention will be described with various devices, components and modules in the computing circuit 100. The various processes of the method can be adjusted according to the implementation situation, but are not limited to this.

FIG2 is a flow chart of a data processing method based on a convolutional neural network according to an embodiment of the present invention. Referring to FIG2, the processing element 120 reads the input data from the memory 110 (step S210). Specifically, the input data may be data of part or all pixels in the image (e.g., color level, brightness, or gray level). Alternatively, the input data may also be a data set formed by speech, text, pattern or other forms.

There are many ways to read input data. In one embodiment, the processing element 120 reads all the input data and uses it as the first part of the data. In another embodiment, the processing element 120 reads a portion of the input data each time according to the amount of data required for the first operation or the capacity of the feature temporary memory 131 and uses it as the first part of the data.

FIG3 is a schematic diagram of input data F according to an embodiment of the present invention. Referring to FIG3 , it is assumed that the size of the input data F is (height H, width W, number of channels C), and the size of the first portion of data F _fi1 read by the processing element 120 is (height H _fi1 , width W _fi1 , number of channels C). The height H _fi1 may be less than or equal to the height H, and the width W _fi1 may be less than or equal to the width W.

It should be noted that the input data may be stored in a specific block or location in the memory 110, but the embodiment of the present invention does not limit the storage location of each element in the input data in the memory 110.

The feature temporary memory 131 stores part or all of the input data from the memory 110. That is, the feature temporary memory 131 stores the first part of the data. The first operator 135 performs a first operation on the first part of the input data to obtain the first output data (step S230). Specifically, the first operation is to perform a first operation (for example, a convolution operation) on the first part of the data and the corresponding weight. The size of the first output data is related to the size of the first filter of the first operation and the size of the first part of the data.

For example, FIG4 is a schematic diagram of a first operation according to an embodiment of the present invention. Referring to FIG4 , the first operation is a convolution operation as an example. The first part of data F _fi1 (size is height H _fi1 , width W _fi1 , number of channels C _fi1 ) and the first filter K _n (size is height H _kd , width W _kd ). If the height H _fi1 is greater than or equal to the height H _kd and the width W _fi1 is greater than or equal to the width W _kd , the first operator 135 can trigger the first operation. The result of the first operation (i.e., the first output data F _fo1 ) has a height H _f1o of height H _fi1 -H _kd +1, a width W _f1o of width W _fi1 -W _kd +1, and the number of channels C _f1o is the same as the number of channels C _fi1 .

For another example, if the size (height, width, number of channels) of the first part of the data F _fi1 is (3, 32, 16), and the size (height, width) of the first filter K _n is (3, 3), then the size (height, width, number of channels) of the first output data F _fo1 is (1, 30, 16).

In one embodiment, the first operator 135 adopts a systolic array structure. The first operator 135 divides the first part of the data into multiple first systolic array inputs, and performs a first operation on these first systolic array inputs respectively to obtain multiple first systolic array outputs. The size of each first systolic array output will be limited by the size of the systolic array. For example, the number of elements of the first systolic array output is less than or equal to the capacity of the systolic array. In addition, these first systolic array outputs based on the same first part of the data constitute the first output data.

For example, FIG. 5A and FIG. 5B are schematic diagrams of a pulse array input and output according to an embodiment of the present invention. Referring to FIG. 5A and FIG. 5B , it is assumed that the size of the pulse array is (number M _sa × number N _sa ). The height _Ha1o of the pulse array output _SA1o is 1, its width W _a1o may be the number M _sa , and its number of channels _Ca1o may be the number N _sa . Therefore, the first operator 135 divides the first portion of data into a pulse array input _SA1i (whose size is height _Ha1i , width W _a1i , number of channels _Ca1i ) and a pulse array input _SA2i (whose size is height _Ha2i , width W _a2i , number of channels _Ca2i ). The two pulse array inputs SA _1i , SA _2i are convolved with the weights of each channel of the filter K _n , and the pulse array outputs SA _1o , SA _2o are obtained accordingly. The height _Ha2o of the pulse array output SA _2o is 1, its width W _a2o can be less than or equal to the number M _sa , and its number of channels C _a2o can be less than or equal to the number N _sa .

For another example, the size (height, width, number of channels) of the first portion of data is (3, 32, 16), the size of the pulse array is 16×16, and the size of the filter _Kn is 3×3. The height _Ha1o of the pulse array output _SA1o is 1, its width W _a1o can be 16, and its number of channels _Ca1o can be 16. In addition, the height _Ha1i of the pulse array input _SA1i is 3, its width W _a1i is 18, and its number of channels _Ca1i is 16. On the other hand, after the first operator 135 distinguishes the pulse array input _SA1i from the first portion of data, it can obtain the pulse array input _SA2i . The height _Ha2i of the pulse array input _SA2i is 3, its width _Wa2i is 16, and its channel number _Ca2i is 16. In addition, the height _Ha2o of the pulse array output _SA2o is 1, its width _Wa2o is 14 (i.e., width _Wa2i - width of filter _Kn + 1), and its channel number _Ca2o is 16.

For another example, Table (1) to Table (3) are data of the first, second and fifteenth first part data stored in the feature temporary memory 131 (the remaining data can be deduced in the same way):

I(i1,j1,n1) represents the value of the input data read at the position (high position i1, wide position j1, channel position n1). The first FIFO unit 132 sequentially inputs those data to the first operator 135 from right to left and from top to bottom.

Table (4) shows the data of the 16-channel 3×3 filter used in the convolution operation:

F _dn (i2, j2, n2) represents the value read by the nth filter at position (high position i2, wide position j2, channel position n2).

Table (5) shows the pulse array output:

A(i3,j3,n3) represents the value of the pulse array output at the position (high position i3, wide position j3, channel position n3), and its mathematical expression is: A(i3,j3,n3)=I(i3,j3,0)×F _dn3 (0,0,0)+I(i3,j3,1)×F _dn3 (0,0,1)+…+I(i3,j3,15)×F _dn3 (0,0,15)+I(i3,j3+1,0)×F _dn3 (0,1,0)+I(i3,j3+1,1)×F _dn3 (0,1,1)+…+I(i3,j3+1,15)×F _dn3 (0,1,15)+I(i3,j3+2,0)×F _dn3 (0,2,0)+I(i3,j3+2,1)×F _dn3 (0,2,1)+…+I(i3,j3+2,15)×F _dn3 (0,2,15)+I(i3+1,j3,0)×F _dn3 (1,0,0)+I(i3+1,j3,1)×F _dn3 (1,0,1)+…+I(i3+1,j3,15)×F _dn3 (1,0,15)+I(i3+1,j3+1,0)×F _dn3 (1,1,0)+I(i3+1,j3+1,1)×F _dn3 (1,1,1)+…+I(i3+1,j3+1,15)×F _dn3 (1,1,15)+I(i3+1,j3+2,0)×F +I(i3+2,j3+1,0)×F _dn3 (2,0,0)+I(i3+2,j3,1)×F _dn3 (2,0,1)+…+I(i3+2,j3,15 _{)×F dn3 (2,0,15) +I(i3+2,j3+1,0)×F dn3 ( 2,1,0} )+I(i3+2,j3+1,1)×F _{dn3 (} 2,1,1)+… ₊ I(i3+2, _j3 +1,15)×F (2,1,15)+I(i3+2,j3+2,0)× _Fdn3 (2,2,0)+I(i3+2,j3+2,1)× _Fdn3 (2,2,1)+…+I(i3+2,j3+2,15)× _Fdn3 (2,2,15)…(1).

0，代表寬的j3

0~15且代表過濾器輸出通道的n3

0~15。 FIG6A is a schematic diagram of a pulse array output SA _3o according to an embodiment of the present invention. Referring to FIG6A , the pulse array output SA _3o has a height H _a3o of 1, a width W _a3o of 16, and a channel number C _a3o of 16. In the mathematical expression (1), the height i3

0, represents wide j3

0~15 and represents the filter output channel n3

0~15.

0，代表寬的j3

16~29且代表過濾器輸出通道的n3

0~15。此外，已完成輸出601為圖6A的脈動陣列輸出SA_3o，且當前處理輸出602為脈動陣列輸出SA_4o。 FIG6B is a schematic diagram of a pulse array output SA _4o according to an embodiment of the present invention. Referring to FIG6B , the pulse array output SA _4o has a height H _a3o of 1, a width W _a4o of 14, and a channel number C _a3o of 16. In the mathematical expression (1), the height i3

0, represents wide j3

16~29 and represents the filter output channel n3

0~15. In addition, the completed output 601 is the pulse array output SA _3o of FIG. 6A , and the currently processed output 602 is the pulse array output SA _4o .

4，代表寬的j3

16~29且代表過濾器輸出通道的n3

0~15。此外，當前處理輸出602為脈動陣列輸出SA_5o。這些脈動陣列輸出SA_3o~SA_5o可形成一筆或更多筆第一輸出資料。 Similarly, FIG. 6C is a schematic diagram of a pulse array output SA _4o according to an embodiment of the present invention. Referring to FIG. 6C , the height H _a3o of the pulse array output SA _5o is 1, its width W _a5o is 14, and its number of channels C _a3o is 16. In the mathematical expression (1), the height i3

4, represents wide j3

16~29 and represents the filter output channel n3

0~15. In addition, the current processing output 602 is the pulse array output SA _5o . These pulse array outputs SA _3o ~SA _5o can form one or more first output data.

In one embodiment, the first operation is a convolution operation, and the first operator 135 reads the first portion of the input data stored in the memory 110 in the first sliding direction. The first operator 135 divides the input data into multiple segments, and continues to read the next segment in a first sliding direction parallel to the height of the input data, and uses it as the first portion of the data.

For example, FIG7A is a schematic diagram illustrating reading input data F _fi1 , F _fi2 according to an embodiment of the present invention. Referring to FIG7A , if the first operation of the first portion of data F _fi1 has been completed, the first operator 135 regards the first portion of data F _fi1 as having completed input 701, and further reads the first portion of data F _fi2 of the next segment in the input data F in the direction D1 (e.g., below the figure) and uses it as the current processing input 702.

FIG7B is a schematic diagram of the first output data F _fo1 , F _fo2 according to an embodiment of the present invention. Referring to FIG7B , the first output data F _fo1 is the output of the convolution operation performed on the first part of the data F _fi1 of FIG7A and is used as the completed output 703. In addition, the first output data F _fo2 is the output of the convolution operation performed on the first part of the data F _fi2 of FIG7A and is used as the current processing output 704. The first output data F _fo2 is also arranged below the first output data F _fo1 according to the direction D1 of FIG7A.

FIG7C is a schematic diagram illustrating the reading of input data F _fi3 according to an embodiment of the present invention. Referring to FIG7C , if the input 701 has been completed and has reached the bottom of the input data F, the first operator 135 reads the first part of the next segment of the input data F _fi3 in the direction D2 (e.g., the right side of the figure) from top to bottom (corresponding to the direction D1 of FIG7A ) and uses it as the current processing input 702.

FIG7D is a schematic diagram of the first output data F _fo3 according to an embodiment of the present invention. Referring to FIG7D , the first output data F _fo3 is the output of the convolution operation on the first part of the data F _fi3 in FIG7C and serves as the current processing output 704. Similarly, the current processing output 704 is arranged to the right of the completed output 703.

FIG7E is a schematic diagram illustrating reading input data _Ffi4 according to an embodiment of the present invention. Referring to FIG7E, the first portion of data _Ffi4 currently processed as input 702 is the last segment of the input data.

FIG7F is a schematic diagram of the first output data F _fo4 according to an embodiment of the present invention. Referring to FIG7F , the first output data F _fo4 is the output of the convolution operation on the first part of the data F _fi4 in FIG7E and serves as the current processing output 704. Similarly, the current processing output 704 is arranged below the completed output 703 and is used to complete the convolution operation on the input data F.

In another embodiment, the first operator 135 reads the first portion of the input data stored in the memory 110 in a second sliding direction (different from the first sliding direction). Similarly, the first operator 135 divides the input data into multiple segments, and continues to read the next segment in a second sliding direction parallel to the width of the input data, and uses it as the first portion of the data.

For example, FIG8 is a schematic diagram illustrating reading input data according to an embodiment of the present invention. Referring to FIG8 , if the first operation of the first partial data F _fi1 has been completed, the first operator 135 regards the first partial data F _fi1 as having completed input 701, and further reads the first partial data F _fi6 of the next segment in the input data F in the direction D2 (for example, the right side of the figure) and uses it as the current processing input 702. Similarly, if the last segment of the same row has been read in the direction D2, the first operator 135 will read the segment below the first partial data F _fi1 . In addition, the arrangement of the first partial data F _fi1 and other first partial data (not shown) can refer to the above description, and will not be repeated here.

Referring to FIG. 2 , the first operator 135 temporarily stores one or more first output data in the first temporary storage area of the first temporary storage memory 151 (step S250). Specifically, unlike the prior art that outputs the first output data to the memory 110, the first output data of the embodiment of the present invention is output to the temporary first temporary storage memory 151 of the second operator 155, thereby reducing the number of accesses to the memory 110.

When the first output data temporarily stored in the first temporary storage memory 151 (or the first temporary storage area) is greater than the first predetermined data amount, the second operator 155 performs a second operation on the first output data to obtain second output data (step S270). Specifically, in the existing multi-convolution layer architecture, the next convolution layer needs to wait until the previous convolution layer operates on all its input data and outputs it to the main memory before reading the input data output by the previous convolution layer from the main memory. Different from the prior art, in addition to temporarily storing in a storage medium other than the memory 110 (e.g., the first temporary storage memory 151 or the second temporary storage memory 171), the embodiment of the present invention can trigger the convolution operation of the next convolution layer whenever the size of the input data required by the next convolution layer (i.e., the first predetermined data amount) is met. At the same time, if the operation of the previous convolution layer on all input data has not been completed, the operations of the two convolution layers can be performed simultaneously. In other words, when the second operator 155 performs the second operation on the first output data, the first operator 135 continues to perform the first operation on the input data.

It is worth noting that the second part of data input by the second operation includes the first output data temporarily stored in the first temporary memory 151, and the size of the second output data is related to the size of the second filter of the second operation. Assume that the second operation is a depthwise convolution operation. Each filter of the depthwise convolution operation only corresponds to the data of one channel in the second part of data. That is, any filter of the depthwise convolution operation only performs convolution operation on the data of one channel. Therefore, the number of filters of the depthwise convolution operation is usually equal to the number of channels of the second part of data. However, each filter of the convolution operation performs convolution operation on the data of all channels. In addition, as long as the height of the temporarily stored first output data increases to the height of the filter and the width of the first output data increases to the width of the filter, the filter can perform depth-wise convolution operations with these temporarily stored first output data (as the second part of data).

In one embodiment, it is assumed that the height of each filter used in the depthwise convolution operation is H _kd and the width of the filter is W _kd . The height of the first output data of each segment is H _f1o and the width of the first output data is W _f1o . When the first output data temporarily stored in the first temporary storage memory 151 or the first temporary storage area is greater than W _kd ×H _kd , the second operator 155 may perform the second operation. When the first output data temporarily stored in the first temporary storage memory 151 or the first temporary storage area is greater than the first predetermined data amount, the height of the first output data temporarily stored in the first temporary storage memory 151 or the first temporary storage area is M _H ×H _f1o and the width is M _W ×W _f1o . _MH and _MW are multiples and positive integers, _MH ×H _f1o is not less than H _kd , and _MW ×W _f1o is not less than W _kd . In other words, when the high _MH ×H _f1o of the first output data stored in advance is less than the high H _kd of the filter and the width _MW ×W _f1o of the first output data stored in advance is less than the width W _kd of the filter, the second operator 155 will continue to wait for the next first output data or pulse array output until the high _MH ×H _f1o of the first output data stored in advance is greater than or equal to the high H _kd of the filter and the width _MW ×W _f1o of the first output data stored in advance is greater than or equal to the width W _kd of the filter.

For example, FIG. 9A is a schematic diagram illustrating a triggering condition of the second operation according to an embodiment of the present invention, and FIG. 9B is a schematic diagram of the temporarily stored first output data according to an embodiment of the present invention. Referring to FIG. 9A , the completed input 901 in the input data corresponds to the completed output 903 of the first output data. If the size of the current processing output 904 corresponding to the current processing input 902 and the completed output 903 meets the size required by the second operation, the second operation can be triggered.

Please refer to FIG9B , assuming that the completed output 903 of FIG9A and the currently processed output 904 form the temporarily stored first output data F _tfo . The size of the pulse array used by the first operator 135 is 16×16, wherein the width W _tfo1 of the pulse array output can be 16 or the width W _tfo2 can be 14. Assume that the height of each filter used in the depthwise convolution operation is 3 and the width of the filter is 3. The widths W _tfo1 and W _tfo2 are both greater than 3. If the fifth pulse array output is temporarily stored in the first temporary memory 151, the size (height, width, number of channels) of the first to fifth pulse array outputs is (1, 16, 16) or (1, 14, 16), that is, the number of channels C _tfo is 16) and the size formed by the output has satisfied the size of 3×3. That is, the pulse array output with a height of 1 is stacked in three layers, so that the height after stacking is 3. At this time, these pulse array outputs can be used as the second part of data and can be used for the second operation.

It should be noted that FIG. 9A and FIG. 9B trigger the second operation as long as the number of stacked layers is equal to the height of the filter. However, in other embodiments, the number of stacked layers may be greater than the height of the filter.

FIG10A is a schematic diagram of the second operation according to an embodiment of the present invention for depth-wise convolution operation. Referring to FIG10A , it is assumed that the size (height, width, number of channels) of the second part data F _si1 is (5, 30, 16), and the size of the filter F _d used in the depth-wise convolution operation is 3×3. I(i4, j4, n4) represents the value of the second part data at the position (height position i4, width position j4, channel position n4). F _dn4 (i5, j5, n5) represents the value of the read n4th filter at the position (height position i5, width position j5). A(i4,j4,n4) represents the value of the second output data or pulse array output at the position (high position i4, wide position j4, channel position n4), and its mathematical expression is: A(i4,j4,n4)=I(i4,j4,n4)×F _dn4 (0,0)+I(i4,j4+1,n4)×F _dn4 (0,1)+I(i4,j4+2,n4)×F _dn4 (0,2)+I(i4+1,j4,n4)×F _dn4 (1,0)+I(i4+1,j4+1,n4)×F _dn4 (1,1)+I(i4+1,j4+2,n4)×F _dn4 (1,2)+I(i4+2,j4,n)×F _dn4 (2,0)+I(i4+2,j4+1,n4)× _Fdn4 (2,1)+I(i4+2,j4+2,n4)× _Fdn4 (2,2)…(2).

FIG10B is a schematic diagram of the second output data F _so1 according to an embodiment of the present invention. Referring to FIG10B , it is assumed that the size (height H _so1 , width W _so1 , number of channels C _so1 ) of the second output data F _so1 currently being processed is (1, 28, 16). The values in the second output data F _so1 are: A(0,0,n4)=I(0,0,n4)×F _dn4 (0,0)+I(0,1,n4)×F _dn4 (0,1)+I(0,2,n4)×F _dn4 (0,2)+I(1,0,n4)×F _dn4 (1,0)+I(1,1,n4)×F _dn4 (1,1)+I(1,2,n4)×F _dn4 (1,2)+I(2,0,n)×F _dn4 (2,0)+I(2,1,n4)×F _dn4 (2,1)+I(2,2,n4)×F _dn4 (2,2)...(3)

A(0,1,n4)=I(0,1,n4)×F _dn4 (0,0)+I(0,2,n4)×F _dn4 (0,1)+I(0,3,n4)×F _dn4 (0,2)+I(1,1,n4)×F _dn4 (1,0)+I(1,2,n4)×F _dn4 (1,1)+ I(1,3,n4)×F _dn4 (1,2)+I(2,1,n)×F _dn4 (2,0)+I(2,2,n4)×F _dn4 (2,1)+I(2,3,n4)×F _dn4 (2,2)…(4)

A(0,27,n4)=I(0,27,n4)×F _dn4 (0,0)+I(0,28,n4)×F _dn4 (0,1)+I(0,29,n4)×F _dn4 (0,2)+I(1,27,n4)×F _dn4 (1,0)+I(1,28,n4)×F _dn4 (1,1)+I(1,29,n4)×F _dn4 (1,2)+I(2,27,n)×F _dn4 (2,0)+I(2,28,n4)×F _dn4 (2,1)+I(2,29,n4)×F _dn4 (2,2)…(5) The rest can be deduced from this formula and will not be elaborated here.

FIG10C is a schematic diagram of the second output data F _so2 according to an embodiment of the present invention. Referring to FIG10C , the completed output 101 is the second output data F _so1 of FIG10B . The second output data F _so2 is the current processing output 102 , and its size may be the same as the second output data F _so1 of FIG10B . The values in the second output data F _so2 are: A(1,0,n4)=I(1,0,n4)×F _dn4 (0,0)+I(1,1,n4)×F _dn4 (0,1)+I(1,2,n4)×F _dn4 (0,2)+I(2,0,n4)×F _dn4 (1,0)+I(2,1,n4)×F _dn4 (1,1)+I(2,2,n4)×F _dn4 (1,2)+I(3,0,n)×F _dn4 (2,0)+I(3,1,n4)×F _dn4 (2,1)+I(3,2,n4)×F _dn4 (2,2)...(6)

A(1,1,n4)=I(1,1,n4)×F _dn4 (0,0)+I(1,2,n4)×F _dn4 (0,1)+I(1,3,n4)×F _dn4 (0,2)+I(2,1,n4)×F _dn4 (1,0)+I(2,2,n4)×F _dn4 (1,1)+I(2,3,n4)×F _dn4 (1,2)+I(3,1,n)×F _dn4 (2,0)+I(3,2,n4)×F _dn4 (2,1)+I(3,3,n4)×F _dn4 (2,2)…(7)

A(1,27,n4)=I(1,27,n4)×F _dn4 (0,0)+I(1,28,n4)×F _dn4 (0,1)+I(1,29,n4)×F _dn4 (0,2)+I(2,27,n4)×F _dn4 (1,0)+I(2,28,n4)×F _dn4 (1,1)+I(2,29,n4)×F _dn4 (1,2)+I(3,27,n)×F _dn4 (2,0)+I(3,28,n4)×F _dn4 (2,1)+I(3,29,n4)×F _dn4 (2,2)…(8)The rest can be deduced from this formula and will not be elaborated here.

FIG10D is a schematic diagram of the second output data F _so3 according to an embodiment of the present invention. Referring to FIG10C , the second output data F _so3 is the current processing output 102 , and its size may be the same as the second output data F _so1 of FIG10B . The values in the second output data F _so3 are: A(2,0,n4)=I(2,0,n4)×F _dn4 (0,0)+I(2,1,n4)×F _dn4 (0,1)+I(2,2,n4)×F _dn4 (0,2)+I(3,0,n4)×F _dn4 (1,0)+I(3,1,n4)×F _dn4 (1,1)+I(3,2,n4)×F _dn4 (1,2)+I(4,0,n)×F _dn4 (2,0)+I(4,1,n4)×F _dn4 (2,1)+I(4,2,n4)×F _dn4 (2,2)...(9)

A(2,1,n4)=I(2,1,n4)×F _dn4 (0,0)+I(2,2,n4)×F _dn4 (0,1)+I(2,3,n4)×F _dn4 (0,2)+I(3,1,n4)×F _dn4 (1,0)+I(3,2,n4)×F _dn4 (1,1)+I(3,3,n4)×F _dn4 (1,2)+I(4,1,n)×F _dn4 (2,0)+I(4,2,n4)×F _dn4 (2,1)+I(4,3,n4)×F _dn4 (2,2)…(10)

A(2,27,n4)=I(2,27,n4)×F _dn4 (0,0)+I(2,28,n4)×F _dn4 (0,1)+I(2,29,n4)×F _dn4 (0,2)+I(3,27,n4)×F _dn4 (1,0)+I(3,28,n4)×F _dn4 (1,1)+I(3,29,n4)×F _dn4 (1,2)+I(4,27,n)×F _dn4 (2,0)+I(4,28,n4)×F _dn4 (2,1)+I(4,29,n4)×F _dn4 (2,2)…(11)The rest can be deduced from this formula and will not be elaborated here.

In one embodiment, the second operator 155 adopts a pulse array structure. The second operator 155 divides the second part of data (i.e., part of the temporarily stored first output data) into a plurality of second pulse array inputs, and performs a second operation on these second pulse array inputs respectively to obtain a plurality of second pulse array outputs. The size of each second pulse array output will be limited by the size of the pulse array. For example, the number of elements of the second pulse array output is less than or equal to the capacity of the pulse array. In addition, these second pulse array outputs based on the same second part of data constitute the second output data. Taking FIG. 10B as an example, if the size of the pulse array is 16×16, the second output data F _so1 includes second pulse array outputs of 1×16×16 and 1×12×16.

For the next convolution layer, FIG11 is a flow chart of a data processing method based on a convolution neural network according to an embodiment of the present invention. Referring to FIG11, in one embodiment, the second operator 155 may temporarily store one or more second output data in a second temporary storage area of the second temporary storage memory 171 (step S111) (step S280). Specifically, similarly, the embodiment of the present invention temporarily stores the output of the previous convolution layer in the buffer of the next convolution layer instead of directly outputting the output data to the memory 110.

When the second output data temporarily stored in the first temporary storage memory 171 or the second temporary storage area is greater than the second predetermined data amount, the third operator 175 can perform a third operation on the second output data to obtain the third output data (step S113). Specifically, the third part of the data input by the third operation includes the second output data temporarily stored in the second temporary storage memory 171, and the size of the third part of the data is related to the filter size of the third operation. Assume that the third operation is a pointwise convolution operation. The size of each filter of the pointwise convolution operation is only 1×1. Similar to the convolution operation, each filter of the pointwise convolution operation also performs a convolution operation on the data of all channels. In addition, as long as the height of the temporarily stored second output data increases to the height of the filter (1) and the width of the second output data increases to the width of the filter (1), the filter can perform depth-wise convolution operations with these temporarily stored second output data (as the third part of data).

In one embodiment, as shown in FIG. 10B to FIG. 10D , each second output data can satisfy the size required for the point-by-point convolution operation. Therefore, the second first-in first-out unit 172 can sequentially input each second output data to the third operator 175. The third operator 175 can perform the third operation on each temporarily stored second output data.

For example, FIG. 12A is a schematic diagram of the first temporarily stored output data F _tfo according to an embodiment of the present invention, and FIG. 12B is a schematic diagram of the second temporarily stored output data F _tso according to an embodiment of the present invention. Referring to FIG. 12A and FIG. 12B, if the second operator 155 has completed the convolution operation on a portion of the temporarily stored multiple first output data, the first temporary storage memory 151 can temporarily store the second pulse array output of size (height, width, number of channels) (1, W _so21 , C _so2 ) or the second output data of (1, W _so21+ W _so21 , C _so2 ), and accordingly become the temporarily stored second output data F _tso . Wherein, the number of channels C _so2 is the same as the number of channels C _tf2 .

FIG13A is a schematic diagram of a third operation according to an embodiment of the present invention. Referring to FIG13A, the third operator 175 uses the second temporarily stored output data _Ftso of FIG12B as the third partial data _Fti (whose width _Wso3 is _{Wso21+ Wso21} ), and performs a third operation on the third partial data _Fti and the filter _Fp used for the point-by-point convolution operation.

FIG13B is a schematic diagram of the third output data F _to according to an embodiment of the present invention. Referring to FIG13A and FIG13B , the third output data F _to is equal to the third partial data F _ti . That is, the width W _to1 is equal to the width W _so3 , and the number of channels C _to1 is equal to the number of channels C _so2 .

In one embodiment, the third operator 175 adopts a pulse array structure. The third operator 175 divides the third portion of data into a plurality of third pulse array inputs, and performs a third operation on these third pulse array inputs respectively to obtain a plurality of third pulse array outputs. The size of each third pulse array output will be limited by the size of the pulse array. For example, the number of elements of the third pulse array output is less than or equal to the capacity of the pulse array. In addition, these third pulse array outputs based on the same third portion of data (i.e., part of the temporarily stored second output data) constitute the third output data. For example, if the size of the third part of the data is 1×28×16 and the size of the pulse array is 16×16, the third output data includes the third pulse array output of 1×16×16 and 1×12×16.

For example, FIG14A is a schematic diagram of a pulse array output SA _6o according to an embodiment of the present invention. Referring to FIG14A, Table (6) is the data of the second output data stored in the second temporary memory 171:

I(i6,j6,n6) represents the value of the input data read at the position (high position i6, wide position j6, channel position n6). The second FIFO unit 172 sequentially inputs those data from right to left and from top to bottom to the third operator 175.

Table (7) shows the data of the 16-channel 1×1 filter used in the point-by-point convolution operation:

F _dn (i7, j7, n7) represents the value read from the nth filter at position (high position i7, wide position j7, channel position n7).

Table (9) shows the pulse array output: Table (9)

A(i6,j6,n6) represents the value of the pulse array output at the position (high position i6, wide position j6, channel position n6), and its mathematical expression is: A(i6,j3,n6)=I(i6,j6,0)×F _dn6 (0,0,0)+I(i6,j6,1)×F _dn6 (0,0,1)+…+I(i6,j6,15)×F _dn6 (0,0,15)...(12).

0~15)：A(0,0,n6)=I(0,0,0)×F_dn6(0,0,0)+I(0,0,1)×F_dn6(0,0,1)+…+I(0,0,15)×F_dn6(0,0,15)...(13)；A(0,1,n6)=I(0,1,0)×F_dn6(0,0,0)+I(0,1,1)×F_dn6(0,0,1)+…+I(0,1,15)×F_dn6(0,0,15)...(14)。 Therefore, the values of the pulse array output SA _6o are ( n 6

0~15): A(0,0,n6)=I(0,0,0)× _Fdn6 (0,0,0)+I(0,0,1)× _Fdn6 (0,0,1)+…+I(0,0,15)× _Fdn6 (0,0,15)...(13); A(0,1,n6)=I(0,1,0)× _Fdn6 (0,0,0)+I(0,1,1)× _Fdn6 (0,0,1)+…+I(0,1,15)× _Fdn6 (0,0,15)...(14).

A(0,15,n6)=I(0,15,0)×F _dn6 (0,0,0)+I(0,15,1)×F _dn6 (0,0,1)+…+I(0,15,15)×F _dn6 (0,0,15)…(15), and the rest can be deduced in the same way and will not be elaborated on.

For another example, FIG. 14B is a schematic diagram of a pulse array output SA _7o according to an embodiment of the present invention. Referring to FIG. 14A , Table (10) is the data of the second output data stored in the second temporary memory 171:

Table (11) shows the pulse array output:

0~15)：A(0,16,n6)=I(0,16,0)×F_dn6(0,0,0)+I(0,16,1)×F_dn6(0,0,1)+…+I(0,16,15)×F_dn6(0,0,15)...(16)；A(0,17,n6)=I(0,17,0)×F_dn6(0,0,0)+I(0,17,1)×F_dn6(0,0,1)+…+I(0,17,15)×F_dn6(0,0,15)...(17)。 Therefore, the values of the pulse array output SA _7o are ( n 6

(0~15): A(0,16,n6)=I(0,16,0)× _Fdn6 (0,0,0)+I(0,16,1)× _Fdn6 (0,0,1)+…+I(0,16,15)× _Fdn6 (0,0,15)…(16); A(0,17,n6)=I(0,17,0)× _Fdn6 (0,0,0)+I(0,17,1)× _Fdn6 (0,0,1)+…+I(0,17,15)× _Fdn6 (0,0,15)…(17).

A(0,27,n6)=I(0,27,0)×F _dn6 (0,0,0)+I(0,27,1)×F _dn6 (0,0,1)+…+I(0,27,15)×F _dn6 (0,0,15)…(18), and the rest is similar and will not be repeated. In addition, the pulse array output SA _6o of FIG. 14A is the completed output 141, and the pulse array output SA _7o is the current processing output 142.

For another example, FIG14C is a schematic diagram of a pulse array output SA _8o according to an embodiment of the present invention. Referring to FIG14A, Table (12) is the data of the second output data stored in the second temporary memory 171:

Table (13) shows the pulse array output:

0~15)：A(2,16,n6)=I(2,16,0)×F_dn6(0,0,0)+I(2,16,1)×F_dn6(0,0,1)+…+I(2,16,15)×F_dn6(0,0,15)...(19)；A(2,17,n6)=I(2,17,0)×F_dn6(0,0,0)+I(2,17,1)×F_dn6(0,0,1)+…+I(2,17,15)×F_dn6(0,0,15)...(20)。 Therefore, the values of the pulse array output SA _8o of the last current processing output 142 are ( n 6

(0~15): A(2,16,n6)=I(2,16,0)× _Fdn6 (0,0,0)+I(2,16,1)× _Fdn6 (0,0,1)+…+I(2,16,15)× _Fdn6 (0,0,15)...(19); A(2,17,n6)=I(2,17,0)× _Fdn6 (0,0,0)+I(2,17,1)× _Fdn6 (0,0,1)+…+I(2,17,15)× _Fdn6 (0,0,15)...(20).

A(2,27,n6)= I(2,27,0)×F _dn6 (0,0,0)+I(2,27,1)×F _dn6 (0,0,1)+…+I(2,27,15)×F _dn6 (0,0,15)…(21), and the rest can be deduced in the same way and will not be elaborated on here.

In one embodiment, when the third operator 175 performs the third operation, the first operator 135 and the second operator 155 continue to perform the first operation and the second operation respectively. In other words, if the first operator 135 and the second operator 155 have not completed the operation of all input data, the operations of the first operator 135, the second operator 155 and the third operator 175 can be performed together.

Please refer to Figure 2. Finally, the third operator 175 outputs the third output data obtained by the third operation to the memory 110 (step S290).

In order to facilitate understanding of the complete process, another embodiment is given below for illustration. Figure 15 is a flow chart of a data processing method of a MobileNet architecture according to an embodiment of the present invention. Referring to Figure 15, the first operator 135 reads the width data of a defined segment from the input data in the memory 110 to make the first part of the data (step S1501). The first operator 135 determines whether the number of items currently being processed is greater than or equal to (the size of the first filter - 1) and whether the remainder obtained by dividing this number by the first stride used in the first operation is 1 (step S1503). If the conditions of step S1503 are met, the first first-in-first-out unit 132 outputs the first part of the data to the first operator 135 in sequence (step S1505). The first operator 135 reads the weights in the first filter used for the convolution operation from the memory 110 (step S1511), performs the convolution operation (step S1513), and outputs the obtained first output data to the first temporary memory 151 (step S1515). The first operator 135 determines whether all the data of the current strip (whose size is the same as the size of the filter) in the current segment have been convolution-operated (step S1517). If the data has not completed the convolution operation, the first FIFO unit 132 continues to output the first part of the data to the first operator 135 (step S1505). If all the data of this piece have been convolved, the first operator 135 determines whether all the data of all the pieces in the current segment have been convolved (step S1518). If one or more pieces of data in the current segment have not completed the convolution operation or the condition of step S1503 is met, the first operator 135 continues to process the next piece of data from the input data in the memory 110 (step S1507). If all the data of each piece in the current segment have been convolved, the first operator 135 determines whether all the data of all the segments have been convolved (step S1519). If there are still segments of data that have not completed the convolution operation, the first operator 135 continues to process the data of the next segment (step S1509). In addition, the first operator 135 resets the number of currently processed items to zero and sets the width of the current processing to: original width + segment width - (first stride used in the first operation - 1 + second stride used in the second operation - 1). If all data in all segments have been convolutionally operated, the first operator 135 completes all convolution operations on the input data (step S1520).

The second operator 155 determines whether the first output data temporarily stored in the first temporary memory 151 is greater than the first predetermined data amount (step S1521). For example, the size of the second filter is 3×3, and the second operator 155 determines whether there are three temporarily stored first output data. The second operator 155 determines whether the remainder obtained by the number of processed items in the first operation and the second stride used in the second operation is equal to zero (step S1523). If the remainder is zero, the second operator 155 reads the temporarily stored first output data and uses it as the second part of the data (step S1525).

If step S1521 and step S1523 do not meet the conditions, the second operator 155 determines whether the data currently being processed is the first batch of first part data (step S1531). If step S1531 does not meet the conditions, all second operations are terminated (step S1540). On the other hand, the second operator 155 reads the weights in the second filter used for the depth-by-depth convolution operation from the memory 110 (step S1533), performs the depth-by-depth convolution operation (step S1535), and outputs the obtained second output data to the second temporary memory 171 (step S1537). The second operator 155 determines whether all data of all bars in the current segment have been subjected to the depth-by-depth convolution operation (step S1538). If one or more data in the current segment have not completed the convolution operation, the second operator 155 moves to the next point index (for example, the second stride apart) (step S1527) and processes the next data. Until all data in all data in the first temporary memory 151 have been subjected to depth-wise convolution operation, the second operator 155 sets the number of currently processed data to: the original number of data + 1, and returns the currently processed width to zero (step S1539). Then, the second operator 155 completes all depth-wise convolution operations on the second input data (step S1540).

The third operator 175 determines whether the second output data temporarily stored in the second temporary storage memory 171 has reached one second output data (step S1541). The third operator 175 reads the temporarily stored second output data and uses it as the third part of data (step S1543). The third operator 175 reads the weight in the third filter used for the point-by-point convolution operation from the memory 110 (step S1551), performs the point-by-point convolution operation (step S1553), and outputs the obtained third output data to the memory 110 (step S1555). The third operator 175 determines whether all the data in all the pieces in the second temporary storage memory 171 have been subjected to the point-by-point convolution operation. If these data have not completed the point-by-point convolution operation, the second FIFO unit 172 continues to output the third part of the data to the third operator 175. If all the data in all the entries in the second temporary memory 171 have completed the point-by-point convolution operation, the third operator 175 completes all the point-by-point convolution operations on the third part of the data (step S1560).

The present invention further provides a computer-readable storage medium (e.g., a hard disk, an optical disk, a flash memory, a solid state disk (SSD), etc.) for storing program code. The computing circuit 100 or other processor can load the program code and execute the corresponding process of one or more data processing methods of the present invention. These processes can be referred to the above description and will not be repeated here.

In summary, in the computing circuit, data processing method and computer-readable storage medium based on the convolutional neural network of the embodiment of the present invention, the first output data and/or the second output data are temporarily stored without being output to the memory, and the second operation and/or the third operation can be started when the temporarily stored data meets the size required by the second operation and/or the third operation. In this way, the number of memory accesses can be reduced and the computing efficiency can be improved.

Although the present invention has been disclosed as above by the embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the relevant technical field can make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the attached patent application.

S210~S290: Steps

Claims (13)

A data processing method based on a convolutional neural network (CNN) includes: reading an input data from a memory; performing a first operation on a first part of the input data through a first operator to obtain a first output data, wherein the first operation is provided with a first filter, and the size of the first output data is related to the size of the first filter of the first operation and the size of the first part of the data; temporarily storing the first output data in a first temporary storage area; when the amount of the first output data temporarily stored in the first temporary storage area is greater than a first predetermined amount of data, M _H ×H _f1o is not less than H _kd , and M _W ×W _f1o is not less than W _kd , a second operation is performed on the first output data through a second operator to obtain a second output data, wherein the second operation is provided with a second filter, the first predetermined data amount is the size of the second filter, the size of the second output data is related to the size of the second filter of the second operation, the height of the second filter is H _kd , the width of the second filter is W _kd , the size of the second filter is W _kd ×H _kd , the height of the first output data of each segment is H _f1o , the width of the first output data of each segment is W _f1o , H _kd , W _kd , H _f1o and W _f1o are positive integers, and the amount of the first output data temporarily stored in the first buffer area is composed of a height of M _H ×H _f1o and a width of M _W ×W _f1o , _MH and _MW are multiples and positive integers; temporarily storing the second output data in a second temporary storage area; and performing a third operation on the second output data to output a third output data to the memory, wherein when the second operation is performed on the first output data, the first operation is continuously performed on the input data. The data processing method based on the convolution neural network as described in claim 1, wherein the second operation is different from the third operation, and the step of temporarily storing the second output data in the second buffer memory includes: when the amount of the second output data temporarily stored in the second buffer is greater than a second predetermined amount of data, performing the third operation on the second output data to obtain the third output data, wherein the third operation is provided with a third filter, and the size of the third output data is related to the size of the third filter. As described in claim 2, the data processing method based on the convolutional neural network, wherein the step of outputting the third output data obtained by performing the third operation on the second output data to the memory includes: when performing the third operation on the second output data, continuously performing the first operation and the second operation. A data processing method based on a convolutional neural network as described in claim 1, wherein the second operation is a depthwise convolution operation, and the step of performing the second operation on the first output data includes: performing the second operation when the amount of the first output data stored in the first buffer is greater than W _kd ×H _kd . A data processing method based on a convolutional neural network as described in claim 2, wherein the third operation is a pointwise convolution operation, the height and width of the third filter are both 1, and the step of performing the third operation on the third input data includes: performing the third operation on each of the temporarily stored second output data. As described in claim 4, the data processing method based on the convolution neural network, wherein the first operation is a convolution operation, and the step of reading the input data from the memory includes: reading the first portion of the input data in a first sliding direction, wherein the first sliding direction is parallel to the height of the input data. A data processing method based on a convolutional neural network as described in claim 4, wherein the first operation is a convolution operation, and the step of reading the input data from the memory includes: reading the first portion of the input data in a second sliding direction, wherein the second sliding direction is parallel to the width of the input data. The data processing method based on the convolution neural network as described in claim 2, wherein the step of performing the first operation on the first part of the input data, performing the second operation on the first output data, or performing the third operation on the second output data includes: dividing the first part of the data into a plurality of first systolic array inputs; and performing the first operation on the first systolic array inputs respectively to obtain a plurality of first systolic array outputs, wherein the first systolic array outputs constitute the first output data. A computation circuit based on a convolutional neural network comprises: a memory for storing input data; a processing element coupled to the memory and comprising: a first operator for performing a first operation on a first part of the input data to obtain a first output data, and temporarily storing the first output data in a first temporary storage of the processing element, wherein the first operation is provided with a first filter, and the size of the first output data is related to the size of the first filter of the first operation and the size of the first part of the data; a second operator for performing a first operation on a first part of the input data to obtain a first output data when the amount of the first output data temporarily stored in the first temporary storage is greater than a first predetermined amount of data, M _H ×H _f1o is not less than H _kd , and M _W ×W _f1o is not less than W _kd , the second operation is performed on the first output data to obtain a second output data, and the second output data is temporarily stored in a second temporary storage, wherein the second operation is provided with a second filter, the first predetermined data amount is the size of the second filter, the size of the second output data is related to the size of the second filter of the second operation, the height of the second filter is H _kd , the width of the second filter is W _kd , the size of the second filter is W _kd ×H _kd , the height of the first output data of each segment is H _f1o , the width of the first output data of each segment is W _f1o , H _kd , W _kd , H _f1o and W _f1o is a positive integer, the amount of the first output data temporarily stored in the first temporary storage memory has a height of _MH ×H _f1o and a width of _MW ×W _f1o , _MH and _MW are multiples and positive integers; the second temporary storage memory is used to store the second output data; and a third operator is used to output a third output data obtained by performing a third operation on the second output data to the memory, wherein when the second operator performs the second operation, the first operator continues to perform the first operation. The computing circuit based on the convolution neural network as described in claim 9, wherein the first operator has a first maximum computing amount within a unit time, the second operator has a second maximum computing amount within the unit time, the third operator has a third maximum computing amount within the unit time, the first maximum computing amount is greater than the second maximum computing amount, and the first maximum computing amount is greater than the third maximum computing amount. The computing circuit based on the convolutional neural network as described in claim 9, wherein when the third operator performs the third operation, the first operator and the second operator continue to perform the first operation and the second operation. The computing circuit based on the convolutional neural network as described in claim 9, wherein the first temporary storage memory and the second temporary storage memory are static random access memories. A computer-readable storage medium for storing a program code, a processor loading the program code to execute a data processing method based on a convolutional neural network as described in any one of claims 1 to 8.

Images