TWI616813B - Convolution operation method - Google Patents

Abstract

The invention discloses a convolution operation method, which comprises the steps of: dividing a large-scale convolution operation block into a plurality of small-scale convolution operation blocks; performing convolution operations on the small-scale convolution operation blocks to respectively Generate a partial result; and add partial results as a convolution result of the large-scale convolution operation block. The present invention also discloses a convolution operation device that can support the convolution operation method by hardware.

Description

Convolution operation method

The present invention relates to a convolution operation device and a convolution operation method, and more particularly to a convolution operation method capable of dividing a large-scale convolution operation block into a plurality of small-scale convolution operation blocks for convolution operation.

Deep learning is one of the important application technologies for the development of artificial intelligence (AI). The Convolutional Neural Network (CNN) is a deep learning efficient recognition technology that has attracted widespread attention in recent years. It consists of a plurality of cascaded eigenfilters or filters, and filters. The convolution operation block size can be from a cell block size of 1 × 1, 3 × 3 to a larger block size, such as a 5 × 5, 7 × 7 or even 11 × 11 large-scale convolution operation block.

However, convolution is a very cost-effective operation, especially for the convolution of large-scale convolutional blocks, which will take up most of the processor's performance. In addition, for the convolution operation unit of the filter of the operation data feature, it is usually only designed to calculate the specific convolution operation block size, or the specific input data size, and the convolution operation unit can only calculate less than the operation. The computational limit of the convolution operation block size or the hardware support limit. However, if a larger convolution operation block size operation is to be performed, the operation of the larger convolution operation block size must be completed by software assistance or additional hardware resources.

Therefore, it is necessary to propose a convolution operation method, which can reduce the limitation of the size of a specific convolution operation block, and can also achieve the operation of a large-scale convolution operation block without additional hardware resources, and also achieves a large The convolution operation result of the scale convolution operation block is one of the current important topics.

In view of the above problems, the present invention provides a convolution operation device and a convolution operation method, which can reduce the limitation of the size of a specific convolution operation block, and can also achieve a large-scale convolution operation area without additional hardware resources. The operation of the block, and the convolution operation result of the large-scale convolution operation block is also obtained.

To achieve the above object, the present invention provides a convolution operation method comprising the steps of: dividing a large-scale convolution operation block into a plurality of small-scale convolution operation blocks; and convolving a small-scale convolution operation block. The operation is to generate a partial result separately; and the partial results are added as a convolution operation result of the large-scale convolution operation block.

In an embodiment, the small-scale convolution operation blocks are the same size.

In an embodiment, the convolution operation method further includes filling a portion of the large-scale convolution operation block beyond 0 in the small-scale convolution operation block.

In an embodiment, in the step of performing a convolution operation, the small-scale convolution operation block performs a convolution operation using at least one convolution unit to respectively generate partial results, and the block size of the small-scale convolution operation block is equal to The maximum convolution size supported by the convolution unit hardware.

In one embodiment, in the step of performing a convolution operation, the small-scale convolution operation block performs parallel convolution operations using a corresponding number of convolution units to respectively generate partial results.

In an embodiment, the large-scale convolution operation block includes a plurality of filter coefficients, and the filter coefficients are assigned to the small-scale convolution operation block according to the arrangement order and the size of the small-scale convolution operation block.

In one embodiment, the large-scale convolution operation block includes a plurality of data, and the data is allocated to the small-scale convolution operation block according to the arrangement order and the size of the small-scale convolution operation block.

In one embodiment, the scale of the large-scale convolution operation block is 5×5 or 7×7, and the size of the small-scale convolution operation block is 3×3.

In an embodiment, the step of adding the partial results further comprises: providing a complex mobile address to each small-scale convolution operation block, and each part of the result is moved in a target according to the mobile address and superimposed on each other. .

In an embodiment, the convolution operation method further comprises determining a convolution operation mode according to a size of a current convolution operation block, wherein when the convolution operation mode is a split mode, the current volume The product operation block is a large-scale convolution operation block, and the large-scale convolution operation block is divided into a plurality of small-scale convolution operation blocks, and the convolution operation is performed on the small-scale convolution operation block to respectively generate the operation unit. The results, and the partial results are added as a result of the convolution operation of the large-scale convolution operation block. When the convolution operation mode is the non-split mode, the current convolution operation block is not divided, and the current convolution operation block is convoluted.

In an embodiment, the convolution operation method further includes performing a partial operation of a subsequent layer of the convolutional neural network.

To achieve the above object, the present invention also provides a convolution operation device which can operate the convolution operation method provided above.

As described above, the convolution operation method provided by the present invention divides a large-scale convolution operation block into a plurality of small-scale convolution operation blocks; convolution operation on a small-scale convolution operation block To generate a partial result separately; and to add partial results as a convolution operation result of a large-scale convolution operation block, the limitation of the size of a specific convolution operation block can be reduced without additional hard The volume resource can also achieve the operation of the large-scale convolution operation block, and the convolution operation result of the large-scale convolution operation block is also obtained.

1‧‧‧ memory

2‧‧‧buffering device

21‧‧‧Memory Control Unit

3‧‧‧Convolutional computing module

30‧‧‧Convolutional unit array

4‧‧‧Interlaced summing unit

5‧‧‧Additional buffer unit

51‧‧‧Partial total block

52‧‧‧ pooling unit

6‧‧‧ coefficient acquisition controller

7‧‧‧Control unit

71‧‧‧ instruction decoder

72‧‧‧Data Read Controller

8‧‧‧Data buffer controller

9‧‧‧Convolution unit

91‧‧‧ address decoder

92‧‧‧Adder

DMA‧‧‧direct memory access

F1~F9‧‧‧Small-scale convolution operation block

PE‧‧‧convolution unit

PE0~PE8‧‧‧Processing unit

(0,0), (0,3), (3,0), (3,3)‧‧‧ mobile address

FIG. 1 is a schematic diagram of a convolution operation on a two-dimensional data.

2 is a schematic diagram of a convolution unit.

3A is a schematic diagram of partitioning a 5×5 large-scale convolution operation block into four 3×3 small-scale convolution operation blocks according to an embodiment of the invention.

FIG. 3B is a schematic diagram of assigning a plurality of filter coefficients to a convolution operation block according to an arrangement order and a scale of a convolution operation block according to an embodiment of the present invention.

FIG. 3C is a schematic diagram of assigning a plurality of data to a convolution operation block according to an arrangement order and a scale of a convolution operation block according to an embodiment of the present invention.

4 is a schematic diagram of dividing a 7×7 large-scale convolution operation block into nine 3×3 small-scale convolution operation blocks according to still another embodiment of the present invention.

Figure 5 is a block diagram of a convolution operation device in accordance with an embodiment of the present invention.

Fig. 6 is a partial schematic view showing the convolution operation device shown in Fig. 5.

Figure 7 is a functional block diagram of a convolution unit in accordance with an embodiment of the present invention.

Hereinafter, a convolution operation device and a convolution operation method according to a preferred embodiment of the present invention will be described with reference to the accompanying drawings, wherein the same elements will be described with the same reference numerals.

Please refer to FIG. 1. FIG. 1 is a schematic diagram of convolution operation on a two-dimensional data. The two-dimensional data has a plurality of rows and columns, which is, for example, an image, and only 5×4 of the pixels therein are schematically shown. A 3×3 matrix size filter is used for convolution of two-dimensional data. The filter has coefficients FC0~FC8, and the moving step of the filter is smaller than the shortest width of the filter. The size of the filter is comparable to a sliding window or a convolution window. The moving window can be moved on the 5×4 image, and each time the movement, the corresponding data P0~P8 in the window is subjected to a 3×3 convolution operation, and the result of the convolution operation can be referred to as a feature value. The interval at which the moving window S moves each time is called stride, and since the stride does not exceed the size of the sliding window or the convolution size, With the moving window stride of this embodiment, it will be less than the moving distance of 3 pixels. Moreover, adjacent convolution operations tend to have overlapping data. In the case where the stride is equal to 1, the data P2, P5, and P8 are new data, and the data P0, P1, P3, P4, P6, and P7 are data that has been input by the previous round of convolution operations. For the application of the general convolutional neural network, the commonly used moving window size is 1×1, 3×3, 5×5, 7×7, and the moving window size of the embodiment is more commonly used (3) ×3).

2 is a schematic diagram of a convolution unit. Referring to FIG. 2, the convolution unit of FIG. 2 can perform the convolution operation of FIG. 1. The convolution unit has 9 multipliers Mul_0~Mul_8 of a 3×3 array, each multiplier having a data input and a filter coefficient. The input and a multiply output OUT, the data input and the filter coefficient input are the two multiplication inputs of each multiplier. The output OUT of each multiplier is connected to the input #0~#8 of the adder, respectively, and the adder adds the outputs of the multipliers to generate a convolution output OUT. After performing a round of convolution operations, the multipliers Mul_0, Mul_3, and Mul_6 input the data currently in them (Q0, Q1, Q2 input in the current round) to the multipliers Mul_1, Mul_4, and Mul_7 in the next stage, and the multiplier Mul_1, Mul_4, and Mul_7 will input the current data (equivalent to the previous round of Q0, Q1, Q2) to the next level. The multipliers Mul_2, Mul_5, Mul_8, so that some of the data that has been input into the convolution unit can be reserved for the next round of convolution operations. The multipliers Mul_0, Mul_3, and Mul_6 receive new data Q0, Q1, and Q2 in the next round. The previous round, the current round, and the next round of convolution operations are separated from each other by at least one clock.

For example, in the general case, the filter coefficients do not need to be updated from time to time, and the coefficients FC0~FC8 are input to the multipliers Mul_0~Mul_8 and can be left in the multipliers Mul_0~Mul_8 for multiplication; or the coefficients FC0~FC8 Always input to the multipliers Mul_0~Mul_8.

In other embodiments, the convolution unit may also be an array different from a 3×3 size, such as a 5×5, 7×7 size array, which is not limited in the present invention. The convolution unit PE can also separately process convolution operations of different sets of input data.

Please refer to FIG. 3A to FIG. 3C together. FIG. 3A is a schematic diagram of dividing a 5×5 large-scale convolution operation block into four 3×3 small-scale convolution operation blocks according to an embodiment of the present invention. FIG. 3B is a schematic diagram of assigning a plurality of filter coefficients to a convolution operation block according to an arrangement order and a scale of a convolution operation block according to an embodiment of the present invention. FIG. 3C is a schematic diagram of assigning a plurality of data to a convolution operation block according to an arrangement order and a scale of a convolution operation block according to an embodiment of the present invention.

In FIG. 3A, a filter for preferentially setting a two-dimensional 5×5 pixel data is provided, which is a 5×5 convolution operation unit array or a 5×5 large-scale convolution operation block (hereinafter referred to as a volume). Product operation block). In Fig. 3B, 5 x 5 pixel data corresponding to the original 5 x 5 large-scale convolution operation block is revealed. In general, usually 5 × 5 pixel data, processing with 5 × 5 large-scale convolution operation block will be more direct and efficient, but if the hardware function of the convolution operation device can not reach support 5 × 5 The convolution operation of the convolution operation block requires convolution operation by other methods.

For this reason, please refer to FIG. 3A first. In FIG. 3A, the original 5×5 large-scale convolution operation block is first divided into a plurality of small-scale convolution operation blocks. In this embodiment, it is divided into four 3×3 small-scale convolution operation blocks, and the sizes of the four small-scale convolution operation blocks are the same. In other embodiments, 5 ×5 large-scale convolution operation block even larger 7×7 large-scale convolution operation block is divided into multiple smaller-scale convolution operation areas The block is, for example, a 1×1 small-scale convolution operation block, and the present invention is not limited. It should be noted that the number of rows and columns of the block of the 5×5 large-scale convolution operation block before the division is not an integer multiple of the number of rows and columns of the small-scale convolution operation block, and four small-scale convolutions. The arithmetic block will exceed the original 5×5 large-scale convolution operation block. Therefore, the convolution operation method of the present invention requires an additional 0 in the portion of each small-scale convolution operation block that exceeds the large-scale convolution operation block. In this embodiment, the original 5×5 large-scale convolution operation block is added with a coefficient of 0 for one row and one column, so that the number of rows and columns of the large-scale convolution operation block after the coefficient 0 is added (6× 6) may be an integer multiple of the number of rows and columns of the 3×3 small-scale convolution operation block, and each of the 3×3 small-scale convolution operation blocks does not overlap each other. After dividing or disassembling the large-scale convolution operation block, a total of 4 3×3 small-scale convolution operation blocks will be generated. For convenience of explanation, the disassembled small-scale convolutions will be described here. The operation blocks are respectively called small-scale convolution operation blocks F1~F4.

After the small-scale convolution operation blocks F1 to F4 are obtained, the convolution operations of the pixel data can be started by using the small-scale convolution operation blocks F1 to F4, and thus a part of the result (image result) is generated separately. . Please refer to FIG. 3B and FIG. 3C. FIG. 3B and FIG. 3C respectively show that the large-scale convolution operation block includes a plurality of filter coefficients and a plurality of data, each filter coefficient and each data can be arranged according to its own order, and small scale. The scale of the convolution operation blocks F1 to F4 is assigned to the small-scale convolution operation blocks F1 to F4.

Further, in the step of performing the convolution operation, the small-scale convolution operation blocks F1 to F4 perform convolution operations using at least one convolution unit to respectively generate partial results, and here, at least four convolutions are used. The unit performs a convolution operation (F4 has only 4 convolution units), and the block size of the small-scale convolution operation blocks F1 to F4 is equal to the maximum convolution size supported by the convolution unit hardware. In other words, the small-scale convolution operation blocks F1 to F4 are the hardware support upper limit of the present embodiment, and at most, only the convolution operation block of the 3×3 scale is supported. Further, the small-scale convolution operation blocks F1 to F4 perform parallel convolution operations using a corresponding number of convolution units to respectively generate partial partial results.

When the small-scale convolution operation blocks F1 to F4 perform convolution operations and respectively generate partial results, finally, the partial results of the small-scale convolution operation blocks F1 to F4 are added as the 5×5 large The result of the convolution operation of the scale convolution operation block. The method of adding the partial results can provide a complex mobile address to each small-scale convolution operation block, and the results of each part can be Moves in a target and superimposes each other according to the provided mobile address. For example, different mobile addresses (0, 0), (0, 3), (3, 0), and (3, 3) may be respectively assigned to the small-scale convolution operation blocks F1, F2, F3, and F4. . Since the small-scale convolution operation blocks F1~F4 do not overlap each other and have different mobile addresses, each small-scale convolution operation block F1~F4 can scan FIG. 3B according to the plurality of filter coefficients assigned. The data in the (pixel data), and the results of the various parts I1~I4 and the final part of the result I5 (the figure does not show I1~I5). Then, after the final partial result I5 initial buffer value is set to 0, the partial results I1 to I4 outputted by the four small-scale convolution operation blocks F1 to F4 are started to be added.

Since the mobile address of the small-scale convolution operation block F1 is (0, 0), part of the result I1 will be directly superimposed on the final partial result I5; and the mobile address of the small-scale convolution operation block F2 For (0,3), the pixel data of the coordinates (X, Y) in the output partial result I2 is superimposed on the coordinate part (X, Y-3) in the final partial result I5; small-scale convolution operation The moving address of the block F3 is (3, 0), so the data of each pixel located at the coordinates (X, Y) in the output partial result I3 is superimposed on the coordinate (X-3, Y) in the final partial result I5. The moving address of the small-scale convolution operation block F4 is (3, 3), so the pixel data of the coordinates (X, Y) in the output partial result I4 will be at coordinates (X-3, Y-3). ) superimposed on the final partial result I5. Thereby, the output partial results I1~I4 of the small-scale convolution operation blocks have been moved in the coordinates according to different mobile addresses and superimposed on each other to generate a final partial result I5.

Therefore, the first step of the disassembling method of the embodiment is: dividing a large-scale convolution operation block into a plurality of small-scale convolution operation blocks (step: S10); and then, the small-scale convolutions The operation block performs a convolution operation to respectively generate a partial result; (step: S20); and adding the partial results as a convolution operation result of the large-scale convolution operation block (step: S30) .

In addition, in step S10, a step of dividing a large-scale convolution operation block into a plurality of small-scale convolution operation blocks, when the large-scale convolution operation block exceeds the large-scale convolution operation block, Then, a step is further included: a portion exceeding the large-scale convolution operation block in the small-scale convolution operation blocks is filled with 0 (step: S11). Moreover, the step S30 adds the partial results as a result of a convolution operation of the large-scale convolution operation block, and further includes a step S31: providing a complex mobile address to the small-scale convolution operations Block, these partial results According to the mobile addresses, they move in a target and are superimposed on each other (step: S31).

Please refer to FIG. 4. FIG. 4 is a schematic diagram of dividing a 7×7 large-scale convolution operation block into nine 3×3 small-scale convolution operation blocks according to still another embodiment of the present invention.

The 7×7 large-scale convolution operation block of this embodiment has a splitting and disassembling method and a convolution operation method similar to the 5×5 large-scale convolution operation block of the previous embodiment, due to the large scale of 7×7. The number of rows and columns of the block in the convolution operation block is also not an integer multiple of the number of rows and columns of the 3×3 small-scale convolution operation block, and the 9 small-scale convolution operation blocks will exceed the original 7 ×7 large-scale convolution operation block. Therefore, the convolution operation method of the present invention requires an additional 0 in the portion of each small-scale convolution operation block that exceeds the large-scale convolution operation block. In this embodiment, the original 7×7 large-scale convolution operation block is added with the coefficient 0 of each two rows and two columns, so that the number of large-scale convolution operation blocks and the number of rows after the coefficient 0 is added. (9×9) may be an integer multiple of the number of rows and columns of the 3×3 small-scale convolution operation block. After dividing or disassembling the large-scale convolution operation block, a total of 9 3×3 small-scale convolution operation blocks will be generated, and the disassembled small-scale convolution operation blocks will be respectively ordered. For small-scale convolution operation blocks F1~F9. And each of the 3×3 small-scale convolution operation blocks does not overlap each other as non-overlapping. Finally, each small-scale convolution operation block F1~F9 can also output partial results I1~I9 and move in coordinates according to different mobile addresses and superimpose each other to generate a final partial result I10.

In addition, in this embodiment, regarding other technical features of dividing a 7×7 large-scale convolution operation block into nine 3×3 small-scale convolution operation blocks and performing convolution operation on the data, refer to the previous embodiment. The relevant description will not be repeated here.

In addition, in an embodiment, the convolution operation method further includes determining a convolution operation mode according to a size of a current convolution operation block. It is processed in an appropriate convolution operation mode for blocks of different sizes.

When the convolution operation mode is the split mode, the current convolution operation block is a large-scale convolution operation block, and the large-scale convolution operation block is divided into a plurality of small-scale convolution operation blocks, and then the small-scale segmentation is performed. The convolution operation block performs a convolution operation to separately generate partial results, and then adds these partial results as a convolution operation result of the large-scale convolution operation block.

When the convolution operation mode is the non-split mode, the current convolution operation block is not divided, and the convolution operation is directly performed on the current convolution operation block.

In addition, the convolution operation method may further include performing a partial operation of a subsequent layer of the convolutional neural network. Partial operations, for example, are to perform operations on the current layer of the convolutional neural network by summing, averaging, maximizing, or other operations of subsequent layers.

The following describes an example of a hardware that can support the aforementioned operations. Please refer to FIG. 5. FIG. 5 is a block diagram of a convolution operation device according to an embodiment of the present invention. The convolution operation device includes a memory 1, a buffer device 2, a convolution operation module 3, an interleaving summation unit 4, a total buffer unit 5, a coefficient extraction controller 6, and a control unit 7. The convolutional computing device can be used in the application of the Convolutional Neural Network (CNN).

The memory 1 stores data to be convoluted, and may be, for example, image, video, audio, statistics, data of one layer of a convolutional neural network, and the like. In the case of image data, for example, pixel data; in the case of video data, for example, pixel data of a video frame or a motion vector, or audio in a video; In terms of data, it is usually a two-dimensional array of data, and in the case of image data, it is usually a two-dimensional array of pixel data. In addition, in the embodiment, the memory 1 is a static random-access memory (SRAM), which can store the volume in addition to the data to be convoluted. The data is completed by the product, and may have a multi-layered storage structure and separately store the data to be calculated and operated. In other words, the memory 1 can be used as a cache memory inside the convolution operation device.

In practical applications, all or most of the data may be stored elsewhere, for example, in another memory, and another memory may be selected as dynamic random access memory (DRAM) or other types. Memory. When the convolution operation device is to perform the convolution operation, the data is completely or partially loaded into the memory 1 by another memory, and then the data is input to the convolution operation module 3 through the buffer device 2 to perform the volume. Product operation. If the input data is streamed from the data stream, the memory 1 will write the latest data from the data stream for convolution at any time.

For example, a control unit or a processing unit can control which mode of convolution operation is to be performed. When the control unit or processing unit finds that the size of the convolution operation block is larger than the maximum size that the hardware can directly calculate, Split mode to operate. For example, the convolution operation module 3 hardware can only directly perform a 3×3 convolution operation, and the control unit or the processing unit will first The current convolution operation block is divided into a plurality of 3×3 operation blocks, and then the 3×3 operation block is sequentially written to the memory 1, and the convolution operation device is commanded to perform 3×3 operation blocks. The convolution operation of ×3, the divided 3×3 operation block is convoluted by the convolution operation module 3 to generate partial results, respectively. These partial results are added as a result of the convolution operation of the original current convolution operation block. For example, the partial results are added by the summation buffer unit 5, and the added result is then written to the memory through the buffer device 2. The body 1, the control unit or the processing unit obtains the convolution operation result of the original current convolution operation block from the memory 1. In addition, these partial results may also be added to the memory 1 through the buffer device 2 without being added by the total buffer unit 5, and the control unit or the processing unit may obtain these partial results from the memory 1 and then These partial results are taken as the result of the convolution operation of the original current convolution operation block.

The buffer device 2 is coupled to the memory 1, the convolution operation module 3, and the total buffer unit 5. Further, the buffer device 2 is also coupled to other elements of the convolution operation device, such as the interleaving summing unit 4 and the control unit 7. In addition, for frame data calculation of image data or video, the order of processing is to read multiple columns at the same time, so in one clock, the buffer device 2 is from the memory 1 The data on the different columns of the same row are input. For this, the buffer device 2 of the present embodiment functions as a buffer device for a column buffer. When the calculation is to be performed, the buffer device 2 can first extract the data required by the convolution operation module 3 from the memory 1, and adjust the data to be successfully written into the convolution operation module 3 after the capture. Data type. On the other hand, since the buffer device 2 is also coupled to the summing buffer unit 5, the data after the calculation of the total buffer unit 5 is also reordered by the buffer device 2 and then transferred back to the memory 1 for storage. . In other words, the buffer device 2 has a function similar to relaying temporary data in addition to the function of line buffering, or the buffer device 2 can be used as a data register having a sorting function.

It is worth mentioning that the buffer device 2 further includes a memory control unit 21, which can be controlled via the memory control unit 21 when the buffer device 2 performs data capture or writing with the memory 1. In addition, since it has a limited memory access width with the memory 1, or is also called bandwidth or bandwidth, the convolution operation module 3 can actually perform convolution operations with the memory. 1 is related to the access width. In other words, the computational efficiency of the convolutional computing module 3 is limited by the aforementioned access width. Therefore, if the input of the memory 1 has a bottleneck, Then the performance of the convolution operation will be affected by the impact.

The convolution operation module 3 has a plurality of convolution units, each convolution unit performs a convolution operation based on a filter and a plurality of current data, and retains part of the current data after the convolution operation. The buffer device 2 acquires a plurality of new data from the memory 1, and inputs the new data to the convolution unit, and the new data is not overlapped with the current data, for example, the previous round convolution operation is not used but the current convolution operation The data to be used. The convolution unit of the convolution operation module 3 performs a second round convolution operation based on the filter, the retained current data, and the new data. The interleaving summation unit 4 is coupled to the convolution operation module 3 to generate a feature output result according to the result of the convolution operation. The summing buffer unit 5 is coupled to the interleaving summing unit 4 and the buffer device 2 to temporarily store the feature output result; wherein, when the convolution operation of the specified range is completed, the buffer device 2 will temporarily store all the data from the summing buffer unit 5. Write to memory 1.

The coefficient acquisition controller 6 is coupled to the convolution operation module 3, and the control unit 7 is coupled to the buffer device 2. In actual application, for the convolution operation module 3, the input source required for the convolution operation module 3 needs to input a coefficient of a filter in addition to the data itself, and the operation is started. The coefficient input of the 3 × 3 convolution unit array is referred to in the present embodiment. The coefficient acquisition controller 6 can directly input the filter coefficients from the external memory by direct memory access DMA (direct memory access). In addition to being coupled to the convolution operation module 3, the coefficient capture controller 6 can also be coupled to the buffer device 2 to accept various commands from the control unit 7, so that the convolution operation module 3 can be controlled by the control unit 7. The control coefficient capture controller 6 performs input of filter coefficients.

Control unit 7 can include an instruction decoder 71 and a data read controller 72. The instruction decoder 71 obtains a control instruction from the data read controller 72 and decodes the instruction, thereby obtaining the size of the current input data, the number of rows of the input data, the number of columns of the input data, the feature number of the input data, and the input data in the memory. The starting address in body 1. Alternatively, the command decoder 71 may obtain the type information of the filter and the number of the output feature from the data read controller 72, and output an appropriate vacant signal to the buffer device 2. The buffer device 2 operates according to the information provided by the instruction decoding, and further controls the operation of the convolution operation module 3 and the summation buffer unit 5, for example, data is input from the memory 1 to the buffer device 2 and the convolution operation module. The timing of 3, the scale of the convolution operation of the convolution operation module 3, and the data from the memory 1 to the buffer device 2 The address, the data from the sum buffer unit 5 to the write address of the memory 1, the convolution operation module 3, and the convolution mode operated by the buffer device 2 are read.

On the other hand, the control unit 7 can also extract the required control commands and convolution information from the external memory by means of direct memory access DMA. After the instruction decoder 71 decodes the instructions, the control commands and The convolution information is retrieved by the buffer device 2, and the instruction may include the stride size of the moving window, the address of the moving window, and the number of image data rows and columns of the feature to be extracted.

The summing buffer unit 5 is coupled to the interleaving summing unit 4, and the summing buffer unit 5 includes a part of the summing block 51 and a pooling unit 52. The partial summing block 51 temporarily stores the data output by the interleaving summing unit 4. The pooling unit 52 performs a pooling operation on the data temporarily stored in the partial summing block 51. Pooling operations are either pooled or averaged.

For example, the summed buffer unit 5 may temporarily store the result of the convolution calculation via the convolution operation module 3 and the output feature result of the interleaving summation unit 4 in the partial summing block 51. Then, the pooling unit 52 performs a pooling operation on the data temporarily stored in the partial summing block 51. The pooling operation may take an average value or take a specific feature on a certain area of the input data. The maximum value is used as a summary feature extraction or statistical feature output. This statistical feature not only has a lower dimension than the previous feature, but also improves the processing result of the operation.

It should be noted that the temporary storage here still temporarily stores the partial operation result in the input data and then temporarily stores it in the partial summing block 51, so it is called a partial summing area. Block 51 and summing buffer unit 5, or simply referred to as PSUM unit and PSUM BUFFER module. On the other hand, the pooling operation of the pooling unit 52 of the present embodiment uses the max pooling method to obtain statistical feature output. In other embodiments, the average pooling (avg pooling) may also be selected. The calculation method of the statistical characteristics is obtained, and the pooling method is determined by the usage requirements. After all the data to be input are processed by the convolution operation module 3 and the interleave summation unit 4, the total buffer unit 5 outputs the final data processing result, and the result can be restored to the memory through the buffer device 2 as well. Body 1, or re-transmission through memory 1 to other components. At the same time, the convolution operation module 3 and the interleave summation unit 4 continue to perform data feature acquisition and calculation to improve the processing performance of the convolution operation device.

The convolution operation device may include a plurality of convolution operation modules 3, and the convolution unit of the convolution operation module 3 and the interleave summation unit 4 are selectively operable in a low-scale convolution mode. And a high-volume convolution model. In the low-scale convolution mode, the interleaving summing unit 4 is configured to interleave the results of the respective convolution operations in the corresponding order in the convolution operation module 3 to output a total result. In the high-scale convolution mode, the interleaving summing unit 4 interleaves the results of the convolution operations of the respective convolution units as outputs.

For example, the control unit 7 can receive a control signal or a mode command, and according to the control signal or mode command, determine which mode the other modules and the unit are to operate in. This control signal or mode command can be derived from other control units or processing units.

Please refer to FIG. 6. FIG. 6 is a partial schematic diagram of the convolution operation device shown in FIG. 5. The coefficient acquisition controller 6 couples the respective 3×3 convolution units in the convolution operation module 3 through the lines of the filter coefficients FC and the control signals Ctrl. When the buffer device 2 acquires the instruction, the convolution information, and the data, it controls each convolution unit to perform a convolution operation.

The interleaving summation unit 4 is coupled to the convolution operation module 3, and the convolution operation module 3 can operate on the different characteristics of the input data and output the feature operation result. For data writing of multiple features, the convolution operation module 3 can output multiple calculation results correspondingly. The function of the interleaving and summing unit 4 is that the result of the operation of the convolutional computing module 3 can be combined to obtain an output characteristic result. After the interleaved summation unit 4 obtains the output feature result, the output feature result is transmitted to the summation buffer unit 5 to facilitate the next stage of processing.

For example, the convolutional neural network has multiple operation layers, such as a convolutional layer, a pooled layer, etc., and the number of layers of the convolutional layer and the pooled layer may be multiple layers, and the output of each layer may be used as an input of another layer or subsequent layers. For example, the output of the Nth layer convolutional layer is the input of the Nth layer pooling layer or the input of other subsequent layers, and the output of the Nth layer pooling layer is the input of the N+1th convolution layer or other subsequent layers. Input, the output of the Nth operation layer may be the input of the N+1th operation layer.

In order to improve the computational efficiency, when performing the operation of the Nth operation layer, the partial operation of the N+i (i>0, N, i is a natural number) layer operation layer can be performed depending on the use of the computing resource (hardware). The effective use of computing resources and can reduce the amount of computation actually in the N+i layer operation layer.

In the present embodiment, in an operation case, for example, a 3×3 convolution operation, the convolution operation module 3 performs a convolutional layer operation of a convolutional neural network, and the interleave summation unit 4 does not perform a convolutional nerve. Partial operation of the subsequent layers of the network, summing up the buffer unit 5 to perform convolution The operation of the same hierarchical pooling layer of the network. In another operation, such as a 1×1 convolution operation, the convolution operation module 3 performs a convolutional layer operation of a convolutional neural network, and the interleaving summation unit 4 performs a subsequent layer of the convolutional neural network. The partial operation, the partial operation is, for example, addition and addition, and the total buffer unit 5 performs the operation of the same hierarchical pooling layer of the convolutional neural network. In other embodiments, the summing buffer unit 5 may perform partial operations of subsequent layers of the convolutional neural network in addition to the operations of the pooling layer. The partial operation described above is, for example, performing a summation operation, an averaging operation, a maximum value operation or the like of a subsequent layer on the current layer of the convolutional neural network.

FIG. 7 is a functional block diagram of a convolution unit according to an embodiment of the present invention. As shown in FIG. 7, the convolution unit 9 includes nine processing units PE0~PE8 (process engine), a bit address decoder 91, and an adder. 92. The convolution unit 9 can be used as the aforementioned convolution unit.

In the 3×3 convolution operation mode, the input data to be convoluted is input to the processing units PE0~PE2 via the line data[47:0], and the processing units PE0~PE2 will input the current clock input data later. The clock is input to the processing unit PE3~PE5 for the next round of convolution operation, and the processing units PE3~PE5 input the current clock input data to the processing unit PE6~PE8 for the next round of convolution operation. The 3 × 3 filter coefficients are input to the processing units PE0 to PE8 through the line fc_bus[47:0]. When the stride is 1, 3 new data will be input to the processing unit, and the 6 old data that has been input will be moved to other processing units. When the convolution operation is performed, the processing units PE0 to PE8 multiply the filter coefficients of the selected address and the input data input to the processing units PE0 to PE8 through the address decoder 91. When the convolution unit 9 performs a 3 × 3 convolution operation, the adder 92 adds the results of the multiplication operations to obtain the result of the convolution operation as the output psum[35:0].

When the convolution unit 9 performs a 1×1 convolution operation, the input data to be convoluted is input to the processing units PE0 to PE2 via the line data[47:0], and the three 1×1 filter coefficients are transmitted through the line fc_bus [ 47:0] Input to processing unit PE0~PE2. When the stride is 1, 3 new data is input to the processing unit. When the convolution operation is performed, the processing units PE0 to PE2 multiply the filter coefficients of the selected address and the input data input to the processing units PE0 to PE2 through the address decoder 91. When the convolution unit 9 performs a 1×1 convolution operation, the adder 92 directly takes the result of the convolution operation of the processing units PE0 to PE2 as the output pm_0[31:0], pm_1[31:0], pm_2[31 :0]. In addition, since the processing units PE3~PE8 do not actually participate in the convolution operation, these processes Units PE3~PE8 can be turned off first to save power. In addition, although the convolution unit 9 has three outputs of 1 × 1 convolution operation, only two of the outputs may be connected to the interleave summing unit 4; or the outputs of three 1 × 1 convolution operations are connected to the interleave plus The total unit 4 determines the number of 1×1 convolution operation results outputted to the interleaving summing unit 4 by controlling whether the processing units PE0 to PE2 are turned off or not.

After all the data of the image has been calculated by the convolution operation module 3, the interleave summation unit 4, and the total buffer unit 5, and the final data processing result has been restored to the memory 1, the buffer device 2 is used. A stop signal is transmitted to the command decoder 71 and the control unit 7, notifying that the control unit 7 has been calculated and waiting for the next processing command.

Thereby, since each convolution unit of the convolution operation device can retain part of the current data after the convolution operation, the buffer device acquires a plurality of new data from the memory and inputs the new data to the convolution unit, the new data is not current The data is repeated, so the processing efficiency of the convolution operation is improved, and it is suitable for the convolution operation of the stream data. Therefore, when processing data for convolution operations and continuous parallel operations, it is possible to have excellent arithmetic performance and low power consumption performance, and is capable of processing data streams.

The convolution operation method can be applied or implemented in the convolution operation device of the foregoing embodiment, and related changes and implementations will not be described again. The convolution operation method can also be applied or implemented in other computing devices. For example, the convolution operation method can be used in a processor capable of executing instructions, and the instructions configured to execute the convolution operation method are stored in a memory, and the processor couples the memory and executes the instructions to perform a convolution operation method. For example, the processor includes a cache memory, a math operation unit, and an internal register, the cache memory stores the data stream, the math operation unit can perform a convolution operation, and the internal register can retain the portion of the current convolution operation. The data is used in the convolutional computing module for the next round of convolution operations.

In summary, the convolution operation method provided by the present invention divides a large-scale convolution operation block into a plurality of small-scale convolution operation blocks; performs convolution operation on the small-scale convolution operation block. To generate a partial result separately; and to add partial results as a convolution operation result of a large-scale convolution operation block, the limitation of the size of a specific convolution operation block can be reduced without additional hard The volume resource can also achieve the operation of the large-scale convolution operation block, and the convolution operation result of the large-scale convolution operation block is also obtained.

The above is intended to be illustrative only and not limiting. Any equivalent modifications or alterations to the spirit and scope of the invention are intended to be included in the scope of the appended claims.

Claims (10)

A convolution operation method comprising the steps of: determining a convolution operation mode according to a size of a current convolution operation block; and when the convolution operation mode is a division mode, the current convolution operation block is a large-scale convolution operation a block, the large-scale convolution operation block is divided into a plurality of small-scale convolution operation blocks; when the convolution operation mode is a non-split mode, the current convolution operation block is not divided, and the The current convolution operation block performs a convolution operation; convolution operations are performed on the small-scale convolution operation blocks to respectively generate a partial result; and the partial results are added as the large-scale convolution operation area The result of a convolution operation of the block. The convolution operation method of claim 1, wherein the small-scale convolution operation blocks have the same size. The convolution operation method of claim 1, further comprising: filling a portion of the small-scale convolution operation block beyond 0 in the small-scale convolution operation block. The convolution operation method according to claim 1, wherein in the step of performing a convolution operation, the small-scale convolution operation blocks perform convolution operations using at least one convolution unit to respectively generate the portion As a result, the block size of the small-scale convolution operation blocks is equal to the maximum convolution size supported by the convolution unit hardware. The convolution operation method according to claim 1, wherein in the step of performing a convolution operation, the small-scale convolution operation blocks are respectively subjected to parallel convolution operations by a corresponding number of convolution units to respectively generate Part of the result. The convolution operation method of claim 1, wherein the large-scale convolution operation block comprises a plurality of filter coefficients, and the filter coefficients are arranged according to an arrangement order and the small-scale convolution operation blocks. Size is assigned to these small-scale convolution operation blocks. The convolution operation method of claim 1, wherein the large-scale convolution operation block includes a plurality of data, and the data is allocated according to an arrangement order and a size of the small-scale convolution operation blocks. These small-scale convolution operation blocks. The convolution operation method according to claim 1, wherein the large-scale convolution operation block has a size of 5×5 or 7×7, and the size of the small-scale convolution operation block is 3×3. . The convolution operation method of claim 1, wherein the step of adding the partial results further comprises: providing a complex mobile address to the small-scale convolution operation blocks, the portions The results are moved in a target and superimposed on each other based on the mobile addresses. For example, the convolution operation method described in claim 1 further includes: performing a partial operation of a subsequent layer of the convolutional neural network.