JP7467786B2 - DATA PROCESSING APPARATUS AND DATA PROCESSING METHOD - Google Patents

Description

Article 30, paragraph 2 of the Patent Act applies. (1) Presented in the proceedings of the 8th International Congress on Advanced Applied Informatics on July 7, 2019. (2) Presented at the 8th International Congress on Advanced Applied Informatics on July 7, 2019. (3) Presented at the IoT Workshop "Trends in Innovative IoT Businesses Using Sensing Edge" - Progress and Use Cases of IoT Devices and Systems for Industry and Infrastructure in 2019, 1st Annual Meeting, on November 27, 2019. (4) Presented at https://ieeeexplorer.ieee. Published at org/document/8992640.

The present invention relates to a data processing device and a data processing method, and in particular to a data processing device and a data processing method suitable for convolution operations in a convolutional neural network.

In recent years, convolutional neural networks (CNNs), which add convolution to neural networks, have been widely recognized as an effective machine learning technique for image recognition and other purposes. Below, we provide an overview of CNNs.

Figure 1 is a diagram illustrating an outline of the system configuration of a CNN. In the CNN shown in Figure 1, layer L1 and layer L2 each include a convolutional layer and a pooling layer.

The convolutional layer is a layer that performs an operation to multiply the input data by the feature weights of the filters (kernels) (an operation to convolve the features). Specifically, when the input data is image data, the convolutional layer multiplies the input data by the feature weights of the different filters to obtain image data corresponding to the number of filters. In other words, by using multiple filters in the convolutional layer, it becomes possible to detect patterns in the input data that can capture various features of the input data (image data).

The pooling layer is a layer that is placed immediately after the convolution layer. The pooling layer shrinks the layer, making it easier to perform subsequent processing, and also reducing the position sensitivity of the features extracted in the convolution layer.

Then, in the CNN, fully connected multilayer perceptrons are arranged in layers L3 to L5 to recognize the input data (image data) (see, for example, Patent Document 1).

JP 2019-003414 A

Here, in the convolution calculations in the above-mentioned CNN (especially the convolution calculations performed in the early layers), a huge number of product-sum calculations are performed. Below, we will explain the product-sum calculations performed in the convolution calculations.

Figure 2 is a diagram explaining the product-sum operation performed in the convolution operation. The input data (input feature map) in the example shown in Figure 2(A) has a data size of Nix x Niy and the number of channels is Nif. The filters (filters) in the example shown in Figure 2(A) have a data size of Nkx x Nky. Furthermore, the output data (output feature map) in the example shown in Figure 2(A) has a data size of Nox x Noy and the number of channels is Nof.

In the convolution operation in this example, as shown in Fig. 2B, "pixel _L (no; x, y) + = pixel _L-1 (ni; x + kx, y + ky) * weight _L-1 (ni, no; kx, ky)" is repeatedly performed as a multiplication and addition operation. Note that in the example shown in Fig. 2B, if there is no "bias (no)", the initial value of "pixel _L (no; x, y)" will be 0.

Here, the number of times the multiply-and-accumulate operation is performed in the example shown in FIG. 2B is Nof×Nox×Noy×Nif×Nkx×Nky (times). Therefore, depending on the size of Nof, etc., the number of times the multiply-and-accumulate operation needs to be performed may become enormous.

Therefore, for example, to utilize CNN in various fields such as robot motion control and autonomous driving, it is necessary to perform the convolution operations (product-accumulation operations) in the convolution layer in parallel as much as possible to speed up the processing time in CNN.

The object of the present invention is to provide a data processing device and a data processing method that enable high-speed convolution operations.

In order to achieve the above object, the data processing device of the present invention is a data processing device that calculates a feature amount of two-dimensional input data using a plurality of filters corresponding to the two-dimensional input data, and has a processing element group consisting of a plurality of processing elements arranged in two dimensions, and a first storage unit having a three-dimensional storage area for each of the plurality of processing elements and for each of the plurality of filters, and the processing element group transmits each of the data included in the two-dimensional input data to one of the plurality of processing elements such that all of the transmission destinations are different processing elements, and for each of the plurality of filters, obtains one feature weight included in each filter, and generates three-dimensional feature weights by duplicating the one feature weight obtained for each of the plurality of filters the same number as the plurality of processing elements and arranging them in the same arrangement as the plurality of processing elements, and transmits the one feature weight obtained for each of the plurality of filters to each of the plurality of processing elements. the first storage unit, and causes each of the processing elements to multiply the data included in the two-dimensional input data transmitted by the one feature weight for each of the filters transmitted to calculate a three-dimensional product; the first storage unit, and each of the processing elements, transmits the data included in the two-dimensional input data held by each of the processing elements to an adjacent processing element so that data not yet used in the calculation of the three-dimensional product is transmitted to each of the processing elements; and the process of acquiring, generating, transmitting each of the one feature weight for each of the filters, calculating, adding, and transmitting the data included in the two-dimensional input data to an adjacent processing element are repeated until the number of times the adding process has been performed reaches the number of feature weights included in each of the filters.

To achieve the above object, the data processing device of the present invention is characterized in that the group of processing elements performs processing in the convolutional layer of a convolutional neural network.

To achieve the above object, the data processing device of the present invention is characterized in that, after the repeated processing, the processing element group outputs the three-dimensional values stored in the first storage unit.

To achieve the above object, the data processing device of the present invention has a second storage unit having a three-dimensional storage area for each of the plurality of processing elements and for each of the plurality of filters, and the processing element group stores the generated three-dimensional feature weights in the second storage unit.

To achieve the above object, the data processing device of the present invention is characterized in that it has a third storage unit having a two-dimensional storage area for each of the plurality of processing elements, and the group of processing elements transmits each of the data contained in the two-dimensional input data stored in the third storage unit to one of the plurality of processing elements such that all of the data are transmitted to different processing elements.

To achieve the above object, the data processing device of the present invention has a plurality of processing element groups each performing processing based on different two-dimensional input data and different filters, and a fourth storage unit having a three-dimensional storage area for each of the plurality of processing elements and for each of the plurality of filters, and after the repeated processing, the plurality of processing element groups calculate a three-dimensional total value by adding up the three-dimensional values stored in the first storage unit of each processing element group, and store the calculated three-dimensional total value in the fourth storage unit.

To achieve the above object, the data processing device of the present invention is characterized in that the group of multiple processing elements outputs the three-dimensional sum stored in the fourth storage unit after the process of calculating the three-dimensional sum.

In addition, the data processing method of the present invention for achieving the above object is a data processing method in a data processing device that calculates a feature amount of two-dimensional input data using a plurality of filters corresponding to the two-dimensional input data, the data processing method comprising: a processing element group consisting of a plurality of processing elements arranged two-dimensionally; and a first storage unit having a three-dimensional storage area for each of the plurality of processing elements and for each of the plurality of filters, the processing element group transmits each of the data included in the two-dimensional input data to one of the plurality of processing elements such that all of the transmission destinations are different processing elements, obtains one feature weight included in each filter for each of the plurality of filters, copies the obtained one feature weight for each of the plurality of filters by the same number as the plurality of processing elements and arranges them in the same manner as the plurality of processing elements to generate three-dimensional feature weights, and transmits the obtained one feature weight for each of the plurality of filters to each of the plurality of processing elements. the first storage unit, and causes each of the processing elements to calculate a three-dimensional product by multiplying the data included in the two-dimensional input data transmitted by the one feature weight for each of the multiple filters transmitted; the first storage unit, and the processing element, each of the processing elements, transmits the data included in the two-dimensional input data held by each of the multiple processing elements to an adjacent processing element so that data not yet used in the calculation of the three-dimensional product is transmitted to each of the multiple processing elements; and the process of acquiring, generating, transmitting the one feature weight for each of the multiple filters, calculating, adding, and transmitting the data included in the two-dimensional input data to an adjacent processing element is repeated until the number of times the adding process has been performed reaches the number of feature weights included in each of the multiple filters.

The data processing device and data processing method of the present invention make it possible to execute the convolution operations in the convolution layer in parallel, thereby enabling the convolution operations to be performed at high speed.

FIG. 1 is a diagram for explaining an outline of the system configuration of a CNN. FIG. 2 is a diagram for explaining a multiply-and-accumulate operation performed in a convolution operation. FIG. 3 is a diagram showing an example of a configuration of a data processing device 10 according to an embodiment of the present invention. FIG. 4 is a diagram showing an example of the configuration of a TPEn according to an embodiment of the present invention. FIG. 5 is a flow chart showing details of the process in TPEn. FIG. 6 is a flow chart showing details of the process in TPEn. FIG. 7 is a flow chart showing details of the process in TPEn. FIG. 8 is a flow chart showing details of the process in TPEn. FIG. 9 is a diagram for explaining a specific example of the processes from S16 to S22. FIG. 10 is a diagram for explaining a specific example of the processes from S16 to S22. FIG. 11 is a diagram for explaining a specific example of the processes from S16 to S22. FIG. 12 is a diagram for explaining a specific example of the processes from S16 to S22. FIG. 13 is a diagram for explaining a specific example of the processes from S16 to S22. FIG. 14 is a diagram for explaining a specific example of the processes from S16 to S22. FIG. 15 is a diagram for explaining a specific example of the processes from S16 to S22. FIG. 16 is a diagram for explaining the parallelism of the multiply-and-accumulate operation performed in the convolution operation.

Below, an embodiment of the present invention will be described with reference to the drawings. However, such an embodiment does not limit the technical scope of the present invention.

[Configuration of data processing device]
First, a description will be given of the configuration of the data processing device 10. Fig. 3 is a diagram showing an example of the configuration of the data processing device 10 according to the embodiment of the present invention.

In the example shown in FIG. 3, the data processing device 10 has a board 12 and a CPU (Central Processing Unit) 13. In the example shown in FIG. 3, the board 12 has a chip 14 and off-chip memory 15 (hereinafter also simply referred to as memory 15). That is, in the data processing device 10, for example, the CPU 13 and the chip 14 share and execute various processes.

The board 12 may be, for example, a Field-Programmable Gate Array (FPGA) board. The chip 14 may be, for example, an FPGA chip or an Application Specific Integrated Circuit (ASIC) chip.

Furthermore, in the example shown in FIG. 3, the chip 14 has a controller 17, an on-chip memory 16 (hereinafter also simply referred to as memory 16), and a soft tensor processor (STP) 20 which is a processor including m TPEs (tensor processing elements) TPE1 to TPEm. Note that hereinafter, TPE1 to TPEm are collectively referred to simply as TPEs.

In the example shown in FIG. 3, memory 16 includes buffer 21, which is an information storage area in which three-dimensional input data X that is the subject of the convolution operation is stored, buffer 22, which is an information storage area in which a filter used in the convolution operation is stored, and buffer 23 (hereinafter also referred to as the fourth storage unit), which is an information storage area in which the results of the convolution operation are stored.

In the example shown in FIG. 3, the controller 17 performs processing to cause each of the TPEs to perform a convolution operation, for example, by transmitting various pieces of information stored in buffer 21, buffer 22, and buffer 23 to each of the TPEs.

[Configuration of TPE]
Next, the configuration of the TPE will be described. Fig. 4 is a diagram showing an example of the configuration of the TPEn in this embodiment. Note that the other TPEs than the TPEn have the same configuration as the TPEn, so the description will be omitted.

In the example shown in FIG. 4, TPEn has Nox×Noy arithmetic units U each performing a convolution operation (multiply-and-accumulate operation), registers R1,1 to R1,Nof (hereinafter collectively referred to as register R1 or first storage unit) in which the results of the convolution operation in arithmetic unit U are stored, registers R2,1 to R2,Nof (hereinafter collectively referred to as register R2 or second storage unit) in which a portion of the filter stored in buffer 22 is stored, and register R3 (hereinafter referred to as third storage unit) in which a portion of the input data X stored in buffer 21 is stored.

Specifically, as shown in FIG. 4 , the controller 17 acquires the first feature weight ω ₁ ^{(1) included in the first type of filter among the Nof types of filters stored in the buffer 22, generates a two-dimensional feature weight ω 1 (1)} by duplicating and arranging the acquired feature weight ω ₁ ⁽¹⁾ Nox × Noy times (the same number as the calculator U), and stores the generated two-dimensional feature weight _{ω 1 (} ^{1 )} in the register R2,1.

Moreover, as shown in FIG. 4 , the controller 17 acquires the first feature weight ω ₂ ^{(1) included in the second type filter among the Nof types of filters stored in the buffer 22, generates a two-dimensional feature weight ω 2 (1)} by duplicating and arranging the acquired feature weight _{ω 2 (} ^{1) Nox} ×Noy times, and stores the generated two-dimensional feature weight ω ₂ ⁽¹⁾ in the register R2,2.

Furthermore, the controller 17 similarly generates two-dimensional feature weights for the first feature weights included in the other types of filters and stores them in register R2.

That is, the controller 17 generates two-dimensional feature weights (hereinafter, these are collectively referred to as three-dimensional feature weights Ω) corresponding to each of the Nof types of filters and stores them in the register R2.

In addition, as shown in FIG. 4, the controller 17 stores in register R3 the two-dimensional input data X corresponding to TPEn among the three-dimensional input data X stored in the buffer 21.

Then, the Nox×Noy calculators U generate Nof two-dimensional feature weights Ω by decomposing the three-dimensional feature weights Ω stored in the register R2. Then, the Nox×Noy calculators U multiply the two-dimensional input data X stored in the register R3 by each of the Nof two-dimensional feature weights Ω generated by the decomposition, thereby calculating Nof two-dimensional products (hereinafter, these are also collectively referred to as three-dimensional products).

That is, each of the Nox×Noy calculators U calculates 1×1×Nof products by multiplying the data corresponding to each calculator U among the two-dimensional input data X stored in register R3 by the feature weight corresponding to each calculator U among the Nof two-dimensional feature weights Ω generated by decomposition. Therefore, the Nox×Noy calculators U as a whole calculate Nox×Noy×Nof three-dimensional products.

Then, the Nox x Noy calculators U add the calculated three-dimensional products to the three-dimensional data Z stored in register R1 (the sum of the three-dimensional products already calculated).

Thereafter, the Nox×Noy computing units U repeat the above process a number of times (K ² times) corresponding to the number of feature weights included in each filter stored in the buffer 22. Details of the process in TPEn will be described below.

[Flowchart showing details of processing in TPE]
5 to 8 are flow charts showing details of the processing in TPEn.

As shown in FIG. 5, the controller 17 sets n to i, which is a variable that identifies each of the two-dimensional input data X included in the three-dimensional input data X (S01). That is, when TPEn is TPE1, the controller 17 sets i to 1. The controller 17 also sets k, which is a variable that identifies the feature weight included in each filter, to 1 (S02).

If n is 1 (YES in S03), the controller 17 obtains three-dimensional data corresponding to the bias from the buffer 22 and stores it in the register R1 (S04).

On the other hand, if n is not 1 (NO in S03), the controller 17 stores three-dimensional data Z, all of whose elements are 0 and whose data size is Nox x Noy x Nof, in the register R1 (S05). In other words, only when TPEn is 1, a multiply-and-accumulate operation is performed so that a bias is added.

Then, the controller 17 acquires the two-dimensional data X corresponding to the i-th channel from the three-dimensional input data X stored in the buffer 21 and stores it in the register R3 (S06).

Next, as shown in FIG. 6, the controller 17 acquires the feature weight of data size 1×1×Nof corresponding to k from among the feature weights included in the i-th channel of each filter stored in the buffer 22 (S11).

Then, the controller 17 generates a three-dimensional feature weight Ω of Nox x Noy x Nof by duplicating each feature weight obtained in the process of S11 into Nox x Noy copies, and stores the resulting weight in the register R2 (S12).

Then, the Nox×Noy calculators U acquire the three-dimensional data Z stored in the register R1 (S13). The Nox×Noy calculators U also acquire the three-dimensional data Ω stored in the register R2 (S14). Furthermore, the Nox×Noy calculators U acquire the two-dimensional data X stored in the register R3 (S15).

Next, the Nox x Noy calculators U decompose the Nox x Noy x Nof three-dimensional data Ω obtained in the process of S14 into Nof two-dimensional data (data with a data size of Nox x Noy), and add the three-dimensional data Z obtained in the process of S13 to the three-dimensional product of each of the decomposed two-dimensional data and the two-dimensional data X obtained in S15 (S16).

That is, each calculator U adds the corresponding data in the three-dimensional data Z to the product of the 1x1 data size data contained in the two-dimensional data decomposed from the three-dimensional data Ω and the 1x1 data size data contained in the data X.

Then, as shown in FIG. 7, the controller 17 updates the three-dimensional data Z stored in the register R1 to the three-dimensional data calculated in the process of S16 (S21).

Next, the controller 17 shifts the two-dimensional data X stored in the register R3 in a direction given by a predetermined method (S22). Specifically, the controller 17 determines the shift direction of the two-dimensional data X so that each data included in the two-dimensional data X circulates through all the feature weights included in each filter. A specific example of the process of S22 will be described later.

Then, the controller 17 adds 1 to k (S23). As a result, if k is not equal to or greater than ^K2 (NO in S24), the controller 17 repeats the process from S06 onwards.

On the other hand, if k is equal to or greater than ^K2 (YES in S24), the controller 17 adds m to i (S25). As a result, if i is not equal to or greater than Nif (NO in S26), the controller 17 repeats the process from S06 onwards. That is, if it is determined that all of the three-dimensional input data X stored in the buffer 21 has not been processed, the controller 17 repeats the process from S06 onwards. A specific example of the process from S16 to S22 will be described below.

[Specific example of processing from S16 to S22]
9 to 15 are diagrams for explaining a specific example of the process from S16 to S22. In the following, as shown in FIG. 9, the TPEn is assumed to have 4×4 calculators U (calculator U _1,1 , U _1,2 , U _1,3 , U _1,4 , U _2,1 , U _2,2 , U _2,3 , U _2,4 , U _3,1 , U _3,2 , U _3,3 , U _3,4 , U _4,1 , U _4,2 , U _4,3 and U _4,4 ). In the following, the number of feature weights (K ² ) included in each filter is assumed to be 9.

(Processing when k is 1)
First, the process when k is 1 will be described.

In this case, as shown in Fig. 10A, the controller 17 transmits (broadcasts) the decomposed two-dimensional feature weight _ω1 from the calculator _U1,1 to each of the calculators _U4,4 . Also, as shown in Fig. 10A, the controller 17 transmits data _X1,1 to _X4,4 included in the two-dimensional input data X stored in the register R3 from the calculator _U1,1 to each of the calculators _U4,4 .

Thereafter, each of the calculators _U1,1 to _U4,4 multiplies the feature weight _ω1 by the data X. Specifically, for example, the calculator _U2,2 calculates the product of the feature weight _ω1 and the data _X2,2 as shown in FIG.

Next, each of the calculators _U1,1 to _U4,4 adds the calculated product to the data corresponding to each calculator U among the three-dimensional data Z stored in the register R3. Specifically, for example, the calculator _U2,2 adds the product of the feature weight ω1 and the data X2,2 to the value corresponding to the calculator _U2,2 (0 when k is 1) among the three-dimensional data Z stored in the register R3, as shown in Step ₁ of FIG. _10B .

Then, the calculators _U1,1 to _U4,4 transmit the data X that they hold to the adjacent calculator U. Specifically, for example, the calculator _U2,2 transmits the data _X2,2 to the calculator _U2,1 as shown in FIG.

Here, when there is no other computing unit U in the transmission direction of the data X, each computing unit U may transmit the data X held by it to the computing unit U on the opposite side by using the trace network shown in Fig. 10(A) etc. Specifically, in the example shown in Fig. 10(A), the computing units _U1,1 , _U2,1 , _U3,1 and _U4,1 (the computing units U located at the left end) transmit the data held by them to the computing units _U1,4 , _U2,4 , _U3,4 and _U4,4 (the computing units U located at the right end), respectively.

This allows each computing unit U to prevent the data X held by each computing unit U from being lost due to a shift, even if there are no other computing units U in the transmission direction of the data X.

In this case, the calculators _U1,1 , _U2,1 , _U3,1 , and _U4,1 need to prevent the data X transmitted to the calculator U on the opposite side via the trace network from being used for product-sum calculation in the calculator U on the opposite side. For this reason, each of the calculators _U1,1 , _U2,1 , _U3,1 , and _U4,1 transmits the data X to the calculators _{U1,4, U2,4} , U3,4, and U4,4 after adding a flag indicating that the data is transmitted from the calculator U on the opposite side via the trace _network ( _a flag indicating that the data is not used for product _- sum calculation).

In this case, TPEn may further include a padding calculator U arranged outside the 4×4 calculators U such as calculator _U1,1 . In the example shown in Fig. 10A, calculators _U1,1 , _U2,1 , _U3,1 , and _U4,1 may transmit data held by each of them to a padding calculator U arranged to the left of each of the calculators U.

In this case, the number of operators U (including the operator U for padding) that TPEn has is calculated using the following formula (1).

（ｋが２の場合の処理）
次に、ｋが２の場合の処理について説明を行う。

(Processing when k is 2)
Next, the process when k is 2 will be described.

In this case, the controller 17 transmits (broadcasts) the two-dimensional feature weight _ω2 after decomposition to each of the calculators _U1,1 to _U4,4 as shown in Fig. 11A. Then, when each of the calculators _U1,1 to _U4,4 holds data X (when data X has been received from an adjacent calculator U in the immediately preceding process of S12), it multiplies the feature weight _ω2 by the data X. Specifically, for example, the calculator _U2,2 calculates the product of the feature weight _ω2 and data _X2,3 (data X transmitted from the calculator _U2,2 ) as shown in Fig. 11A.

Next, each of the calculators _U1,1 to _U4,4 adds the calculated product to a value of the three-dimensional data Z stored in the register R3 that corresponds to each calculator U. Specifically, for example, the calculator _U2,2 adds the product of the feature weight ω2 and the data X2,3 to a value of the three-dimensional data Z stored in the register R3 that corresponds to the calculator _U2,2 , as shown in Step _{2 of} FIG.

Then, the calculators _U1,1 to _U4,4 transmit the data X they hold to the adjacent calculators U. Specifically, for example, the calculator _U2,2 transmits data _X2,3 to the calculator _U3,2 as shown in FIG.

(Processing when k is 3)
Next, the process when k is 3 will be described.

In this case, the controller 17 transmits (broadcasts) the decomposed two-dimensional feature weight _ω3 to each of the calculators _U1,1 to _U4,4 as shown in Fig. 12(A). Then, when each of the calculators _U1,1 to _U4,4 holds data X, it multiplies the feature weight _ω3 by the data X. Specifically, for example, the calculator _U2,2 calculates the product of the feature weight _ω3 and the data _X1,3 as shown in Fig. 12(A).

Next, each of the calculators _U1,1 to _U4,4 adds the calculated product to a value of the three-dimensional data Z stored in the register R3 that corresponds to each calculator U. Specifically, for example, as shown in Step 3 of Fig. 12B, calculator _U2,2 adds the product of the feature weight _ω3 and the data _X1,3 to a value of the three-dimensional data Z stored in the register R3 that corresponds to calculator _U2,2 .

Then, the computing units _U1,1 to _U4,4 transmit the held data X to the adjacent computing units U. Specifically, for example, the computing unit _U2,2 transmits data _X1,3 to the computing unit _U2,3 .

Thereafter, for example, as shown in FIG. 13 and FIG. 14 , the calculator _U2,2 multiplies the feature weight _ω1 by the data _X2,2 (operation when k is 1), the feature weight _ω2 by the data _X2,3 (operation when k is 2), and the feature weight _ω3 by the data _X1,3 (operation when k is 3), and also multiplies the feature weight _ω4 by the data _X1,2 (operation when k is 4), the feature weight _ω5 by the data _X1,1 (operation when k is 5), the feature weight _ω6 by the data _X2,1 (operation when k is 6), the feature weight _ω7 by the data _X3,1 (operation when k is 7), the feature weight _ω8 by the data _X3,2 (operation when k is 8), and the feature weight _ω9 by the data _X3,3 (operation when k is 9).

In other words, each time the value of k is incremented, each calculator U broadcasts the feature weight and shifts the data X, while performing a convolution operation between the feature weight contained in each filter and the input data X, thereby performing the convolution operation in parallel for each calculator U.

When each calculator U multiplies a feature weight by data X, it performs multiplication of the feature weight included in each filter by the input data X in parallel for each Nof type of filter, as shown in FIG. 15.

Specifically, when k is 1, the calculator _U2,2 calculates the product Z2,2,1 by multiplying the first feature weight _ω1 ⁽¹⁾ included in the first type of filter by the data _X2,2 . In this case, the calculator _U2,2 calculates the product _Z2,2,2 by multiplying the first feature weight _ω2 ⁽¹⁾ included in the second type of filter by the data _X2,2 . Similarly, _{the calculator U2,2 performs} multiplications corresponding to each of the Nof types of filters.

This allows each calculator U to perform parallel processing with a degree of multiplicity that corresponds to the product of the number of calculators U (Nox x Noy) and the number of filter types (Nof).

Furthermore, in the data processing device 10 of this embodiment, the processing performed by each computing unit U as described above is performed in each TPE.

As a result, the data processing device 10 is able to perform parallel processing with a degree of multiplicity corresponding to the product of the number of arithmetic units U in each TPE (Nox x Noy), the number of filter types (Nof), and the number of TPEs (n). Therefore, the data processing device 10 is able to perform convolution operations in CNN at high speed.

Returning to FIG. 7, if i is equal to or greater than Nif (YES in S26), the controller 17 repeats the process from S27 onward. That is, upon completion of the convolution calculation in each TPE, the controller 17 starts the process of calculating the sum of the values calculated in each TPE.

For example, if TPEn is other than TPE1 (NO in S27), the controller 17 receives three-dimensional data Z' from the adjacent TPE(n-1) (S31), as shown in FIG. 8.

Next, the Nox×Noy calculators U add the three-dimensional data Z stored in the register R1 to the three-dimensional data Z' received in the process of S31 (S32).

Then, the Nox x Noy calculator U stores the three-dimensional data Z calculated in the process of S32 in register R1 (S33).

If TPEn is not TPEm (YES in S34), TPEn transmits the three-dimensional data Z stored in register R1 to the adjacent TPE(n+1) (S35). In addition, TPEn stores the three-dimensional data Z stored in register R1 in buffer 23 (S36).

This enables the data processing device 10 to calculate the sum of the results of the convolution operations calculated at each TPE.

Note that if TPEn is TPE1 (YES in S27), TPEn also performs the process of S35.

On the other hand, if TPEn is TPEm (NO in S34), TPEn stores the three-dimensional data Z stored in register R1 in buffer 23 without performing the process of S35 (S36). That is, in this case, controller 17 stores the final result of the convolution operation performed in each TPE in buffer 23.

In this way, the data processing device 10 in this embodiment is able to perform multiply-and-accumulate operations on Nof, Nox, Noy, and Nif in parallel. Therefore, as shown in FIG. 16, the data processing device 10 in this embodiment is able to increase the parallelism of the multiply-and-accumulate operations in the convolution operation compared to the case of using other methods.

Therefore, in the data processing device 10 of this embodiment, it is possible to significantly reduce the number of times that multiply-and-accumulate operations are performed. Therefore, in the data processing device 10 of this embodiment, it is possible to perform convolution operations in the convolution layer of the CNN at high speed.

In addition, for example, when it is required that the execution time of a convolution operation be equal to or less than a threshold value, the data processing device 10 of this embodiment makes it possible to reduce the number of times that a multiply-add operation is executed, and therefore allows the execution speed of each multiply-add operation to be set to be slow. In other words, this means that it becomes possible to reduce the operating frequency of the data processing device 10 and to suppress the drive voltage of the circuit. Therefore, the data processing device 10 of this embodiment makes it possible to reduce the power consumption associated with the execution of a convolution operation.

As a result, the data processing device 10 in this embodiment can be used in various fields where various constraints are required (for example, fields such as robot operation control and automatic driving of automobiles).

1: TPE
2: TPE
n: TPE
m: TPE
10: Data processing device 12: Board 13: CPU
14: Chip 15: Memory 16: Memory 17: Controller 20: STP
21: Buffer 22: Buffer 23: Buffer R1: Register R2: Register R3: Register U: Arithmetic unit

Claims (8)

A data processing device that calculates a feature amount of two-dimensional input data using a plurality of filters corresponding to the two-dimensional input data,
a processing element group consisting of a plurality of processing elements arranged two-dimensionally;
a first storage unit having a three-dimensional storage area for each of the plurality of filters for each of the plurality of processing elements,
The processing element group includes:
Sending each of the data included in the two-dimensional input data to one of the plurality of processing elements such that each of the sending destinations is a different processing element;
Obtaining one feature weight included in each of the plurality of filters;
generating three-dimensional feature weights by duplicating the one feature weight for each of the plurality of filters by the same number as the plurality of processing elements and arranging the duplicated feature weights in the same arrangement as the plurality of processing elements;
Transmitting the one feature weight for each of the plurality of filters to each of the plurality of processing elements;
causing each of the plurality of processing elements to multiply data included in the transmitted two-dimensional input data by the one feature weight for each of the transmitted plurality of filters to calculate a three-dimensional product;
adding the three-dimensional product to the value stored in the three-dimensional storage area in the first storage unit;
causing each of the plurality of processing elements to transmit data included in the two-dimensional input data held by each of the plurality of processing elements to an adjacent processing element such that data that has not yet been used in the calculation of the three-dimensional product is transmitted to each of the plurality of processing elements;
repeating the obtaining process, the generating process, the process of transmitting the one feature weight for each of the plurality of filters, the calculating process, the adding process, and the process of transmitting the data included in the two-dimensional input data to an adjacent processing element until the number of times the adding process has been performed reaches the number of feature weights included in each of the plurality of filters.
23. A data processing device comprising: In claim 1,
The processing elements perform processing in a convolutional layer of a convolutional neural network.
23. A data processing device comprising: In claim 1,
the processing element group outputs the three-dimensional values stored in the first storage unit after the repeated processing.
23. A data processing device comprising: In claim 1,
a second storage unit having a three-dimensional storage area for each of the plurality of filters for each of the plurality of processing elements;
the processing element group stores the generated three-dimensional feature weights in the second storage unit;
23. A data processing device comprising: In claim 1,
a third storage unit having a two-dimensional storage area for each of the plurality of processing elements;
the processing element group transmits each of the data included in the two-dimensional input data stored in the third storage unit to any of the plurality of processing elements such that each of the data destinations is a different processing element;
23. A data processing device comprising: In claim 1,
a plurality of processing element groups each performing processing based on the two-dimensional input data different from each other and the plurality of filters different from each other;
a fourth storage unit having a three-dimensional storage area for each of the plurality of filters for each of the plurality of processing elements,
The plurality of processing element groups include
After the repeated process, a three-dimensional sum is calculated by adding up the three-dimensional values stored in the first storage unit of each processing element group;
The calculated three-dimensional sum value is stored in the fourth storage unit.
23. A data processing device comprising: In claim 6,
the plurality of processing elements output the three-dimensional sum value stored in the fourth storage unit after the process of calculating the three-dimensional sum value.
23. A data processing device comprising: 1. A data processing method for a data processing device that calculates a feature amount of two-dimensional input data using a plurality of filters corresponding to the two-dimensional input data, comprising:
a processing element group consisting of a plurality of processing elements arranged two-dimensionally;
a first storage unit having a three-dimensional storage area for each of the plurality of filters for each of the plurality of processing elements,
The processing element group includes:
Sending each of the data included in the two-dimensional input data to one of the plurality of processing elements such that each of the sending destinations is a different processing element;
Obtaining one feature weight included in each of the plurality of filters;
generating three-dimensional feature weights by duplicating the one feature weight for each of the plurality of filters by the same number as the plurality of processing elements and arranging the duplicated feature weights in the same arrangement as the plurality of processing elements;
Transmitting the one feature weight for each of the plurality of filters to each of the plurality of processing elements;
causing each of the plurality of processing elements to multiply data included in the transmitted two-dimensional input data by the one feature weight for each of the transmitted plurality of filters to calculate a three-dimensional product;
adding the three-dimensional product to the value stored in the three-dimensional storage area in the first storage unit;
causing each of the plurality of processing elements to transmit data included in the two-dimensional input data held by each of the plurality of processing elements to an adjacent processing element such that data that has not yet been used in the calculation of the three-dimensional product is transmitted to each of the plurality of processing elements;
repeating the obtaining process, the generating process, the process of transmitting the one feature weight for each of the plurality of filters, the calculating process, the adding process, and the process of transmitting the data included in the two-dimensional input data to an adjacent processing element until the number of times the adding process has been performed reaches the number of feature weights included in each of the plurality of filters.
23. A data processing method comprising:

Images