JP7462140B2 - Neural network circuit and neural network operation method - Google Patents

Description

The present invention relates to a neural network circuit and a neural network calculation method.

In recent years, attention has been focused on edge computing, which processes data on computers closer to users (hereafter referred to as "edge computers"), rather than the conventional cloud computing, which consolidates data collected by multiple computers and processes it on a single computer. Furthermore, with recent advances in convolutional neural network technology, it has been proposed to run convolutional neural networks on edge computers.

JP 2018-142049 A JP 2018-160086 A JP 2018-116469 A JP 2017-21483 A JP 2019-57072 A

However, edge computers often do not have sufficient performance to run convolutional neural networks. For example, edge computers have the problem that they require a large number of memory accesses to run convolutional neural networks, resulting in slow calculation speeds.

In view of the above, the present invention aims to provide a technology for improving the calculation speed in convolutional neural networks.

One aspect of the present invention is a neural network circuit that executes processing of a convolutional layer or a fully connected layer in a convolutional neural network having a fully connected layer and multiple convolutional layers, the neural network circuit comprising: a first storage unit that stores activation data, which is the value of each element of data input to each layer and expressed as a tensor; a second storage unit that stores weight data for processing executed in the convolutional layer or the fully connected layer; and multiple calculation units that read one of the activation data from the first storage unit at a predetermined timing and obtain a term value that is the value of a term of the product of the read activation data and one of the weight data that satisfies a predetermined condition related to the weight data among the weight data, and the activation data read by the calculation unit is output to another of the calculation units that is previously associated with the activation data after the calculation unit calculates the term value.

One aspect of the present invention is the above-mentioned neural network circuit, in which the processing of the convolution layer is a convolution integral, and the weight data when the convolution integral is performed are filter coefficients, which are the values of a filter for performing the convolution integral.

One aspect of the present invention is the above-mentioned neural network circuit, in which the weight data when the processing of the fully connected layer is executed is a weight coefficient used in the processing of the fully connected layer.

One aspect of the present invention is a neural network circuit that executes processing of a convolutional layer or a fully connected layer in a convolutional neural network having a fully connected layer and multiple convolutional layers, the neural network circuit comprising: a first storage unit that stores activation data, which is the value of each element of data that is input to each layer and is expressed as a tensor; a second storage unit that stores weight data for processing executed in the convolutional layer or the fully connected layer; and multiple calculation units that read out one of the activation data from the first storage unit at a predetermined timing and obtain a term value that is the value of a term of the product of the read out activation data and one of the weight data that satisfies a predetermined condition related to the weight data. The activation data read by the calculation unit is output to another calculation unit associated with the activation data after the calculation unit calculates the term value. The neural network calculation method includes an acquisition step in which the calculation unit reads one of the activation data from the first storage unit at a predetermined timing and acquires a term value that is the value of a term of a product of the read activation data and one of the weight data that satisfies a predetermined condition related to the weight data, and an output step in which the activation data read by the calculation unit is output to the other calculation unit associated with the activation data after the calculation unit calculates the term value.

This invention makes it possible to improve the calculation speed in convolutional neural networks.

FIG. 2 is a diagram showing an example of a functional configuration diagram of the neural network circuit 1 according to the embodiment. FIG. 4 is an explanatory diagram illustrating an example of an information flow in a unit processing unit 100 of the embodiment. FIG. 4 is an explanatory diagram illustrating an example of an information flow between unit processing sections 100 of the embodiment. FIG. 4 is an explanatory diagram illustrating an example of a flow of configuration data between unit processing units 100 of the embodiment. FIG. 4 is an explanatory diagram illustrating an example of a processing flow in a product-sum calculation unit 102 according to the embodiment. FIG. 2 is a first explanatory diagram illustrating a convolution process executed by the neural network circuit 1 according to the embodiment. FIG. 2 is a second explanatory diagram illustrating the convolution operation executed by the neural network circuit 1 according to the embodiment. 5 is a diagram for explaining the flow of information output from a first buffer memory unit 11 to each unit processing unit 100 in the embodiment. 5 is a diagram for explaining the flow of information output from the second buffer memory unit 12 to each unit processing unit 100 in the embodiment. FIG. 5 is a diagram for explaining the flow of information output from the unit processing unit 100 to a first buffer memory unit 11 in the embodiment. 5 is a diagram for explaining the flow of information output from the unit processing unit 100 to a second buffer memory unit 12 in the embodiment. FIG. 5A and 5B are diagrams for explaining destinations of information output from a first buffer memory unit 11 to a main processing unit 10 according to an embodiment. 4 is a diagram for explaining the source of information output from the main processing unit 10 to a first buffer memory unit 11 according to the embodiment. FIG. FIG. 4 is an explanatory diagram for explaining a first process of the convolution process in the embodiment. FIG. 11 is an explanatory diagram for explaining a second process of the convolution process in the embodiment. FIG. 11 is an explanatory diagram for explaining a third process of the convolution process in the embodiment. FIG. 11 is an explanatory diagram for explaining a fourth process of the convolution process in the embodiment. FIG. 11 is an explanatory diagram for explaining a fifth process of the convolution process in the embodiment. FIG. 11 is an explanatory diagram for explaining a sixth process of the convolution process in the embodiment. FIG. 11 is an explanatory diagram for explaining a seventh process of the convolution process in the embodiment. 11 is a flowchart illustrating an example of the flow of a convolution process according to the embodiment. FIG. 4 is an explanatory diagram illustrating an overview of a full join process according to an embodiment. FIG. 4 is a first explanatory diagram illustrating an example of a data flow between the unit processing units 100 when the all-join process of the embodiment is executed. FIG. 11 is a second explanatory diagram illustrating an example of the data flow between the unit processing units 100 when the all-join process of the embodiment is executed. FIG. 11 is an explanatory diagram illustrating an example of a first process of a full joining process according to the embodiment. FIG. 11 is a first explanatory diagram illustrating an example of a second process of the full joining process according to the embodiment. FIG. 11 is a second explanatory diagram illustrating an example of the second process of the full joining process in the embodiment. FIG. 11 is an explanatory diagram illustrating an example of a third process of the full joining process according to the embodiment. FIG. 11 is an explanatory diagram illustrating an example of a fourth process of the full joining process according to the embodiment. FIG. 11 is an explanatory diagram illustrating an example of a fifth process of the full joining process according to the embodiment. FIG. 11 is an explanatory diagram illustrating an example of a sixth process of the full joining process according to the embodiment. 11 is a flowchart showing an example of the flow of a full join process in the embodiment.

The neural network circuit and neural network calculation method of the embodiment will be described below with reference to the drawings. In the following description, connection means electrical connection. In the following description, the same circuit means that the relative positions between circuit elements are the same. Relative position means the position of another circuit element as viewed from a reference circuit element. The same circuit does not necessarily mean that the connection relationships between circuit elements are the same.

FIG. 1 is a diagram showing an example of a functional configuration diagram of a neural network circuit 1 according to an embodiment. The neural network circuit 1 acquires data to be processed by a convolutional neural network having a fully connected layer from a first external memory 91, a second external memory 92, and a control external memory 93 connected to the neural network circuit 1. The first external memory 91 and the second external memory 92 are configured using a storage device such as a semiconductor storage device. The first external memory 91 and the second external memory 92 are, for example, a DRAM (Dynamic Random Access Memory). The control external memory 93 is configured using a storage device such as a semiconductor storage device. The control external memory 93 is, for example, a flash memory.

The neural network circuit 1 executes convolution integrals in each convolution layer of the convolution neural network based on the acquired data. The neural network circuit 1 also executes processing in the fully connected layer of the convolution neural network, which has a fully connected layer, based on the acquired data. Specifically, the processing in the fully connected layer is a product-sum operation.

The neural network circuit 1 includes a first buffer memory unit 11, a second buffer memory unit 12, a control buffer memory unit 13, a control unit 14, and a main calculation unit 10.

The first buffer memory unit 11 is constructed using a temporary recording medium from among memory devices such as semiconductor memory devices. The first buffer memory unit 11 is connected to the first external memory 91, the main calculation unit 10, and the control unit 14. The first buffer memory unit 11 temporarily stores data output by the first external memory 91. The first buffer memory unit 11 temporarily stores data output by the main calculation unit 10. The first buffer memory unit 11 operates under the control of the control unit 14. The first buffer memory unit 11 outputs the stored data to the main calculation unit 10 or the first external memory 91 at a predetermined timing.

The first buffer memory unit 11 stores, for example, activation data. The activation data is the value of each element of data (hereinafter referred to as "layer input data") that is input to each layer of a convolutional neural network and is expressed as a tensor. The layer input data may be expressed as a 0th-order tensor (i.e., a scalar), a 1st-order tensor (i.e., a vector), or a 2nd-order or higher-order tensor. For example, when the number of channels is v1 and the data of each channel is expressed as a matrix with v2 rows and v3 columns, the layer input data is data expressed as a 3rd-order tensor.

The first buffer memory unit 11 stores, for example, configuration data. The configuration data is data that indicates the connection relationships between multiple unit processing units 100 provided in the main processing unit 10, the details of which will be described later. A unit processing unit 100 may be connected to one other unit processing unit 100, or may be connected to multiple other unit processing units 100.

The second buffer memory unit 12 is configured using a temporary recording medium from among memory devices such as semiconductor memory devices. The second buffer memory unit 12 is connected to the second external memory 92, the main processing unit 10, and the control unit 14. The second buffer memory unit 12 temporarily stores data output by the second external memory 92. The second buffer memory unit 12 operates under the control of the control unit 14.

The second buffer memory unit 12 outputs the stored data to the main calculation unit 10 at a predetermined timing. The second buffer memory unit 12 stores, for example, the values of each element (hereinafter referred to as "filter coefficients") of one or more filters (hereinafter referred to as "convolution filters") in each layer for performing convolution. The second buffer memory unit 12 also stores, for example, the weights (hereinafter referred to as "fully connected coefficients") in the processing of the fully connected layer.

The control buffer memory unit 13 is configured using a temporary recording medium from among memory devices such as semiconductor memory devices. The control buffer memory unit 13 is connected to the control external memory 93 and the control unit 14. The control buffer memory unit 13 temporarily stores data output by the control external memory 93. The control buffer memory unit 13 outputs the stored data to the control unit 14 at a predetermined timing. The control buffer memory unit 13 stores control commands such as microcode, for example.

The control unit 14 controls the operation of each functional unit included in the neural network circuit 1. Specifically, the control unit 14 includes a processor such as a CPU (Central Processing Unit) and a memory connected by a bus, and executes a program. The control unit 14 controls the operation of each functional unit included in the neural network circuit 1 by executing the program. For example, the control unit 14 controls the operation of the first buffer memory unit 11, the second buffer memory unit 12, and the main calculation unit 10 to control the execution of a convolution process described later. For example, the control unit 14 controls the operation of the first buffer memory unit 11, the second buffer memory unit 12, and the main calculation unit 10 to control the execution of a full connection process described later. For example, the control unit 14 controls the timing of inputting configuration data from the first buffer memory unit 11 to the main calculation unit 10. For example, the control unit 14 has a function of selecting the configuration data to be input from the first buffer memory unit 11 to the main calculation unit 10 according to the timing of input. For example, the control unit 14 controls the operation of a connection unit 101 described later that changes the connection destination based on the configuration data.

The main calculation unit 10 executes the convolution integral of each convolution layer of the convolutional neural network. The main calculation unit 10 also executes processing in the fully connected layer of the convolutional neural network that has a fully connected layer.

The main processing unit 10 has a plurality of unit processing units 100 arranged in a lattice pattern. Each unit processing unit 100 has a connection unit 101 and a product-sum calculation unit 102. The connection unit 101 is an interface for connecting the product-sum calculation unit 102 to other unit processing units 100 that are associated in advance, the first buffer memory unit 11, and the second buffer memory unit 12. The other unit processing units 100 that are associated in advance with the connection unit 101 are other unit processing units 100 indicated by the configuration data.

The connection unit 101 changes the destination unit processing unit 100 based on the configuration data at a predetermined timing. More specifically, the connection unit 101 changes the wiring so as to connect to the destination indicated by the configuration data at a predetermined timing. The method of changing the wiring is similar to that of an FPGA (field-programmable gate array). Specifically, the method similar to that of an FPGA is a method of downloading configuration data from the outside to the configuration memory. The predetermined timing is, for example, the start of operation of the neural network circuit 1. The predetermined timing may be, for example, a timing after the processing of each convolution layer of the convolutional neural network is completed and before the processing by the fully connected layer is started.

The product-sum calculation unit 102 executes a product-sum calculation. The product-sum calculation unit 102 acquires activation based on the result of the product-sum calculation. The product-sum calculation unit 102 inputs the acquired activation to a predetermined activation function and acquires the result of the calculation using the activation function (hereinafter referred to as the "activation result"). The activation function may be a step function, a sigmoid function, a ReLU function, an identity function, or a softmax function. The activation result is input to the next layer, for example, as activation data.

Each unit processing unit 100 is connected to the first buffer memory unit 11 via the connection unit 101 by a conductor dedicated to each unit processing unit 100. Also, each unit processing unit 100 is connected to the second buffer memory unit 12 via the connection unit 101 by a conductor dedicated to each unit processing unit 100. "Dedicated" means that it is not used to send or receive information between the first buffer memory unit 11 or the second buffer memory unit 12 and other unit processing units 100.

FIG. 2 is an explanatory diagram illustrating an example of the flow of information in the unit processing unit 100 of the embodiment. The connection unit 101 includes an input control circuit 111, a weight control circuit 112, and an output control circuit 113. The product-sum calculation unit 102 includes a product-sum calculation circuit 201 and an activation circuit 202.

The input control circuit 111 acquires configuration data from the first buffer memory unit 11. Based on the acquired configuration data, the input control circuit 111 connects to the connection destination indicated by the acquired configuration data. The input control circuit 111 acquires the activation data output by the first buffer memory unit 11. The input control circuit 111 acquires the activation data output by the connected unit processing unit 100.

The input control circuit 111 outputs activation data to the connected unit processing unit 100. The input control circuit 111 outputs configuration data to the connected unit processing unit 100. The connected unit processing unit 100 is the connection destination indicated by the configuration data connected based on the configuration data. The input control circuit 111 includes a register 114. The register 114 stores the acquired configuration data and activation data.

The weight control circuit 112 acquires the filter coefficients and all the coupling coefficients from the second buffer memory unit 12. The weight control circuit 112 includes a register 115. The register 115 stores the acquired filter coefficients and all the coupling coefficients.

The output control circuit 113 is connected to the sum-of-products calculation unit 102. The output control circuit 113 acquires the activation result, which is the calculation result of the sum-of-products calculation unit 102. The output control circuit 113 outputs the activation result to the first buffer memory unit 11. The output control circuit 113 includes a register 116. The register 116 stores the acquired activation result. The output control circuit 113 includes a conductor (hereinafter referred to as the "feedback conductor 118") for inputting the activation result to the input control circuit 111. The feedback conductor 118 is used when the unit processing unit 100 feeds back the activation result to itself without going through an external memory. More specifically, when the unit processing unit 100 feeds back the activation result to itself without going through an external memory, the activation result output by the output control circuit 1 is input to the input control circuit 111 via the feedback conductor 118 under the control of the control unit 14.

The sum-of-products calculation circuit 201 is a circuit connected to the input control circuit 111, the weight control circuit 112, and the activation circuit 202. The sum-of-products calculation circuit 201 acquires activation data from the first buffer memory unit 11 or another connected unit processing unit 100 via the input control circuit 111. The sum-of-products calculation circuit 201 acquires filter coefficients or full coupling coefficients from the second buffer memory unit 12 via the weight control circuit 112. The sum-of-products calculation circuit 201 executes sum-of-products calculation based on the acquired activation data and the filter coefficients or full coupling coefficients to acquire activity. The sum-of-products calculation circuit 201 includes a register 117. The register 117 stores intermediate values that are the results of the sum-of-products calculation. The register 117 may store activity. The sum-of-products calculation circuit 201 outputs the acquired activity to the activation circuit 202.

The activation circuit 202 acquires the output (i.e., activity) of the product-sum calculation circuit 201. The activation circuit 202 inputs the acquired activity to a predetermined activation function. The activation circuit 202 acquires an activity result that is the result of the calculation using the activation function. The activation circuit 202 outputs the acquired activity result to the output control circuit 113.

FIG. 3 is an explanatory diagram illustrating an example of the flow of information between the unit processing units 100 of the embodiment.
3 shows the flow of information between five unit processing units 100, 100-1, 100-2, 100-3, 100-4, and 100-5, as a specific example for explaining the flow of information between the unit processing units 100. The unit processing unit 100-1 is connected to four other unit processing units 100, 100-2, 100-3, 100-4, and 100-5. The unit processing unit 100-1 outputs information to the unit processing unit 100-2, which is one of the other unit processing units 100 to which it is connected. The information is, for example, activation data. The unit processing unit 100-1 acquires information from the unit processing unit 100-2, which is one of the other unit processing units 100 to which it is connected.

The unit processing unit 100-1 outputs information to the unit processing unit 100-3, which is one of the other unit processing units 100 to which it is connected. The unit processing unit 100-1 acquires information from the unit processing unit 100-3, which is one of the other unit processing units 100 to which it is connected. The unit processing unit 100-1 outputs information to the unit processing unit 100-4, which is one of the other unit processing units 100 to which it is connected. The unit processing unit 100-1 acquires information from the unit processing unit 100-4, which is one of the other unit processing units 100 to which it is connected. The unit processing unit 100-1 outputs information to the unit processing unit 100-5, which is one of the other unit processing units 100 to which it is connected. The unit processing unit 100-1 acquires information from the unit processing unit 100-5, which is one of the other unit processing units 100 to which it is connected.

FIG. 4 is an explanatory diagram illustrating an example of the flow of configuration data between the unit processing units 100 of the embodiment.
The configuration data input to the unit processing unit 100 is stored in the register 114. Based on the configuration data stored in the register 114, the unit processing unit 100 is connected to another unit processing unit 100 indicated by the configuration data. The configuration data is output to the other unit processing unit 100 as the connection destination.

FIG. 5 is an explanatory diagram illustrating an example of a processing flow in the product-sum calculation unit 102 according to the embodiment.
The product-sum operation unit 102 executes the product-sum operation. In Fig. 5, x represents activation data, and w represents a filter coefficient or a full connection coefficient. The intermediate value of the product-sum operation is stored in the register 117. The activation (u in Fig. 5) which is the result of the product-sum operation is input to the activation function. In Fig. 5, z represents the activation result corresponding to the activation u. For simplicity of explanation, the neural network circuit 1 will be explained below taking as an example a case where the activation function is an identity function with a value of 1, and the activation result z is the same as the activation u.

<Processing of convolution integrals in the convolution layer>
The convolution integral process of the convolution layer (hereinafter referred to as "convolution process") executed by the main calculation unit 10 will be described with reference to Figs. 6 to 21.

FIG. 6 is a first explanatory diagram illustrating the convolution process executed by the neural network circuit 1 of the embodiment.
In FIG. 6, for the sake of simplicity, the convolution process is described using nine elements from the value x(i, j) of the element in row i and column j to the value x(i+2, j+2) of the element in row (j+2) and column (j+2) of the layer input data D101 represented by a second-order tensor with row I and column J. I is an integer of 1 or more, and J is an integer of 1 or more. i is an integer of 0 or more (I-1) and j is an integer of 0 or more (J-1). The number of channels of the layer input data D101 is 1. Hereinafter, A of the activation data x(A,B) is referred to as the first index of the two-index display of the activation data x. Hereinafter, B of the activation data x(A,B) is referred to as the second index of the two-index display of the activation data x. The two-index display of the activation data x means that the activation data x is a value that is distinguished from other activation data x by two indexes. In the following explanation, in addition to the above-mentioned A and B, the symbols C, D, and E will also be used. Like A and B, C, D, and E are also symbols that indicate indicators for distinguishing one thing from another.

As described above, each layer may have more than one convolution filter. In FIG. 6, the convolution filter F101 is shown as a specific example of one of the multiple convolution filters. The convolution filter F101 is a convolution filter with two rows and two columns and four filter coefficients. Specifically, the convolution filter F101 has four filter coefficients: w(0,0), w(0,1), w(1,0), and w(1,1). For ease of explanation, the neural network circuit 1 will be described below using an example in which the convolution filter is expressed as a second-order tensor, particularly as a square matrix. The filter coefficient w(A,B) indicates that the filter coefficient w is the value of the element in the Ath row and Bth column of the convolution filter. Hereinafter, the A of the filter coefficient w(A,B) will be referred to as the first index of the two-index display of the filter coefficient w. Hereinafter, the B of the filter coefficient w(A,B) will be referred to as the second index of the two-index display of the filter coefficient w.

Figure 6(a) shows that a feature map with two rows and two columns is obtained by performing a convolution integral between the layer input data D101 and the convolution filter F101. The value of each element of the feature map is an activity. Specifically, Figure 6(a) shows that a feature map having activation u(i, j), activation u(i, j+1), activation u(i+1, j), and activation u(i+1, j+1) is calculated as a result of the convolution integral between the layer input data D101 and the convolution filter F101. Hereinafter, A in activation u(A, B) is referred to as the first index of the two-index display of activation u. Hereinafter, B in activation u(A, B) is referred to as the second index of the two-index display of activation u. Note that the two-index display of activation u means that the activation u is a value that is distinguished from other activation u by two indexes.

Specifically, activity u(i, j) is expressed by the following formula (1):

The definitions of P and Q in formula (1) are explained below. When the convolution filter is a tensor with P rows and Q columns, p is an integer greater than or equal to 0 and less than P. Note that when the convolution filter is a tensor with P rows and Q columns, q is an integer greater than or equal to 0 and less than Q. For example, when the convolution filter is a tensor with 2 rows and 2 columns, q is a value of 0 or 1, respectively.

The feature map obtained by processing in the nth convolutional layer (hereinafter referred to as the "nth convolutional layer") is input to the next convolutional layer as layer input data for the next stage (i.e., the (n+1)th convolutional layer). n is a number for distinguishing between layers of a convolutional neural network, and the earlier the layer in the convolutional neural network is processed, the smaller the number is. n is an integer between 0 and (N-1), and N is an integer greater than or equal to 2. Hereinafter, n is referred to as the layer number. For example, the layer that is processed next after the nth convolutional layer is the (n+1)th convolutional layer. If the next layer is a fully connected layer, the feature map is input to the fully connected layer as layer input data for the fully connected layer.

Figure 6(b) illustrates an example of the process flow for calculating activation u(i,j), which is the value of one element of a feature map, while showing the elements of the layer input data D101 used in the calculation of each term of the product-sum operation that calculates activation u(i,j). Since the convolution filter F101 is a tensor with 2 rows and 2 columns, activation u(i,j) is expressed as the sum of four terms, from the first term to the fourth term. In Figure 6(b), the process for calculating activation u(i,j) is explained using an example in which the first term is calculated first, then the second term, then the third term, and finally the fourth term.

Figure 6(b) shows that the calculation of the first term uses four activation data from the layer input data D101: activation data x(i, j), activation data x(i, j+1), activation data x(i+1, j), and activation data x(i+1, j+1). Figure 6(b) shows that the calculation of the second term uses four activation data from the layer input data D101: activation data x(i, j+1), activation data x(i, j+2), activation data x(i+1, j+1), and activation data x(i+1, j+2). FIG. 6(b) shows that the calculation of the third term uses four activation data from the layer input data D101: activation data x(i+1, j), activation data x(i+1, j+1), activation data x(i+2, j), and activation data x(i+2, j+1). FIG. 6(b) shows that the calculation of the fourth term uses four activation data from the layer input data D101: activation data x(i+1, j+1), activation data x(i+1, j+2), activation data x(i+2, j+1), and activation data x(i+2, j+2).

For ease of explanation, FIG. 6(b) shows that the calculations are performed in the order of the first term, the second term, the third term, and the fourth term, but the calculations do not necessarily have to be performed in the order of the first term, the second term, the third term, and the fourth term. The calculations may be performed in the order of the first term, the third term, the second term, and the fourth term, for example.

FIG. 7 is a second explanatory diagram illustrating the convolution operation executed by the neural network circuit 1 of the embodiment.
FIG. 7 illustrates a convolution operation using an example in which the layer input data to the nth convolution layer has three channels and the data of each channel is represented by a matrix of four rows and four columns. In FIG. 7, the layer input data to the nth convolution layer has three channels and the data of each channel is represented by a matrix of four rows and four columns, so the data is represented by a third-order tensor. Hereinafter, the number of channels of the layer input data (hereinafter referred to as the "number of input channels") is represented by the symbol NumCH _in . Hereinafter, the number of channels of the feature map (hereinafter referred to as the "number of output channels") is represented by the symbol NumCH _out . The number of output channels NumCH _out is equal to the number of convolution filters.

Figure 7 shows that two convolution filters are applied, one for each channel of the layer input data.

FIG. 7 shows that when two convolution filters, Filter 1 and Filter 2, each having 3 rows and 3 columns, are applied to the layer input data, the feature map has an output channel number NumCH _out of 2 and data for each channel is represented by a matrix having 2 rows and 2 columns.

In the neural network circuit 1, multiple convolution filters are applied to each channel of the layer input data, so the activation u calculated by the neural network circuit 1 is expressed by the following equation (2).

The following describes CH _out and CH _in , and the indices x, u, and w in formula (2).

Ch _out in formula (2) is a value for distinguishing each channel of the feature map. For example, Ch _out =V (V is a non-negative integer) represents the Vth channel among the multiple channels included in the feature map. Hereinafter, CH _out will be referred to as the output channel number.

Ch _in in formula (2) is a value for distinguishing each channel of the layer input data. For example, Ch _in = V represents the Vth channel (V is a non-negative integer) among the multiple channels included in the layer input data. Hereinafter, Ch _in (i.e., V) is referred to as the input channel number.

The activation data x in formula (2) is distinguished from other activation data x by four indices. Specifically, the activation data x is distinguished from other activation data by being represented as x(A, B, C, D). The activation data x(A, B, C, D) indicates that the activation data x is the value of an element of the layer input data input to the layer with layer number A, and is the value of the element in row C and column D of the data of the input channel number B. That is, the first index A of the four-indicator display of the activation data x represents the layer number n of the input destination of the layer input data having the activation data x. The second index B of the four-indicator display of the activation data x represents the input channel number Ch _in . In addition, the third index C and the fourth index D of the four-indicator display of the activation data x indicate that the activation data x is the value of the element located in row C and column D of the second-order tensor. The above-mentioned activation data x (A, B) represents the activation data x (A, B, C, D) when the number of input channels and the number of output channels are 1. The four-index representation of the activation data x means that the activation data x is a value that can be distinguished from other activation data x by four indexes.

The activation u (A, B, C, D) indicates that the activation u is the result of an operation in a layer with layer number A, and is the value of an element in row C and column D of the data of output channel number B in the feature map. That is, the first index A of the four-index display of the activation u represents the layer number n of the layer in which the operation to output the activation u was executed. The second index B of the four-index display of the activation u represents the output channel number Ch _out . In addition, the third index C and the fourth index D of the four-index display of the activation u indicate that the activation u is a value stored in the first buffer memory unit 11 as the value of an element located in row C and column D of a second-order tensor. Note that the above-mentioned activation u (A, B) represents the activation u (A, B, C, D) when the number of input channels and the number of output channels are 1. Note that the two-index display of the activation u means that the activation u is a value that is distinguished from other activation u by four indexes.

w (A, B, C, D, E) represents a filter coefficient. The first index A of the five-index display of the filter coefficient w represents the layer number n where the convolution integral operation is performed. The second index B of the five-index display of the filter coefficient w represents the input channel number Ch _in to which the activation data x to be multiplied by the filter coefficient w when the convolution integral is performed belongs. The third index C of the five-index display of the filter coefficient w represents the output channel number CH _out in the feature map which is the result of the convolution integral. The fourth index D and the fifth index E of the five-index display of the filter coefficient w indicate that the filter coefficient w is the value of the element located in the D row and E column of the second-order tensor representing the convolution filter. The five-index display of the filter coefficient w means that the filter coefficient w is a value that is distinguished from other filter coefficients w by five indexes.

b(n, CH _out ) is a bias. The bias b(n, CH _out ) is a bias used in the calculation in the layer with layer number u, and indicates that it is a bias used in the calculation to calculate the activity u of the output channel number CH _out . Note that the bias b may be 0, and is 0 in the convolution process. Therefore, for simplicity of explanation, the convolution process in the neural network circuit 1 will be explained below using the case where the bias b is 0 as an example.

Figure 8 is a diagram illustrating the flow of information output from the first buffer memory unit 11 to each unit processing unit 100 in the embodiment.

FIG. 8 shows (R×S) (R is an integer equal to or greater than CH _in , and S is an integer equal to or greater than J) unit processing units 100 from unit processing unit 100_0_0 to unit processing unit 100_(R_1)_(S_1). R is preferably equal to the number of input channels NumCH _in . S is preferably equal to the number of columns J of the layer input data. Hereinafter, for ease of explanation, unit processing unit 100_(r_1)_(s_1) will be referred to as PB(r-1, s-1). Note that r is an integer equal to or greater than 0 (R-1), and s is an integer equal to or greater than 0 (S-1). Hereinafter, A in PB(A,B) will be referred to as the first index of PB. B in PB(A,B) will be referred to as the second index of PB. The first index of PB indicates the column to which the unit processing units 100 arranged in a lattice pattern belong. The second index of PB indicates the row to which the unit processing units 100 arranged in a lattice pattern belong. Two unit processing units 100 having the same PB second index and a difference between their PB first indexes of 1 are adjacent to each other. Two unit processing units 100 having the same PB first index and a difference between their PB second indexes of 1 are adjacent to each other.

The first buffer memory unit 11 can transmit information directly to each unit processing unit 100. Direct transmission means that there is no need to transmit information to other unit processing units 100 via the unit processing unit 100.

FIG. 9 is a diagram illustrating the flow of information output from the second buffer memory unit 12 to each unit processing unit 100 in the embodiment.
The second buffer memory unit 12 is capable of directly transmitting information to each unit processing unit 100 .

FIG. 10 is a diagram illustrating the flow of information output from the unit processing unit 100 to the first buffer memory unit 11 in the embodiment.
The first buffer memory unit 11 can directly receive information from each unit processing unit 100. Being directly receivable means that it is not necessary to receive information transmitted by another unit processing unit 100 via the unit processing unit 100.

FIG. 11 is a diagram illustrating the flow of information output from the unit processing unit 100 to the second buffer memory unit 12 in the embodiment.
The processing unit 100 is capable of transmitting information to another predetermined processing unit 100. The other predetermined processing unit 100 is a connection destination processing unit 100 indicated by the configuration data.

Figure 11 shows that information can be transmitted from PB(0,0) to PB(1,0). Figure 11 shows that information can be transmitted from PB(1,0) to PB(2,0). Figure 11 shows that information can be transmitted from PB(S-1,0) to PB(0,0). In this way, each unit processing unit 100 other than the unit processing unit 100 whose PB first index is (S-1) can transmit information to the unit processing unit 100 whose PB second index is the same and whose first index increases by 1 in the convolution operation. The unit processing unit 100 whose PB first index is (S-1) can transmit information to the unit processing unit 100 whose PB second index is the same and whose PB first index is 0.

12 is a diagram for explaining the output destination of information output from the first buffer memory unit 11 to the main calculation unit 10 in the embodiment. In the following, for the sake of simplicity, the neural network circuit 1 will be explained using an example in which the layer input data is data with an input channel number NumCh _in of 64, and the data possessed by each channel is data expressed by a second-order tensor with 64 rows and 64 columns. However, in the following explanation, the number of input channels does not have to be 64. Furthermore, the data possessed by each channel does not have to be 64 rows and 64 columns as long as it is data expressed by a second-order tensor. Furthermore, the data possessed by each channel does not have to be a square matrix as long as it is data expressed by a second-order tensor.

Figure 12 shows that the layer input data is data for 64 channels with input channel numbers from 0 to 63. Figure 12 also shows that the data for each channel stored in the first buffer memory unit 11 is data represented as a 64-row, 64-column tensor, with 64 rows from row 0 to row 63 and 64 columns from column 0 to column 63.

Figure 12 shows that the data in the 0th row of each channel of the layer input data is output to the main calculation unit 10. It shows that one activation data is input to the unit processing unit 100 of the main calculation unit 10. For example, activation data x(0,0,0,0) in the 0th row and 0th column of the data with input channel number 0 is input to PB(0,0).

Figure 12 shows that layer input data is input to the main calculation unit 10 according to the following first activation data input rule and second activation data input rule. The first activation data input rule is a rule that activation data x with the same input channel number is input to unit processing units 100 with the same PB first index. The second activation data input rule is a rule that activation data x with different second indexes and the same third index are input to unit processing units 100 with the same PB second index.

FIG. 13 is a diagram for explaining the source of information output from the main processing unit 10 to the first buffer memory unit 11 in the embodiment.
For ease of explanation, the neural network circuit 1 will be described below using as an example a case where the feature map has an output channel number NumCh _out of 64, and the data held by each channel is data expressed by a second-order tensor with 64 rows and 64 columns. However, in the following explanation, the number of output channels does not have to be 64. Furthermore, the data held by each channel does not have to be 64 rows and 64 columns as long as it is data expressed by a second-order tensor. Furthermore, the data held by each channel does not have to be a square matrix as long as it is data expressed by a second-order tensor.

Figure 13 shows that the activation u (i.e., activation result z) that is the result of the calculation of each unit processing unit 100 of the main calculation unit 10 is stored in the first buffer memory unit 11 as the value of each element in the 0th row of each channel in the feature map. For example, u(0,0,0,0), which is the result of the calculation by PB(0,0), is stored in the first buffer memory unit 11 as the value of the element in the 0th row and 0th column of the output channel number 0 in the feature map.

Fig. 13 shows that the activity u calculated by the unit processing units 100 having the same first index of PB is stored in the first buffer storage unit 11 as the value of an element having the same output channel number among the elements of the feature map. Fig. 13 shows that the activity u calculated by the unit processing units 100 having the same second index of PB is stored in the first buffer storage unit 11 as the value of an element having a different output channel number Ch _out but the same input channel number Ch _in among the elements of the feature map.

An example of the flow of the convolution process will be described with reference to Figures 14 to 20. In the convolution process, first, the data of the first row of data of each channel of the layer input data is input to the main calculation unit 10. Next, the second to sixth processes of the convolution process described below are performed on a predetermined filter coefficient (hereinafter referred to as the "initial filter coefficient") among the filter coefficients of the convolution filter. Next, the second to sixth processes are performed for each of the other filter coefficients of the convolution filter according to a predetermined order for the filter coefficients. Thereafter, the data of the second row of data of each channel of the layer input data is input to the main calculation unit 10, and the second to sixth processes are performed for each filter coefficient. In the same manner, the same processes are performed until the last row (for example, the 64th row) of data of each channel of the layer input data.

Hereinafter, an example of the flow of the convolution process will be described using an example in which the convolution filter has four filter coefficients, _w11 , _w12 , _w21 , and _w22 . Also, an example of the flow of the convolution process will be described using an example in which the initial filter coefficient is _w11 . For the sake of simplicity, in the following, in Figs. 14 to 18, an example of the flow of the convolution process will be described using an example in which the data in the first row of data in each channel of the layer input data is input to the main arithmetic unit 10 and the filter coefficient is _w11 . Then, in Fig. 19, a change in data in the main arithmetic unit 10 when the filter coefficient for executing the second to sixth processes is changed from _w11 to _w12 will be described. Fig. 20 describes data input to the main arithmetic unit 10 when convolution processes are executed for data in other rows of data in each channel of the layer input data.

Figure 14 is an explanatory diagram explaining the first process of the convolution process in the embodiment. Figure 14 shows that in the first process of the convolution process, layer input data is input to the main calculation unit 10 according to the first activation data input rule and the second activation data input rule.

Figure 15 is an explanatory diagram illustrating the second process of the convolution process in the embodiment. The second process is executed after the first process.

15 , w _AB represents a filter coefficient w whose fourth index in the five-index display is A and whose fifth index in the five-index display is B. For example, w ₁₂ represents a filter coefficient w whose fourth index in the five-index display is 1 and whose fifth index in the five-index display is 2. In the second process of the convolution process, the filter coefficient w whose fourth index and fifth index in the five-index display are 1 are input to the main calculation unit 10.

Figure 15 shows that in the second process of the convolution process, the filter coefficients of the convolution filter stored in the second buffer memory unit 12 are input to the main calculation unit 10 according to the filter input rule. The filter input rule is a rule that a filter coefficient w that satisfies rule condition 2 is input to a unit processing unit 100 that satisfies the following rule condition 1. Rule condition 1 includes a condition that the unit processing unit 100 is represented by a first index of a four-index display that represents the same layer number as the layer number represented by the first index of a five-index display of the filter coefficient w. Rule condition 1 includes a condition that the unit processing unit 100 is a unit into which activation data x is input, the second index of a four-index display that represents the same input channel number as the input channel number represented by the second index of a five-index display of the filter coefficient w. Rule condition 2 includes a condition that the filter coefficient w is represented by a first index of a five-index display that represents the same layer number as the layer number represented by the first index of a four-index display of the activation data x input to the unit processing unit 100 that satisfies rule condition 1. Rule condition 2 includes the condition that the filter coefficient w represented by the second index of the five-index display has the same input channel number as the input channel number represented by the second index of the four-index display of the activation data x input to the unit processing unit 100 that satisfies condition 1.

In the second step of the convolution process, each unit processing unit 100 executes a process of calculating the product of the filter coefficient _w11 input to each unit processing unit 100 and the activation data x input to each unit processing unit 100 in the first step of the convolution process.

16 is an explanatory diagram illustrating the third process of the convolution process in the embodiment. The third process is executed after the second process.
In the third process of the convolution process, each unit processing unit 100 outputs the activation data x used in the immediately preceding process to the connected unit processing unit 100. For example, Fig. 16 shows that PB (63, 0) outputs the activation data x (0, 63, 0, 0) used in the second process of the convolution process to PB (0, 0). That is, the activation data x (0, 63, 0, 0) is input to PB (0, 0).

Figure 17 is an explanatory diagram explaining the fourth process of the convolution process in the embodiment. The fourth process is executed after the third process. In the fourth process of the convolution process, the filter coefficient w is input to the main calculation unit 10 according to the filter input rule.

18 is an explanatory diagram illustrating the fifth process of the convolution process in the embodiment. The fifth process is executed after the fourth process.
In the fifth process of the convolution process, each unit processing unit 100 outputs the activation data x used in the immediately preceding process to the connected unit processing unit 100. For example, FIG. 18 shows that PB(63,0) outputs the activation data x(0,62,0,0) used in the fourth process of the convolution process to PB(0,0). That is, the activation data x(0,62,0,0) is input to PB(0,0).

19 is an explanatory diagram illustrating the sixth process of the convolution process in the embodiment. The sixth process is executed after the fifth process.
In the sixth process of the convolution process, each PB receives activation data x from a connected PB whose second index is one value larger than the first index. Comparing Fig. 14 with Fig. 19, the activation data x stored in each PB in Fig. 19 differs by one in the value of the fourth index of the four-index display from the activation data x stored in each PB in Fig. 14.

After the sixth process of the convolution process is executed, the second to sixth processes are executed for the next filter coefficient. For example, if the next filter coefficient is the _w12 filter coefficient after the second to sixth processes are executed for the _w11 filter coefficient, the second to sixth processes are executed for all filter coefficients of the convolution filter for the first row data of each channel of the layer input data. Next, the process shown in FIG. 20 is executed.

FIG. 20 is an explanatory diagram illustrating the seventh process of the convolution process in the embodiment.
20 shows that the second row of data for each channel of the layer input data is input to the main calculation unit 10. After the seventh process is executed, the second to sixth processes are executed for all filter coefficients of the convolution filter. Next, the next row of data for each channel of the layer input data is input to the main calculation unit 10. For example, if the previous row is the second row, the next row is the third row. For example, if the previous row is the i-th row, the next row is the (i+1)-th row.

Figure 21 is a flowchart illustrating an example of the flow of convolution processing in an embodiment. In Figure 21, the sub-convolution processing is processing in which the second to sixth processes of the convolution processing are performed for all filter coefficients.

The convolution process is started (step S101). The activation data x of the first row of each channel of the layer input data is input to the main calculation unit 10 from the first buffer memory unit 11 (step S102). The execution of the sub-convolution process is started (step S103). In the sub-convolution process, the second process of the convolution process is executed first (step S104). Next, the third process of the convolution process is executed (step S105). Next, the fourth process of the convolution process is executed (step S106). Next, the fifth process of the convolution process is executed (step S107). Next, the sixth process of the convolution process is executed (step S108). When the processes of steps S104 to S108 are executed for all filter coefficients, the sub-convolution process for the activation data x input in step S102 is completed (step S109). It is determined whether or not the sub-convolution process has been performed on the activation data x of all rows of the layer input data (step S110). The determination in step S110 is made, for example, by the control unit 14. If there are rows on which the sub-convolution process has not been performed (step S110: YES), the activation data x of the rows on which the sub-convolution process has not yet been performed in each channel of the layer input data is input from the first buffer memory unit 11 to the main calculation unit 10 (step S111). If there are multiple rows on which the sub-convolution process has not yet been performed, the activation data x is input in a predetermined processing order. The predetermined order is, for example, in order of the smallest row number. Next, the process returns to step S103, the process up to step S109 is performed, and the determination in step S110 is made. On the other hand, if there are no rows on which the sub-convolution process has not yet been performed (step S110: NO), the convolution process ends (step S112).

In the neural network circuit 1 of the embodiment configured in this manner, in the processing of the convolutional layer, data input from the first buffer memory unit 11 and the second buffer memory unit 12 to the main processing unit 10 is output from the unit processing unit 100 to another unit processing unit 100. Therefore, the main processing unit 10 has few opportunities to read data from the first buffer memory unit 11 and the second buffer memory unit 12. This means that the neural network circuit 1 has few opportunities to read data from the first external memory 91 and the second external memory 92. Therefore, the neural network circuit 1 configured in this manner can improve the calculation speed in the convolutional neural network in the processing of the convolutional layer. In addition, since there are few opportunities to read data from the first external memory 91 and the second external memory 92, the neural network circuit 1 is a circuit with low power consumption.

<Processing in the fully connected layer>
The processing in the fully connected layer (hereinafter referred to as "fully connected processing") executed by the main processing unit 10 will be described with reference to Fig. 22 to Fig. 32. In the description of the processing in the fully connected layer, the symbol w represents a fully connected coefficient rather than a filter coefficient.

FIG. 22 is an explanatory diagram for explaining an overview of the full join process in the embodiment.
In Fig. 22, _x0 , _x1 , _x2 , and _x3 represent activation data. In Fig. 22, _w00 , _w01 , _w02 , and _w03 represent total coupling coefficients. In Fig. 22, _u0 , _u1 , _u2 , and _u3 represent activity.

In the total coupling process, x ₀ , x ₁ , x ₂ and x ₃ are multiplied by a total coupling coefficient determined in advance. In FIG. 22, the total coupling coefficient w _AB indicates a value multiplied by the activation data x _B in the calculation to calculate the activation u _A. That is, the total coupling coefficient w is a value that is distinguished from other total coupling coefficients by two indices, and the first index A of the total coupling coefficient w _AB indicates that it is a value used to calculate the activation u _A. In addition, the second index B of the total coupling coefficient w _AB indicates the activation data x _B to be multiplied. In the total coupling process, the sum of the values after multiplication is output as one activation. For example, in the example of FIG. 22, the sum of the product of x ₀ and w ₀₀ , the product of x ₁ and w ₀₁ , the product of x ₂ and w ₀₂ , and the product of x ₃ and w ₀₃ is output as the activation u ₀ . The same is true for the other activities u. If the activity u calculated in the example of FIG. 22 is expressed by a formula, it can be expressed by the following formulas (3) to (5).

In this way, in the fully connected process, a plurality of activations _uA are calculated. Therefore, in order to execute the fully connected process, the control buffer storage unit 13 stores in advance information indicating the unit processing unit 100 used to calculate the activation _uA in the process of the fully connected layer for each activation _uA (hereinafter referred to as "fully connected corresponding information").

FIG. 23 is a first explanatory diagram illustrating an example of the flow of data between the unit processing units 100 when the all-join process of the embodiment is executed.
Fig. 23 shows that data is input to the unit processing unit 100 from other unit processing units 100 that are designated as connection destinations by the configuration data. In the example of Fig. 23, for example, data is input to PB(1,1) from PB(0,1), PB(1,0), PB(1,2), and PB(2,1).

FIG. 24 is a second explanatory diagram illustrating an example of the flow of data between the unit processing units 100 when the all-join process of the embodiment is executed.
Fig. 24 shows that data is output from the unit processing unit 100 to another unit processing unit 100 that is specified as a connection destination by the configuration data. In the example of Fig. 24, for example, data is output from PB(1,1) to PB(0,1), PB(1,0), PB(1,2) and PB(2,1).

An example of the flow of the full connection process will be described using Figures 25 to 31. The full connection process includes the first to sixth processes described below. In the full connection process, after the first to sixth processes described below are executed, the fourth to sixth processes are repeated until a specific end condition is met. The specific end condition is, for example, that all of the connection coefficients have been used to calculate the activity u.

Figure 25 is an explanatory diagram illustrating an example of the first process of the full connection process in an embodiment. In the first process of the full connection process, activation data x is input from the first buffer memory unit 11 to a unit processing unit 100 that is associated in advance. The activation data in the full connection process is, for example, activity calculated by a convolution process.

The example of FIG. 25 indicates that activation data _x0 is input to PB(0,0). The example of FIG. 25 indicates that activation data _x1 is input to PB(0,1). The example of FIG. 25 indicates that activation data _x2 is input to PB(1,0). The example of FIG. 25 indicates that activation data _x3 is input to PB(1,1). That is, the example of FIG. 25 indicates that activation data _x0 , _x1 , _x2 , and _x3 in the calculation of formula (5) are read out to the main calculation unit 10. That is, the example of FIG. 25 indicates that the value surrounded by the dotted line in the following formula (7) is read out to the unit processing unit 100.

26 and 27 are diagrams illustrating the second step of the full join process.
FIG. 26 is a first explanatory diagram for explaining an example of the second process of the full-bonding process in the embodiment. The second process is executed after the first process. In the second process of the full-bonding process, all the bond coefficients stored in the second buffer storage unit 12 are input to the main calculation unit 10 according to the first rule for inputting all bond coefficients. The first rule for inputting all bond coefficients is a rule that all bond coefficients w that satisfy the following rule conditions 4 and 5 are input to the unit processing unit 100 that satisfies the following rule condition 3. Rule condition 3 is a condition that the activation data x _B indicated by the second index of the total bond coefficients w _AB to be input is the unit processing unit 100 input by the first process. Rule condition 4 is a condition that the activation data x _B indicated by the second index has already been input by the first process to the unit processing unit 100 of the input destination. Rule condition 5 is a condition that the full-bond correspondence information indicates that the unit processing unit 100 of the input destination is the unit processing unit 100 for calculating the activation u _A indicated by the first index.

Fig. 26 shows that all coupling coefficients are input from the second buffer storage unit 12 to the main calculation unit 10. Specifically, Fig. 26 shows that all coupling coefficients _w00 are input to PB(0,0), and all coupling coefficients w22 are input to PB(1,0). That is, the example of Fig. 26 shows that the values surrounded by dotted lines in the following formula (7) are read out to the corresponding unit processing unit 100.

FIG. 27 is a second explanatory diagram illustrating an example of the second process of the full join process in the embodiment.
Fig. 27 shows that all coupling coefficients are input from the second buffer memory unit 12 to the main calculation unit 10. Specifically, Fig. 27 shows that all coupling coefficients _w11 are input to PB(0,1), and all coupling coefficients w33 are input to PB(1,1). That is, the example of Fig. 27 shows that the values surrounded by dotted lines in the following formula (8) are read out to the corresponding unit processing unit 100.

28 is a diagram illustrating an example of the third process of the full join process according to the embodiment. The third process is executed after the second process.
In the third process of the full connection process, the product of the values input to each unit processing unit 100 in the first and second processes of the full connection process is calculated in each unit processing unit 100. That is, in the example of Fig. 25 to Fig. 27, the value of the term surrounded by the dotted line in the following equation (9) is calculated. The calculated value (hereinafter referred to as the "full connection product result") is stored in the register 117 provided in each unit processing unit 100.

29 is a diagram illustrating an example of the fourth process of the full join process in the embodiment. The fourth process is executed after the third process or after the sixth process described below.
In the fourth process of the full connection process, each unit processing unit 100 outputs the activation data x used in the immediately preceding process to the connected unit processing unit 100. For example, FIG. 29 shows that PB(1,0) outputs the activation data _x2 used in the third process of the full connection process to PB(0,0). That is, the activation data _x2 is input to PB(0,0). More specifically, in the example of FIG. 29, the value of the term surrounded by the dotted line in the following formula (10) is input to the corresponding unit processing unit 100 by the fourth process of the full connection process.

30 is a diagram illustrating an example of the fifth process of the all-join process according to the embodiment. The fifth process is executed after the fourth process.
In the fifth process of the full coupling process, all coupling coefficients stored in the second buffer memory unit 12 are input to the main calculation unit 10 according to the second rule for inputting all coupling coefficients. The second rule for inputting all coupling coefficients is a rule that all coupling coefficients w that satisfy the following rule conditions 7 and 8 are input to the unit processing unit 100 that satisfies the following rule condition 6. Rule condition 6 includes a condition that the unit processing unit 100 is the unit to which the activation data x _B indicated by the second index of the all coupling coefficients w _AB was input in the process immediately before the full coupling process (i.e., the fourth process of the full coupling process). Rule condition 7 includes a condition that the activation data x _B input to the unit processing unit 100 that satisfies rule condition 3 is the all coupling coefficient indicated by the second index. Rule condition 8 includes a condition that the activation u _A indicated by the first index is the activation u _A indicated by the first index of the all coupling coefficients input to the unit processing unit 100 that satisfies rule condition 3 in the second process of the full coupling process.

For example, in the example of FIG. 30, a total coupling coefficient w ₀₂ is input to PB(0,0). For example, in the example of FIG. 30, a total coupling coefficient w ₁₀ is input to PB(0,1). For example, in the example of FIG. 30, a total coupling coefficient w ₂₃ is input to PB(1,0). For example, in the example of FIG. 30, a total coupling coefficient w ₃₁ is input to PB(1,1). More specifically, in the example of FIG. 30, the value of the term surrounded by a dotted line in the following formula (11) is input to the corresponding unit processing unit 100 by the fifth process of the total coupling process.

31 is a diagram illustrating an example of the sixth process of the full join process according to the embodiment. The sixth process is executed after the fifth process.
In the sixth process of the full connection process, the product of the activation data input in the fourth process of the full connection process and the full connection coefficients input in the fifth process of the full connection process is calculated. The calculated value is stored in the register 117 as the full connection product result. For example, in the example of FIG. 31, the value of the term surrounded by the dotted line in the following formula (12) is calculated.

FIG. 32 is a flowchart showing an example of the flow of the all-join process in the embodiment.
The full-combination process is started (step S201). The first process of the full-combination process is executed (step S202). Next, the second process of the full-combination process is executed (step S203). Next, the third process of the full-combination process is executed (step S204). Next, the fourth process of the full-combination process is executed (step S205). Next, the fifth process of the full-combination process is executed (step S206). Next, the sixth process of the full-combination process is executed (step S207). It is determined whether all of the coupling coefficients have been used in the calculation of the activity u (step S208). The determination in step S208 is made by, for example, the control unit 14. If there are any coupling coefficients that have not been used in the calculation of the activity u (step S208: YES), the process in step S205 is executed. On the other hand, if there are no all coupling coefficients that have not been used in the calculation of the activation u (step S208: NO), each processing unit 100 calculates the sum of all coupling product results that each processing unit 100 has stored in the register 117 after step S201 (step S209). The calculated sum is the activation u. After step S209, the all coupling process ends (step S210).

In the neural network circuit 1 of the embodiment configured in this manner, in the processing of the fully connected layer, data input from the first buffer memory unit 11 and the second buffer memory unit 12 to the main processing unit 10 is output from the unit processing unit 100 to another unit processing unit 100. Therefore, the main processing unit 10 has few opportunities to read data from the first buffer memory unit 11 and the second buffer memory unit 12. This means that the neural network circuit 1 has few opportunities to read data from the first external memory 91 and the second external memory 92. Therefore, the neural network circuit 1 configured in this manner can improve the calculation speed in the convolutional neural network in the processing of the fully connected layer. In addition, since there are few opportunities to read data from the first external memory 91 and the second external memory 92, the neural network circuit 1 is a circuit with low power consumption.

In addition, the neural network circuit 1 of the embodiment configured in this manner can change the connection relationships between the unit processing units 100 based on the configuration data under the control of the control unit 14. Therefore, the neural network circuit 1 is a highly versatile circuit that can execute both the processing of the convolution layer and the processing of the fully connected layer with the same circuit.

(Modification)
In the main processing unit 10, the unit processing units 100 do not necessarily need to be arranged two-dimensionally in a lattice pattern. The multiple unit processing units 100 may be arranged in any manner as long as they are arranged at positions where they can be connected to other unit processing units 100 in accordance with the connection relationship indicated by the configuration data. For example, the unit processing units 100 may be arranged three-dimensionally. For example, each unit processing unit 100 may be located at each vertex of a cubic lattice.

The first buffer memory unit 11 is an example of a first memory unit. The second buffer memory unit 12 is an example of a second memory unit. The unit processing unit 100 is an example of a calculation unit. The filter coefficients and the total coupling coefficients are examples of weight data. The product of the activation data and the filter coefficients and the product of the activation data and the total coupling coefficients are examples of term values. Each filter coefficient in FIG. 15 and FIG. 17 and each total coupling coefficient in FIG. 26 to FIG. 31 are examples of one piece of weight data that satisfies a predetermined condition related to the weight data. Rule condition 1, rule condition 2, rule condition 4, rule condition 5, rule condition 7, and rule condition 8 are examples of a predetermined condition related to the weight data.

The above describes an embodiment of the present invention in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and includes designs that do not deviate from the gist of the present invention.

1...Neural network circuit, 10...Main calculation unit, 11...First buffer memory unit, 12...Second buffer memory unit, 13...Control buffer memory unit, 14...Control unit, 100...Unit processing unit, 101...Connection unit, 102...Product-sum calculation unit, 111...Input control circuit, 112...Weight control circuit, 113...Output control circuit, 114, 115, 116, 117...Register, 118...Feedback conductor, 201...Product-sum calculation circuit, 202...Activation circuit

Claims (4)

A neural network circuit for executing processing of a convolutional layer or a fully connected layer in a convolutional neural network having a fully connected layer and a plurality of convolutional layers, comprising:
A first storage unit that stores activation data, which is data input to each layer and is represented by a tensor, and is a value of each element of the data;
A second storage unit that stores weight data for processing executed in the convolution layer or the fully connected layer;
A plurality of calculation units that read out one of the activation data from the first storage unit at a predetermined timing, and obtain a term value that is a value of a term of a product of the read out activation data and one of the weight data that satisfies a predetermined condition related to the weight data;
Equipped with
The activation data read by the calculation unit is output to another calculation unit associated with the activation data after the calculation unit calculates the term value ,
The calculation unit is used in common for the processing of the convolution layer and the processing of the fully connected layer.
Neural network circuit. The process of the convolution layer is a convolution integral,
The weight data when performing the convolution integral is a filter coefficient, which is each value of a filter for performing the convolution integral.
2. The neural network circuit of claim 1. The weight data at the time of execution of the processing of the fully connected layer is a weight coefficient used in the processing of the fully connected layer.
2. The neural network circuit of claim 1. A neural network circuit for executing processing of a convolutional layer or a fully connected layer in a convolutional neural network having a fully connected layer and a plurality of convolutional layers, the neural network circuit comprising: a first storage unit for storing activation data, which is a value of each element of data represented by a tensor and input to each layer; a second storage unit for storing weight data for processing executed in the convolutional layer or the fully connected layer; and a plurality of arithmetic units for reading one of the activation data from the first storage unit at a predetermined timing and acquiring a term value, which is a term value of a product of the read activation data and one of the weight data that satisfies a predetermined condition related to the weight data, the activation data read by the arithmetic unit is output to another of the arithmetic units associated in advance after the arithmetic unit calculates the term value , the arithmetic unit being used in common for processing of the convolutional layer and processing of the fully connected layer, the neural network arithmetic unit comprising:
An acquisition step in which the calculation unit reads out one of the activation data from the first storage unit at a predetermined timing, and acquires a term value that is a value of a term of a product of the read out activation data and one of the weight data that satisfies a predetermined condition related to the weight data;
an output step in which the activation data read by the calculation unit is output to another calculation unit associated with the activation data after the calculation unit calculates the term value;
The neural network operation method includes:

Images