JP2021022087A - Processor for neural network, processing method for neural network, and program - Google Patents

Abstract

To provide a processor for neural network which executes high-performance neural network processing while suppressing increase in hardware scale.SOLUTION: A processor for neural network 100 writes data for processing a feature extraction layer which is fixed when a model of a neural network is decided into a ROM which requires small hardware scale, and holds data for processing a determination layer which is likely to be modified in a data-rewritable random access memory. The processor for neural network 100 executes processing using the neural network in the above environment. Thus, high-performance neural network processing can be executed while suppressing increase in hardware scale.SELECTED DRAWING: Figure 1

Description

The present invention relates to a neural network technique.

In recent years, various techniques using CNN (Convolutional Neural Network), which is one of the neural network techniques, have been developed (see, for example, Patent Document 1). Among the CNNs, the technology using DCNN (Deep Convolutional Neural Network) with many intermediate layers has been attracting particular attention because it has achieved results in various fields.

JP 2015-197702

DCNN realizes high recognition performance in various tasks such as general object recognition and semantic segmentation. On the other hand, since the number of parameters required for executing the process of DCNN is very large, the hardware scale becomes large when DCNN is realized by hardware. In particular, since the scale of hardware for mounting a memory for storing and holding parameters (for example, weight coefficient data) becomes large, it is difficult to reduce the cost when DCNN is realized by hardware.

Therefore, in view of the above problems, it is an object of the present invention to realize a neural network processor, a neural network data processing method, and a program that execute high-performance neural network processing while suppressing an increase in hardware scale. And.

In order to solve the above problems, the first invention is a neural network processor for executing neural network processing including processing of a feature extraction layer and processing of a determination layer, and is a read-only memory and random. It includes an access memory, a control unit, and an internal product processing unit.

The read-only memory stores and holds data for processing in the feature extraction layer.

The random access memory can read data and write data, and stores and holds data for processing in the determination layer.

The control unit controls the read-only memory and the random access memory.

The inner product processing unit reads the data for processing the feature extraction layer from the read-only memory as the first data, executes the processing of the feature extraction layer using the first data, and processes the determination layer from the random access memory. Data is read out as the second data, and the processing of the determination layer is executed using the second data.

In this neural network processor, read-only memory (ROM) that requires a small hardware scale when the processing data (for example, weighting coefficient data) of the feature extraction layer, which is determined once the model of the neural network is determined, is converted into hardware. The processing data (for example, weighting coefficient data) of the determination layer whose data may be changed by writing to is held in a random access memory (RAM) capable of rewriting the data. Then, in this neural network processor, the processing by the neural network is executed according to the above state. That is, in this neural network processor, there is a possibility that the processing data (for example, weight coefficient data) of the feature extraction layer, which is not changed, is held and changed in the read-only memory which requires a small hardware scale. Since the processing data (for example, weight coefficient data) of a certain judgment layer is held in the random access memory and the neural network processing is executed, the high-performance neural network processing is executed while suppressing the increase in the hardware scale. be able to.

The second invention is the first invention, in which the read-only memory executes the processing of the feature extraction layer and the header data including the data for specifying the read time of the data for the processing of the feature extraction layer. The data for processing of the feature extraction layer is stored and retained so that the data for processing of the feature extraction layer can be output in the order required.

Then, when the control unit outputs a read command signal to the read-only memory, the read-only memory supplies the data for processing of the feature extraction layer in the order required when executing the processing of the feature extraction layer. Output.

In this neural network processor, the weighting coefficient data determined according to the network configuration of the feature extraction layer is read-only so that it is read out in the order required when executing the prediction processing (in the processing order at the time of the prediction processing). It is stored in the memory. Therefore, in this neural network processor, only a read command signal (for example, a single trigger signal) is input to the read-only memory at the time of executing the prediction processing, and the internal product processing unit is required when executing the prediction processing. The weight coefficient data for processing the feature extraction layer can be acquired in this order. That is, in this neural network processor, the weight coefficient data for processing the necessary feature extraction layer can be acquired in the processing order only by inputting the read command signal (for example, a single trigger signal) to the read-only memory. , It is not necessary to specify a complicated address for the ROM as in the prior art.

The third invention is the second invention, in which the control unit reads the header data stored in the read-only memory, and based on the header data, from the read-only memory for processing the feature extraction layer. The time when the data output processing is completed is specified, and the data for the processing of the determination layer is controlled to be stored and held in the random access memory at the time before the specified time.

In this neural network processor, it is possible to specify the time to read the data for processing the feature extraction layer (for example, the weighting coefficient data) from the header data stored in the read-only memory. Therefore, in this neural network processor, data for processing (weight coefficient data) of the determination layer is, for example, predetermined based on the time for reading data for processing (for example, weight coefficient data) of the specified feature extraction layer. It can be transferred in advance from the ROM of the above to the random access memory. As a result, in this neural network processor, as soon as the processing of the feature extraction layer is completed, the data for processing the determination layer (for example, the weighting coefficient data) can be acquired (read) from the random access memory. Can be. That is, in this neural network processor, as soon as the processing of the feature extraction layer is completed, the processing of the determination layer can be executed. As a result, in this neural network processor, the processing for the neural network can be speeded up (the time until the processing of the determination layer is completed can be shortened).

The fourth invention is any one of the first to third inventions, further comprising a decompression unit that executes decompression processing on the compressed data.

The read-only memory stores and holds the data for processing of the feature extraction layer as compressed data.

The decompression unit executes decompression processing on the compressed data stored and held in the read memory.

The inner product processing unit executes the processing of the feature extraction layer using the data expanded by the expansion unit.

In this neural network processor, the data of the feature extraction layer is stored and held in the read-only memory as compressed data, so that the memory capacity required for the read-only memory is further reduced. Therefore, in this neural network processor, the hardware scale of the read-only memory can be further reduced. As a result, the hardware scale of this neural network processor can be further reduced.

A fifth invention is a neural network processor for executing neural network processing including processing of a feature extraction layer and processing of a determination layer.
A read-only memory that stores and holds data for processing in the feature extraction layer,
A random access memory that can read and write data and stores and holds data for processing in the judgment layer,
A control unit that controls read-only memory and random access memory,
It is a processing method for a neural network executed by using a processor for a neural network provided with. The processing method for a neural network includes a first step and a second step.

In the first step, the data for processing the feature extraction layer is read from the read-only memory as the first data, and the processing of the feature extraction layer is executed using the first data.

In the second step, the data for processing the determination layer is read from the random access memory as the second data, and the processing of the determination layer is executed using the second data.

As a result, it is possible to realize a processing method for a neural network that has the same effect as that of the first invention.

The sixth invention is a program for causing a computer to execute the processing method for a neural network according to the fifth invention.

Thereby, it is possible to realize a program for causing a computer to execute a processing method for a neural network having the same effect as that of the first invention.

According to the present invention, it is possible to realize a neural network processor, a neural network data processing method, and a program that execute high-performance neural network processing while suppressing an increase in hardware scale.

The schematic block diagram of the neural network processor 100 which concerns on 1st Embodiment. A timing chart of processing executed by the neural network processor 100. The flowchart of the process executed by the neural network processor 100. The schematic block diagram of the neural network processor 100A which concerns on 1st modification of 1st Embodiment. The figure which shows the CPU bus configuration.

[First Embodiment]
The first embodiment will be described below with reference to the drawings.

<1.1: Configuration of processor for neural network>
FIG. 1 is a schematic configuration diagram of a neural network processor 100 according to the first embodiment.

As shown in FIG. 1, the neural network processor 100 includes a first interface unit IF1, a control unit 1, a first ROM (Read Only Memory) 2, a second ROM 3, a quantization processing unit 4, and a first RAM (1st RAM). It includes a Random Access Memory) 5, a second RAM 6, an inner product processing unit 7, and a bus B1. As shown in FIG. 1, each functional unit of the neural network processor 100 is connected by a bus B1, and necessary data, commands, and the like can be input / output via the bus B1. It should be noted that a part or all of the above functional parts may be directly connected, if necessary, instead of being connected by bus.

The first interface unit IF1 inputs the data Din to be processed from the outside, and outputs the data including the processing result by the neural network processor 100 to the outside as the data Dout. The first interface unit IF1 outputs (writes) the input data Din to the first RAM 5 via the bus B1, and outputs (writes) the data including the processing result by the neural network processor 100 from the first RAM 5 via the bus B1. Get (read).

The control unit 1 performs overall control of the neural network processor 100, control of each functional unit, and processing necessary for neural network processing. The control unit CPU 1 is realized by a CPU (Central Processing Unit) or a CPU core.

The first ROM 2 is a read-only memory that stores and holds data used for processing the feature extraction layer (for example, parameter data and neural network weighting coefficient data). As shown in FIG. 1, the first ROM 2 is connected to the bus B1 and is also connected to the inner product processing unit 7. The first ROM 2 may be connected to the inner product processing unit 7 via the bus B1. When the first ROM 2 receives the trigger signal Sig_trg from the control unit 1, for example, the header data is output to the control unit 1 via the bus B1, and the header data and the data used for the processing of the feature extraction layer are inner products. It is continuously output to the processing unit 7.

The second ROM 3 is a read-only memory and is connected to the bus B1. The second ROM 3 stores and holds data (for example, parameter data, neural network weighting coefficient data) used for processing of the determination layer. The second ROM 3 transfers data used for processing of the determination layer (for example, parameter data, neural network weighting coefficient data) to the second RAM 6 via the bus B1 in accordance with a command from the control unit 1.

The quantization processing unit 4 is connected to the first RAM so that data can be transmitted and received. The quantization processing unit 4 performs quantization processing on the data of the feature map which is the input of the feature extraction layer (for example, the convolutional layer of the DCNN (Deep Convolution Natural Network)). In addition, the quantization processing unit 4 performs quantization processing on the input data of the determination layer (for example, the fully connected layer of DCNN). The quantization processing unit 4 reads the data to be quantized from the predetermined area of the first RAM 5, and writes the data after the quantization processing to the predetermined area of the first RAM 5. Further, as shown in FIG. 1, the quantization processing unit 4 may be directly connected to the first RAM 5 or may be connected via a bus (for example, bus B1). ..

The first RAM 5 is a RAM (Random Access Memory) for storing and holding data necessary for executing a neural network process. As shown in FIG. 1, the first RAM 5 is connected to the bus B1 and is also connected to the quantization processing unit 4.

The second RAM 6 is a random access memory. As shown in FIG. 1, the second RAM 6 is connected to the bus B1 and is also connected to the inner product processing unit 7. The second RAM 6 stores and holds data (for example, parameter data, neural network weighting coefficient data) transferred from the second ROM 3 and used for processing the determination layer. The second RAM 6 outputs the data used for the processing of the determination layer stored and held to the inner product processing unit 7 at a predetermined timing in accordance with the command from the control unit 1.

As shown in FIG. 1, the inner product processing unit 7 includes a second interface unit 71 and an inner product calculation processing unit 72. As shown in FIG. 1, the inner product processing unit 7 is connected to the first ROM2, the first RAM5, the second RAM6, and the bus B1. The inner product processing unit 7 receives the control signal Ctr1 from the control unit 1 via, for example, the bus B1, and executes a predetermined process according to the received control signal Ctr1.

The second interface unit 71 is an interface with the first ROM 2, the first RAM 5, and the second RAM 6.
(1) When the processing of the feature extraction layer is executed, the second interface unit 71 reads out the data for processing the feature extraction layer (for example, parameters, weighting factors) D_wij_feature from the first ROM2, and reads the read data. It is output as data D_wij to the inner product calculation processing unit 72. Further, the second interface unit 71 reads the data D_Qin after the quantization process from the predetermined area of the first RAM 5, and outputs the read data as the data D_Qin to the inner product calculation processing unit 72.
(2) When the processing of the determination layer is executed, the second interface unit 71 reads the data for processing the determination layer (for example, parameter, weighting coefficient) D_wij_deci from the second RAM 6, and reads the read data into the data D_wij. Is output to the inner product calculation processing unit 72. Further, the second interface unit 71 reads the data D_Qin after the quantization process from the predetermined area of the first RAM 5, and outputs the read data as the data D_Qin to the inner product calculation processing unit 72.

The inner product calculation processing unit 72 inputs the data D_Qin and the data D_wij output from the second interface unit 71. The inner product calculation processing unit 72 executes the inner product calculation process using the data D_Qin and the data D_wij, and acquires the data of the inner product calculation processing result as the data Do. Then, the inner product calculation processing unit 72 outputs the acquired data Do to the first RAM 5.

<1.2: Operation of neural network processor>
The operation of the neural network processor 100 configured as described above will be described below.

FIG. 2 is a timing chart of processing executed by the neural network processor 100.

FIG. 3 is a flowchart of processing executed by the neural network processor 100.

Hereinafter, the operation of the neural network processor 100 will be described with reference to FIGS. 1 to 3.

Generally, a CNN includes an input layer, a convolutional layer, and a fully connected layer. For example, image data is input as input data Din to the first interface unit IF1 of the neural network processor 100, image recognition processing by CNN is executed, and the image recognition processing result is output to the outside as output data Dout.

In CNN, in the processing of the convolution layer or the processing of the fully connected layer, a weighting operation process is executed on the input data, and an activation function (for example, a ramp function (ReLU: Rectifier Liner Unit)) is performed on the processing result. , Sigmoid function, Softmax function, etc.), the output of the convolution layer or the fully connected layer can be obtained.

For convenience of explanation, a case where processing by CNN is executed in the neural network processor 100 (an example) will be described below. It should be noted that the convolutional layer of CNN will be described below assuming that it is composed of N layers (N: natural numbers).

(Step S1):
In step S1, the neural network processor 100 executes input processing of data to be processed (for example, when performing image recognition processing, image data to be processed). Specifically, in the neural network processor 100, the data Din to be processed is acquired (input) from the outside by the first interface unit IF1. Then, the data Din is transferred from the first interface unit IF1 to the first RAM 5 via the bus B1, and the data Din is stored and held in a predetermined area of the first RAM 5.

Further, in the neural network processor 100, the quantization process is executed. Specifically, the quantization processing unit 4 reads data Din from a predetermined area of the first RAM 5, executes quantization processing on the read data, and acquires the data after quantization as data D_Qin. .. Then, the quantization processing unit 4 writes the acquired data D_Qin after quantization into a predetermined area of the first RAM 5.

The quantization process may include, for example, a dynamic range adjustment, a binarization process, and an offset adjustment process in addition to the process of quantizing the data.

(Step S2, Step S3):
In step S2, the control unit 1 transmits a trigger signal Sig_trig to the first ROM 2 via the bus B1.

When the first ROM 2 receives the trigger signal Sig_trig from the control unit 1, the first ROM 2 reads the header data D_head stored and held in the first ROM 2, and the header data D_head is transmitted to the control unit 1 and the inner product processing unit 7 via the bus B1. Is output to the second interface unit 71 (step S3).

The header data D_head includes, for example, data such as the number of layers constituting the feature extraction layer, the data length of each layer, and the number of data.

(Steps S41 to S45):
In steps S41 to S45, the processing for each layer of the feature extraction layer is repeatedly executed for the number of layers of the feature extraction layer.

After outputting the header data D_head, the first ROM 2 reads out the data of the feature extraction layer in order from the data of the first layer of the feature extraction layer to the data of the Nth layer, and the read data is internally loaded as data D_wij_feature. Output to the processing unit 7 (weight coefficient data acquisition processing, step S42).

Specifically, (1) weight coefficient data of the first layer of the feature extraction layer, (2) weight coefficient data of the second layer of the feature extraction layer, ..., (N) Nth layer of the feature extraction layer. The weighting coefficient data are read from the first ROM 2 in the above order, and are output to the inner product processing unit 7 as data D_wij_feature.

In step S43, the inner product calculation process is executed. Specifically, the inner product calculation process is executed as follows.

The second interface unit 71 of the inner product processing unit 7 acquires the weighting coefficient data of the first layer of the feature extraction layer output from the first ROM 2 as data D_wij_feature, and uses the data as data D_wij to obtain the inner product calculation processing unit 72. Output to.

Further, the second interface unit 71 of the inner product processing unit 7 reads the quantized data of the data input to the first layer of the feature extraction layer from the first RAM 5 and acquires it as data D_Qin. Then, the second interface unit 71 outputs the acquired data D_Qin to the inner product calculation processing unit 72.

なお、内積演算処理部７２は、行列演算処理を行う場合、上記内積演算処理を繰り返し実行することで、行列演算処理結果を取得する。例えば、内積演算処理部７２は、２つの行列Ａ、Ｂの積（行列の積）を求める演算を行う場合、行列Ａのｉ行目の１行に含まれる要素からなる行ベクトルと、行列Ｂのｊ列目の１列に含まれる要素からなる列ベクトルとに対して、上記と同様に内積演算処理を実行することで、行列Ａ、Ｂの積により取得される行列のｉ行ｊ列目の要素の値（データ）を取得することができる。 The inner product calculation processing unit 72 executes the inner product calculation process using the data D_Qin and the data D_wij. For example, when the data D_Qin is n-dimensional vector data (n: natural number) and the data D_wij (weight coefficient data) is n-dimensional vector data and is expressed as follows:
D_Qin = [x ₁ x ₂ x ₃ ... x _n ]
D_wij = [w ₁ w ₂ w ₃ ... w _n ]
The inner product calculation processing unit 72 acquires the inner product calculation result by executing a process of calculating the inner product of two n-dimensional vectors, that is, a process corresponding to the following mathematical expression.

When performing the matrix operation processing, the inner product calculation processing unit 72 acquires the matrix operation processing result by repeatedly executing the inner product calculation processing. For example, when the inner product calculation processing unit 72 performs an operation for obtaining the product (matrix product) of two matrices A and B, a row vector composed of elements included in the first row of the i-th row of the matrix A and a matrix B The i-th row and j-th column of the matrix obtained by the product of the matrices A and B by executing the internal product operation processing in the same manner as above for the column vector consisting of the elements included in the first column of the j-th column of. You can get the value (data) of the element of.

By processing in this way, the inner product calculation processing unit 72 can acquire the processing result of an arbitrary matrix operation.

In the inner product calculation processing unit 72, the processing of the first layer of the feature extraction layer (for example, the convolution processing of CNN (weighting addition processing for data)) is executed as described above, and the first layer of the feature extraction layer Acquire the data corresponding to the output of each synapse.

Note that the weight coefficient data that matches the processing order of the first layer of the feature extraction layer is input to the inner product calculation processing unit 72 as data D_wij in that order. Therefore, the inner product calculation processing unit 72 inputs the data D_wij and the corresponding data after quantization (data to be processed (data read from the first RAM 5)) to the inner product calculation processing unit 72. By repeatedly executing the inner product calculation process in order, data corresponding to the output of each synapse in the first layer of the feature extraction layer can be acquired.

When the inner product calculation processing unit 72 acquires data corresponding to the output of each synapse in the first layer of the feature extraction layer (this data is referred to as “data D0”) by the above processing, the acquired data is subjected to the data. The process by the activation function is executed and the data Do is acquired. That is, the inner product calculation processing unit 72
Do = f_act (D0)
f_act (): Activation function (for example, ReLU function, Softmax function, sigmoid function, etc.)
Data Do is acquired by executing the process corresponding to.

In step S44, the inner product calculation processing unit 72 outputs the data Do acquired as described above to the first RAM 5, and writes the data Do to a predetermined area of the first RAM 5.

As described above, in the neural network processor 100, the processing of the first layer of the feature extraction layer is executed. In the timing chart of FIG. 2, between the time t11 and the time t12, the data for processing the first layer of the feature extraction layer (weight coefficient data) D_wij_feature (in FIG. 2, the first layer of the feature extraction layer) from the first ROM2. The data for processing is read as data Df_1st_L) in the order of processing, and the internal product calculation process is executed using the read data (weight coefficient data) D_wij_feature and the data D_Qin after quantization of the processing target. Will be done. Then, in the neural network processor 100, as described above, the process corresponding to the activation function is executed, and the data Do is acquired.

Then, in the neural network processor 100, the processing of the second layer, the third layer, ..., The Nth layer of the feature extraction layer is executed in the same manner as the processing of the first layer described above (steps S41 to 1). S45, processing between times t12 and t2 in FIG. 2).

Then, the data Do (data acquired by the processing of the feature extraction layer) acquired by the above processing is output (written) to the first RAM 5.

(Step S51):
In step S51, a process of specifying the data read end time of the feature extraction layer is executed. Specifically, the control unit 1 reads the header data D_header from the first ROM 2 (time t1 to t11 in FIG. 2), and from the data included in the header data D_header, the feature extraction layer is transferred from the first ROM 2 to the inner product processing unit 7. The time T1 required to transfer the data for processing (weight coefficient data) of the above is calculated.

Ｌ（ｋ）．Ｒｅａｌ＿Ｄ＿ｔｉｍｅ
＝Ｌ（ｋ）．Ｒｅａｌ＿Ｄ＿ｎｕｍ×Ｌ（ｋ）．Ｒｅａｌ＿Ｄ＿ａｃｃｕｒａｃｙ
Ｌ（ｋ）．Ｉｎｔ＿Ｄ＿ｔｉｍｅ
＝Ｌ（ｋ）．Ｉｎｔ＿Ｄ＿ｎｕｍ×Ｌ（ｋ）．Ｉｎｔ＿Ｄ＿ａｃｃｕｒａｃｙ
Ｌ（ｋ）．Ｒｅａｌ＿Ｄ＿ｔｉｍｅ：第ｋ層の実数演算に必要な処理時間
Ｌ（ｋ）．Ｉｎｔ＿Ｄ＿ｔｉｍｅ：第ｋ層の整数演算に必要な処理時間
Ｌ（ｋ）．Ｒｅａｌ＿Ｄ＿ｎｕｍ：第ｋ層の実数演算対象のデータ数
Ｌ（ｋ）．Ｒｅａｌ＿Ｄ＿ａｃｃｕｒａｃｙ：第ｋ層の実数演算の計算精度
Ｌ（ｋ）．Ｉｎｔ＿Ｄ＿ｎｕｍ：第ｋ層の整数演算対象のデータ数
Ｌ（ｋ）．Ｉｎｔ＿Ｄ＿ａｃｃｕｒａｃｙ：第ｋ層の整数演算の計算精度
そして、制御部１は、上記により取得した時間Ｔ１と、特徴抽出層の処理用のデータ（重み係数データ）の転送が開始される時刻ｔ１１とから、特徴抽出層の処理用のデータ（重み係数データ）の転送が完了する時刻ｔ２を、
ｔ２＝ｔ１１＋Ｔ１
により算出する。これにより、制御部１は、特徴抽出層の処理用のデータ（重み係数データ）の転送が完了する時刻ｔ２を特定する。 For example, the header data D_header acquired from the first ROM 2 may include
(1) The number of layers N (N: natural number) included in the feature extraction layer and
(2) The time required for real number calculation in the processing of the k-th layer (k: natural number, 1 ≦ k ≦ N) included in the feature extraction layer, and the k-th layer (k: natural number, 1 ≦ N) included in the feature extraction layer. Calculation accuracy of real number operation in the processing of k ≦ N) and
(3) The time required for integer calculation in the processing of the kth layer (k: natural number, 1 ≦ k ≦ N) included in the feature extraction layer, and the kth layer (k: natural number, 1 ≦ N) included in the feature extraction layer. The calculation accuracy of integer arithmetic in the processing of k ≦ N) and
Is included. Then, the control unit 1 determines the time T1 required to transfer the data (weighting coefficient data) for the processing of the feature extraction layer by, for example, executing the processing corresponding to the following mathematical expression based on the above data. calculate.

L (k). Real_D_time
= L (k). Real_D_num × L (k). Real_D_accuracy
L (k). Int_D_time
= L (k). Int_D_num × L (k). Int_D_accuracy
L (k). Real_D_time: Processing time required for real number calculation of layer k L (k). Int_D_time: Processing time required for integer operation of layer k L (k). Real_D_num: Number of data to be calculated as a real number in the k-th layer L (k). Real_D_accuracy: Calculation accuracy of real number operation of layer k L (k). Int_D_num: Number of data to be calculated as an integer in the k-th layer L (k). Int_D_accuracy: Calculation accuracy of the integer calculation of the k-th layer Then, the control unit 1 starts from the time T1 acquired by the above and the time t11 when the transfer of the data (weight coefficient data) for processing of the feature extraction layer is started. The time t2 at which the transfer of the data (weighting coefficient data) for processing of the feature extraction layer is completed is set.
t2 = t11 + T1
Calculated by As a result, the control unit 1 specifies the time t2 at which the transfer of the data (weighting coefficient data) for processing of the feature extraction layer is completed.

(Step S52):
In step S52, the transfer process of the weighting coefficient data of the determination layer is executed. Specifically, the control unit 1 acquires the time T2 required to transfer the weighting coefficient data D_wij_deci of the determination layer stored and held in the second ROM 3 to the second RAM 6. Then, the control unit 1 outputs a command to transfer the weighting coefficient data D_wij_deci of the determination layer stored and held in the second ROM 3 to the second ROM 6 to the second ROM 3 and the second RAM 6 at a time before the time t20 (= t2-T2). Then, at a time before the time t20 (= t2-T2), the data transfer process of the weighting coefficient data D_wij_deci of the determination layer is started.

As a result, the transfer process of the weighting coefficient data D_wij_deci of the determination layer from the second ROM 3 to the second RAM 6 can be completed by the time t2 when the data read process of the feature extraction layer is completed.

In the neural network processor 100, by performing the processing of the feature extraction layer (steps S41 to S45) and the processing of steps S51 and S52 in parallel in this way, immediately after the time t2 when the processing of the feature extraction layer is completed. Therefore, the processing of the determination layer can be executed.

(Steps S6, S7):
In steps S6 and S7, the processing of the determination layer is executed (processing at times t2 to t3 in FIG. 2).

By time t2 (start time of processing of the determination layer), the processing of the feature extraction layer is completed, and the data for processing the determination layer (weight coefficient data for processing the determination layer) from the second ROM 3 to the third 2 The data transfer process to RAM 6 is completed. Therefore, in step S6, the second interface unit 71 of the inner product processing unit 7 acquires (reads) the processing data of the determination layer (weight coefficient data for processing of the determination layer) D_wij_deci from the second RAM 6. Then, the second interface unit 71 outputs the acquired data D_wij_deci as data D_wij to the inner product calculation processing unit 72.

Further, the second interface unit 71 of the inner product processing unit 7 reads the quantized data of the data input to the determination layer from the first RAM 5 and acquires the data as data D_Qin. Then, the second interface unit 71 outputs the acquired data D_Qin to the inner product calculation processing unit 72.

なお、内積演算処理部７２は、行列演算処理を行う場合、上記内積演算処理を繰り返し実行することで、行列演算処理結果を取得する。例えば、内積演算処理部７２は、２つの行列Ａ、Ｂの積（行列の積）を求める演算を行う場合、行列Ａのｉ行目の１行に含まれる要素からなる行ベクトルと、行列Ｂのｊ列目の１列に含まれる要素からなる列ベクトルとに対して、上記と同様に内積演算処理を実行することで、行列Ａ、Ｂの積により取得される行列のｉ行ｊ列目の要素の値（データ）を取得することができる。 The inner product calculation processing unit 72 executes the inner product calculation process using the data D_Qin and the data D_wij. For example, when the data D_Qin is n-dimensional vector data (n: natural number) and the data D_wij (weight coefficient data) is n-dimensional vector data and is expressed as follows:
D_Qin = [x ₁ x ₂ x ₃ ... x _n ]
D_wij = [w ₁ w ₂ w ₃ ... w _n ]
The inner product calculation processing unit 72 acquires the inner product calculation result by executing a process of calculating the inner product of two n-dimensional vectors, that is, a process corresponding to the following mathematical expression.

When performing the matrix operation processing, the inner product calculation processing unit 72 acquires the matrix operation processing result by repeatedly executing the inner product calculation processing. For example, when the inner product calculation processing unit 72 performs an operation for obtaining the product (matrix product) of two matrices A and B, a row vector composed of elements included in the first row of the i-th row of the matrix A and a matrix B The i-th row and j-th column of the matrix obtained by the product of the matrices A and B by executing the internal product operation processing in the same manner as above for the column vector consisting of the elements included in the first column of the j-th column of. You can get the value (data) of the element of.

By processing in this way, the inner product calculation processing unit 72 can acquire the processing result of an arbitrary matrix operation.

In the inner product calculation processing unit 72, as described above, the processing of the determination layer (for example, the out mapping processing for adjusting the number of dimensions of the data (Out Mapping), the processing of the fully connected layer (corresponding to the vector inner product processing), (Processing of the Softmax layer, etc.) is executed, and data corresponding to the output result of the processing of the determination layer is acquired as data Do. Then, the acquired data Do is output from the inner product calculation processing unit 72 to the first RAM 5, and the data Do is written in a predetermined area of the first RAM 5.

In the processing of the determination layer as well, the processing corresponding to the activation function may be executed as necessary, as in the processing of the feature extraction layer.

The data of the processing result of the determination layer acquired by the above processing is read from the first RAM 5 by, for example, the first interface unit IF1, and is transmitted to the outside as output data Dout, for example.

As described above, in the neural network processor 100, the ROM (book) that requires a small hardware scale when the processing data (weight coefficient data) of the feature extraction layer, which is determined once the neural network model is determined, is converted into hardware. In the case of the embodiment, the processing data (weight coefficient data) of the determination layer, which is written to the first ROM 2) and whose data may be changed, is held in the RAM (second RAM 6 in the case of the present embodiment) that can rewrite the data. To do. Then, the neural network processor 100 executes the processing by the neural network in the above state. That is, in the neural network processor 100, the processing data (weight coefficient data) of the feature extraction layer that is not changed is held in the ROM that requires a small hardware scale, and the determination layer that may be changed. Since the processing data (weight coefficient data) is held in the RAM and the neural network processing is executed, it is possible to execute the high-performance neural network processing while suppressing the increase in the hardware scale.

Further, in the neural network processor 100, the weighting coefficient data determined according to the network configuration of the feature extraction layer is read out in the order required when executing the prediction processing (in the processing order at the time of the prediction processing). It is stored in a ROM (in the present embodiment, the first ROM 2). Therefore, in the neural network processor 100, when the prediction processing is executed, the inner product processing unit 7 simply inputs a single trigger signal (trigger signal Sig_trig output from the control unit 1 to the first ROM 2) to the first ROM 2. The weight coefficient data for the processing of the feature extraction layer can be acquired in the order required when the processing is executed. That is, in the neural network processor 100, the weighting coefficient data for processing the necessary feature extraction layer can be acquired in the processing order only by inputting a single trigger signal to the first ROM. Therefore, as in the conventional technique, the ROM. There is no need to specify complicated addresses for. In the above embodiment, the case where the data required for the processing of the feature extraction layer from the first ROM 2 is continuously transferred from the first ROM 2 to the inner product processing unit 7 by the single trigger signal will be described. However, it is not limited to this. For example, in the neural network processor 100, by designating one address of the first ROM2, the data required for the processing of the feature extraction layer from the first ROM2 is continuously transferred from the first ROM2 to the inner product processing unit in the processing order. Data may be transferred to 7.

Further, in the neural network processor 100, as described above, it is possible to specify the time for reading the data (weighting coefficient data) for processing of the feature extraction layer from the header data stored in the first ROM 2. Therefore, in the neural network processor 100, the data for processing of the determination layer (weighting coefficient data) is obtained from the second ROM 3 to the second RAM 6 based on the time for reading the processing data (weighting coefficient data) of the specified feature extraction layer. Can be transferred to. As a result, in the neural network processor 100, as soon as the processing of the feature extraction layer is completed, the data (weighting coefficient data) for the processing of the determination layer can be acquired (read) from the second RAM 6. Can be done. That is, in the neural network processor 100, as soon as the processing of the feature extraction layer is completed, the processing of the determination layer can be executed. As a result, in the neural network processor 100, the processing for the neural network can be speeded up (the time until the processing of the determination layer is completed can be shortened).

In the above description, the case where the data for processing the determination layer (weighting coefficient data) is stored and held in the second ROM 3 has been described, but the present invention is not limited to this, and the data for processing the determination layer (weight coefficient data) ( The weighting coefficient data) may be input to the neural network processor 100 from the outside. In this case, the neural network processor 100 executes the processing as follows. That is, the processing data (weighting coefficient data) of the determination layer is input to the first interface unit IF1 from the outside as data Din. Then, in the neural network processor 100, as described above, the data for processing the determination layer (weighting coefficient) is based on the time for reading the data for processing the specified feature extraction layer (weighting coefficient data). Data) is transferred in advance from the first interface unit IF1 to the second RAM 6. As a result, in the neural network processor 100, even when the data for processing the determination layer (weight coefficient data) is input from the outside in the neural network processor 100, as soon as the processing of the feature extraction layer is completed, The data (weight coefficient data) for processing of the determination layer can be acquired (read) from the second RAM 6.

That is, in the neural network processor 100, even when the data for processing the determination layer (weighting coefficient data) is input from the outside, the processing of the determination layer can be executed as soon as the processing of the feature extraction layer is completed. Can be in a state. As a result, in the neural network processor 100, the processing for the neural network can be speeded up (the time until the processing of the determination layer is completed can be shortened).

Further, since it is only necessary to store and hold only the data necessary for the determination process (processing of the determination layer) in the second RAM 6, the memory capacity of the second RAM 6 can be reduced.
Further, in the neural network processor 100,
(1) The first treatment of the feature extraction layer (for example, treatment of the first layer to the Nth layer) is executed.
(2) Execute the first judgment layer processing,
(3) The second processing of the feature extraction layer (for example, the processing of the first layer to the Nth layer) is executed.
(4) Execute the second judgment layer processing,
In this case, the data necessary for executing the first determination layer process is stored and held in the second RAM 6 before the first process (2), and the first determination layer process is executed before the first process (4). The data required for this is stored and held in the second RAM 6.

At this time, in the second RAM 6, only the data different from the data required for executing the first determination layer process among the data required for executing the second determination layer process (the next determination layer process) is updated. You just have to do it. As a result, in the neural network processor 100, the memory capacity of the second RAM 6 can be reduced, and the data transfer amount (the amount of data to be updated) of the second RAM 6 can be reduced.

≪First modification≫
Next, a first modification of the first embodiment will be described. The same parts as those in the above embodiment are designated by the same reference numerals, and detailed description thereof will be omitted.

FIG. 4 is a schematic configuration diagram of the neural network processor 100A according to the first modification of the first embodiment.

The neural network processor 100A of this modification has a configuration in which the extension unit 8 is added to the neural network processor 100 of the first embodiment. Other than that, the neural network processor 100A of this modification is the same as the neural network processor 100 of the first embodiment.

In the neural network processor 100A of this modification, the compressed data is written in the first ROM 2.

The decompression unit 8 reads the compressed data D1 stored and held in the first ROM 2, executes the decompression process on the compressed data D1, and acquires the data D_wij_feature after the decompression process. That is, the stretching unit 8 acquires the data D_wij_feature output from the first ROM 2 in the first embodiment by executing the stretching process. Then, the stretching unit 8 outputs the data D_wij_feature after the stretching process to the inner product processing section 7.

In the neural network processor 100A, the processing in the inner product processing unit 7 is executed by using the data D_wij_feature output from the decompression unit 8 instead of the data D_wij_feature output from the first ROM of the first embodiment. The processing in the inner product processing unit 7 is the same as that in the first embodiment.

In the neural network processor 100A of this modification, since the data of the feature extraction layer is stored and held in the first ROM 2 as compressed data, the memory capacity required in the first ROM 2 is further increased as compared with the first embodiment. Less. Therefore, in the neural network processor 100A of this modification, the hardware scale of the first ROM 2 can be further reduced. As a result, the neural network processor 100A of the present modification can further reduce the hardware scale as compared with the neural network processor 100 of the first embodiment.

[Other Embodiments]
Some or all of the functional parts of the neural network processors 100 and 100A may be realized by microcode or by predetermined hardware together with microcode.

Further, in the neural network processors 100 and 100A, as disclosed in the following Prior Art Document A, a neural network using a Binarized-DCNN (DCNN: Deep Convolution Natural Network) (hereinafter referred to as "BNN") is realized. It may be. In this case, the weighting coefficient data of the feature extraction layer realized by BNN may be stored in the ROM (for example, the first ROM 2) as uncompressed data or compressed data.
(Prior Art Document A):
Ryuji Kamiya et al. “Speeding up identification calculation and model compression by Binarized-DCNN” Shingaku Giho 116 (366), 47-52, 2016-12-15 Society of Electronics, Information and Communication Engineers In addition, in processors 100 and 100A for neural networks, You may implement a neural network (this is called "multi-value neural network") in which the binary base matrix obtained by the vector decomposition of BNN is used as the multi-value base matrix. In this case, the weighting coefficient data of the feature extraction layer realized by the multi-value neural network may be stored in the ROM (for example, the first ROM 2) as uncompressed data or compressed data.

In the above embodiment (including a modification), the case where the neural network processors 100 and 100A have a configuration in which the RAMs include the first RAM 5 and the second RAM 6 has been described, but the present invention is not limited thereto. In the neural network processors 100 and 100A, for example, the first RAM 5 and the second RAM 6 may be realized by one RAM.

Further, various data transfers and signal transmission / reception in the above-described embodiment (including modified examples) are not limited to the embodiments described above, and data transfer is performed by directly connected signal lines in each functional unit. , Signal transmission / reception may be executed, or data transfer and signal transmission / reception may be executed via a bus.

Further, in the above embodiment (including a modification), the number of ROMs and RAMs has been described as an example, but the present invention is not limited to this, and the number and arrangement of ROMs and RAMs are other than the above. You may. Further, the ROM and RAM may be provided outside the neural network processors 100 and 100A.

Each block (each functional unit) of the neural network processors 100 and 100A described in the above embodiment may be individually integrated into one chip by a semiconductor device such as an LSI, or one chip so as to include a part or all of them. It may be converted. Further, each block (each functional unit) of the neural network processor 100 described in the above embodiment may be realized by a semiconductor device such as a plurality of LSIs.

Although it is referred to as an LSI here, it may be referred to as an IC, a system LSI, a super LSI, or an ultra LSI depending on the degree of integration.

Further, the method of making an integrated circuit is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connection and settings of the circuit cells inside the LSI may be used.

In addition, a part or all of the processing of each functional block of each of the above embodiments may be realized by a program. Then, a part or all of the processing of each functional block of each of the above embodiments is performed by the central processing unit (CPU) in the computer. Further, the program for performing each process is stored in a storage device such as a hard disk or a ROM, and is read and executed in the ROM or the RAM.

Further, each process of the above embodiment may be realized by hardware, or may be realized by software (including a case where it is realized together with an OS (operating system), middleware, or a predetermined library). Further, it may be realized by mixed processing of software and hardware.

For example, when each functional unit of the above embodiment (including a modification) is realized by software, the hardware configuration (for example, CPU, GPU, ROM, RAM, input unit, output unit, etc.) shown in FIG. 5 is busted. (Hardware configuration connected by Bus) may be used to realize each functional unit by software processing.

Further, the execution order of the processing methods in the above-described embodiment is not necessarily limited to the description of the above-described embodiment, and the execution order can be changed without departing from the gist of the invention.

A computer program that causes a computer to perform the above-mentioned method and a computer-readable recording medium that records the program are included in the scope of the present invention. Here, examples of computer-readable recording media include flexible disks, hard disks, CD-ROMs, MOs, DVDs, DVD-ROMs, DVD-RAMs, large-capacity DVDs, next-generation DVDs, and semiconductor memories. ..

The computer program is not limited to the one recorded on the recording medium, and may be transmitted via a telecommunication line, a wireless or wired communication line, a network typified by the Internet, or the like.

Further, the wording "part" may be a concept including "circuitity". The circuit may be realized in whole or in part by hardware, software, or a mixture of hardware and software.

The functions of the elements disclosed herein are general purpose processors, dedicated processors, integrated circuits, ASICs, configured to perform the disclosed elements or programmed to perform the disclosed functions. It may be implemented using an application-specific integrated circuit ”), a conventional circuit configuration and / or a circuit configuration or processing circuit configuration including a combination thereof. A processor is considered as a processing circuit configuration or circuit configuration when it contains transistors and other circuit configurations. In the present disclosure, a circuit configuration, unit or means is hardware that performs the listed functions, or hardware that is programmed to perform such functions. The hardware may be any hardware disclosed herein or something else known that is programmed to perform the listed functions or configured to perform those functions. When the hardware is a processor that may be considered as a type of circuit configuration, the circuit configuration, means or unit is a combination of hardware and software, the software and / or processor used to configure the hardware. is there.

The specific configuration of the present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the gist of the invention.

100, 100A Neural network processor 1 Control unit 2 1st ROM
3 2nd ROM
5 1st RAM
6 2nd RAM
7 Inner product processing unit 8 Stretching unit

Claims (6)

A neural network processor for executing neural network processing including feature extraction layer processing and judgment layer processing.
A read-only memory that stores and holds data for processing in the feature extraction layer,
A random access memory that can read and write data and stores and holds data for processing in the determination layer.
A control unit that controls the read-only memory and the random access memory,
The data for processing the feature extraction layer is read from the read-only memory as the first data, the processing of the feature extraction layer is executed using the first data, and the processing of the determination layer is performed from the random access memory. Data for reading as second data, and using the second data, an inner product processing unit that executes the processing of the determination layer, and
A processor for neural networks. The read-only memory is
Header data including data for specifying the read time of data for processing of the feature extraction layer, and
The data for processing of the feature extraction layer is stored and held so that the data for processing of the feature extraction layer can be output in the order required when the processing of the feature extraction layer is executed.
When the control unit outputs a read command signal to the read-only memory, the read-only memory is required when executing the processing of the feature extraction layer with data for processing the feature extraction layer. Output in the order of
The processor for a neural network according to claim 1. The control unit
The header data stored in the read-only memory is read, and based on the header data, the time when the output processing of the data for processing of the feature extraction layer is completed is specified from the read-only memory.
Control so that the data for processing of the determination layer is stored and held in the random access memory at a time before the specified time.
The processor for a neural network according to claim 2. It also has a decompression section that executes decompression processing on the compressed data.
The read-only memory is
The data for processing of the feature extraction layer is stored and retained as compressed data.
The extension part
The decompression process is executed on the compressed data stored and held in the read memory, and the decompression process is executed.
The inner product processing unit
Using the data stretched by the stretched portion, the processing of the feature extraction layer is executed.
The processor for a neural network according to any one of claims 1 to 3. A neural network processor for executing neural network processing including feature extraction layer processing and judgment layer processing.
A read-only memory that stores and holds data for processing in the feature extraction layer,
A random access memory that can read and write data and stores and holds data for processing in the determination layer.
A control unit that controls the read-only memory and the random access memory,
It is a processing method for a neural network executed by using a processor for a neural network provided with.
The first step of reading the data for processing the feature extraction layer from the read-only memory as the first data and executing the processing of the feature extraction layer using the first data.
A second step of reading data for processing of the determination layer from the random access memory as second data and executing processing of the determination layer using the second data.
A processing method for a neural network including. A program for causing a computer to execute the processing method for a neural network according to claim 5.

Images