JP2020021208A - Neural network processor, neural network processing method, and program - Google Patents

Abstract

To realize a neural network processor capable of mounting a high-performance compact model in a low-specification device such as an embedded device or a mobile device without requiring learning.SOLUTION: The neural network processor configured to make a part common where similar processing is executed in a process of a convolution layer and a process of a fully connected layer, executes a combining process of a norm mode and an inner product operation mode, so that it can execute the process of a convolution layer and the process of a fully connected layer. Thus, the neural network processor can execute multi-valued neural network processing at high speed while being configured to suppress an increase in hardware scale.SELECTED DRAWING: Figure 1

Description

  The present invention relates to neural network technology.

  In recent years, various techniques using CNN (Convolutional Neural Network), which is one of neural network techniques, have been developed (for example, see Patent Document 1). Among CNNs, a technology using DCNN (Deep Convolutional Neural Network) having a large number of intermediate layers has attracted particular attention because it has achieved results in various fields.

JP 2015-197702 A

  DCNN achieves high recognition performance in various tasks such as general object recognition and semantic segmentation. On the other hand, the DCNN requires an extremely large amount of calculation and a large number of parameters to execute the processing, and therefore requires a huge processing time and a large memory when executing the processing.

  In the DCNN, the recognition accuracy tends to be improved by making the layer deeper, which causes a problem that the model size increases in addition to the identification time (processing time). In order to use DCNN in a low-spec device such as an embedded device or a mobile device, high-speed identification calculation and compression of a model size are major issues.

  That is, it is difficult to directly mount a learned model acquired by learning in a large-scale system on a low-spec device (for example, an edge terminal) such as an embedded device or a mobile device. It is necessary to build a model that has been developed.

  In order to mount a learned model learned and acquired by a large-scale system on a low-spec device such as an embedded device or a mobile device (for example, an edge terminal), the size of the learned model must be reduced in the low-spec device. It is necessary to construct a model that has been trained, and use the learning data used for the trained model to learn again in the compact model (this learning is referred to as “re-learning”).

  In other words, there is a problem that re-learning is required to mount a learned model learned and acquired by a large-scale system on a low-spec device (for example, an edge terminal) such as an embedded device or a mobile device.

  In view of the above problem, the present invention can mount a high-performance compact model in a low-spec device (eg, edge terminal) such as an embedded device or a mobile device without the need for re-learning. It is an object to realize a neural network processor, a neural network data processing method, and a program.

  In order to solve the above problem, a first invention is a neural network processor for executing a multivalued neural network process including a convolutional layer process and a fully connected layer process, comprising: , A quantization processing unit and an inner product processing unit.

  The control unit sets a scaling coefficient vector that is real number vector data, and sets a multi-value basis matrix having multi-value data as elements.

  The quantization processing unit performs a quantization process on the feature map input to the convolutional layer and the feature vector input to the fully connected layer. Further, the quantization processing unit sets an offset value such that the minimum value of the feature map and the minimum value of the feature vector are smaller than predetermined values, and based on the maximum value and the minimum value of the feature map and the feature vector. The quantization processing is executed using the quantization width acquired by the above.

  The inner product processing unit executes an inner product operation process using (1) a norm mode for calculating the norm of the feature map and the feature vector, and (2) a multi-valued base matrix and the feature map or the feature vector after the quantization process. And an inner product operation mode. The inner product processing unit executes the process of the convolutional layer and the process of the fully connected layer by executing the process of the norm mode and the process of combining the inner product operation mode.

  In this neural network processor, in the processing of the convolutional layer and the processing of the fully connected layer, a part where the same processing is executed is shared, and two modes ((1) norm mode and (2) inner product operation mode) are used. By executing the processing in which the processing is combined, it is possible to execute the processing of the convolutional layer and the processing of the fully connected layer. Therefore, this neural network processor can execute neural network processing at high speed while suppressing an increase in hardware scale.

A second invention is the first invention, wherein the inner product processing unit includes a microcode acquisition unit that acquires a norm mode microcode and an inner product operation mode microcode, and an arithmetic operation based on the microcode. An arithmetic operation processing unit that executes processing.
(1) When set to norm mode,
The microcode acquisition unit acquires the norm mode microcode, and the arithmetic operation processing unit executes an arithmetic operation process based on the norm mode microcode.
(2) When the inner product calculation mode is set,
The microcode acquisition unit acquires the inner product operation mode microcode, and the arithmetic operation processing unit executes an arithmetic operation process based on the inner product operation mode microcode.

  In this neural network processor, in the processing of the convolutional layer and the processing of the fully connected layer, a part where the same processing is executed is shared, and two modes ((1) norm mode and (2) inner product operation mode) are used. The processing is executed by processing with microcode corresponding to each mode. In this neural network processor, the processing of different parts in the processing of the convolutional layer and the processing of the fully connected layer is realized by combining the above two modes of processing in an appropriate order. Therefore, this neural network processor can execute neural network processing at high speed while suppressing an increase in hardware scale.

A third invention is the first or the second invention, wherein the inner product processing unit executes processing of the convolutional layer,
(1) The norm mode processing is repeatedly executed for the number of feature maps of the convolutional layer to be processed,
(2) Each time the norm mode processing is performed for each feature map, the process in the inner product calculation mode is repeatedly executed for the number of outputs of the convolutional layer to be processed.

  This allows the neural network processor to execute the processing of the convolutional layer by combining the processing in the two modes.

A fourth invention is the invention according to any one of the first to third inventions, wherein the inner product processing unit executes a process of a fully connected layer,
(1) The norm mode process is executed once for all connected layers to be processed,
(2) The process in the inner product calculation mode is repeatedly executed for the number of outputs of all the connected layers to be processed.

  As a result, the neural network processor can execute the processing of the fully connected layer by combining the processing in the two modes.

  A fifth invention is a neural network processing method for executing a multivalued neural network process including a convolutional layer process and a fully connected layer process, comprising: a control step; a quantization processing step; Inner product processing step.

  The control step sets a scaling coefficient vector, which is real vector data, and sets a multi-value basis matrix having multi-value data as elements.

  In the quantization step, a quantization process is performed on the feature map input to the convolutional layer and the feature vector input to the fully connected layer. Further, the quantization processing step sets an offset value such that a minimum value of the feature map and a minimum value of the feature vector are smaller than predetermined values, and sets a maximum value and a minimum value of the feature map and the feature vector. Is performed using the quantization width acquired based on the.

  The inner product processing step executes (1) a norm mode for calculating the norm of the feature map and the feature vector, and (2) an inner product calculation process using the multi-valued basis matrix and the feature map or the feature vector after the quantization process. And an inner product operation mode. The inner product processing step executes the process of the norm mode and the process of combining the inner product operation mode, thereby executing the processing of the convolutional layer and the processing of the fully connected layer.

  This makes it possible to realize a neural network processing method having the same effects as the first invention.

  A sixth invention is a program for causing a computer to execute the neural network processing method according to the fifth invention.

  Thereby, it is possible to realize a program for causing a computer to execute a neural network processing method having the same effects as the first invention.

  According to the present invention, a neural network processor capable of mounting a high-performance compact model in a low-spec device (for example, an edge terminal) such as an embedded device or a mobile device without requiring re-learning, A neural network processing method and a program can be realized.

FIG. 1 is a schematic configuration diagram of a binary neural network processor 100 according to a first embodiment. The schematic block diagram of the inner product processing part 3 which concerns on 1st Embodiment. FIG. 9 is a diagram for explaining processing in an Offset mode. FIG. 7 is a diagram for explaining processing in a Norm mode. FIG. 4 is a diagram for explaining processing in a DP mode (inner product operation processing mode). FIG. 2 is a diagram showing a CPU bus configuration.

[First Embodiment]
The first embodiment will be described below with reference to the drawings.

<1.1: Configuration of Binary Neural Network Processor>
FIG. 1 is a schematic configuration diagram of a binary neural network processor 100 according to the first embodiment.

  FIG. 2 is a schematic configuration diagram of the inner product processing unit 3 according to the first embodiment.

  As shown in FIG. 1, the binary neural network processor 100 includes an input / output interface IF1, a control unit CPU1, an arithmetic processing unit PL1, and a bus B1. The input / output interface IF1, the control unit CPU1, and the arithmetic processing unit PL1 are connected by a bus B1, as shown in FIG. 1, and input and output necessary data, commands, and the like via the bus B1. be able to. Note that some or all of the functional units may be directly connected as necessary, instead of being connected to a bus.

  The input / output interface IF1 receives data Din to be processed from the outside, and outputs data including a processing result by the binary neural network processor to the outside as data Dout.

  The control unit CPU1 performs the overall control of the binarized neural network processor 100, the control of each function unit, and the processing necessary for the binarized neural network processing. The control unit CPU1 is realized by a CPU (Central Processing Unit) or a CPU core.

  For example, the control unit CPU1 acquires (sets) a scaling coefficient vector v_c and a binary basis matrix M that approximate parameters (weighted data) of a learned model in a large-scale system, and acquires (sets) the acquired (set) scaling coefficient vector v_c. And the binary basis matrix M are stored and held in the area CV and the area BinMtx0 / 1 of the internal RAM R1, respectively.

  The scaling coefficient vector v_c and the binary basis matrix M may be input from the outside to the binary neural network processor 100 via the input / output interface IF1.

  As shown in FIG. 1, the arithmetic processing unit PL1 includes a DMA control unit 1, a quantization processing unit 2, an inner product processing unit 3, and an internal RAM R1.

  The DMA control unit 1 performs a DMA transfer process (DMA: Direct Memory Access).

  The quantization processing unit 2 performs a quantization process on data of a feature map which is an input of a convolutional layer of DCNN (Deep Convolution Neural Network). Further, the quantization processing unit 2 performs a quantization process on the input data of the DCNN in all the connection layers.

  As shown in FIG. 2, the inner product processing unit 3 includes an AND processing unit 31, a selector 32, a count processing unit 33, a microcode acquisition unit 34, and an ALU 35 (ALU: Arithmetic Logic Unit).

  The AND processing unit 31 includes data D2 (for example, data of an integer part of a weight vector obtained from the internal RAM R1 (area BinMtx0 / 1)) and data D3 (for example, data obtained from the internal RAM R1 (area BinInT)). , Which is bit-separated and further subjected to a quantization process), performs an AND process on the data D2 and the data D3, and outputs the data including the execution result to the selector 32 as data D4. Output.

  The selector 32 inputs data D2 and data D4 and a signal dp_mode indicating a mode. The selector 32 selects one of the data D2 and the data D4 based on the signal dp_mode, and outputs the selected data to the count processing unit 33 as data D5.

  The count processing unit 33 receives the data D5 output from the selector 32 and performs a count process on the data D5. Then, the count processing unit 33 outputs the data including the processing result to the ALU 35 as data D6.

The microcode acquisition unit 34 acquires a microcode μCode (for example, a microcode corresponding to a mode), and outputs the acquired microcode μCode to the ALU 35. As the mode, for example, (1) Offset mode, (2) Norm mode, and (3) DP mode are set.
In the “Offset mode”, when performing a quantization process on a feature map input to the convolutional layer and a feature vector input to the fully connected layer, the minimum value of the feature map and the minimum value of the feature vector are This is a mode for executing a process of acquiring an offset value set to be smaller than a predetermined value.
The “Norm mode” is a mode for executing a process of calculating a norm of a feature map and a feature vector.
The “DP mode” is a mode for executing an inner product calculation process using a multivalued basis matrix and a feature map or a feature vector after the quantization process.

  The ALU 35 includes data D1 (for example, data of a real part (scale coefficient vector) of a weight vector obtained from the internal RAM R1 (area CV)), data D6 output from the count processing unit 33, and a microcode obtaining unit. The microcode μCode output from 34 is input. The ALU 35 performs an arithmetic operation based on the microcode μCode, and outputs data including the result of the arithmetic operation as data Do.

  The internal RAM R1 is a RAM (Random Access Memory) for storing and holding data necessary for executing the processing for the binarized neural network.

<1.2: Operation of Binary Neural Network Processor>
The operation of the binary neural network processor 100 configured as described above will be described below.

  Generally, the CNN includes an input layer, a convolution layer (convolution layer), and a fully connected layer. For example, image data is input as input data Din to the input / output interface IF1 of the binarized neural network processor 100, image recognition processing is performed by the CNN, and the image recognition processing result is output to the outside as output data Dout. You.

  In the processing of the convolutional layer or the processing of the fully connected layer, the CNN performs a weight calculation process on input data, and activates a function (eg, a ramp function (ReLU: Restricted Linear Unit)) on the processing result. , Sigmoid function, Softmax function, etc.), the output of the convolutional layer or the fully connected layer is obtained.

Also, as disclosed in the following prior art document A, Binarized-DCNN (Deep Convolution Neutral Network (DCNN)) (hereinafter referred to as “BNN”) introduces a binary decomposition of a Quantization sub-layer and a coupling coefficient. By replacing the inner product calculation between real numbers with the inner product calculation between binary values, it is possible to speed up identification calculation and reduce the model size without re-learning an existing network model. The operation between the binary values of BNN can be performed at high speed by a logical operation such as XOR or AND and a bit count.
(Prior art document A):
Ryuji Kamiya “High-speed Classification Calculation and Model Compression Using Binarized-DCNN” IEICE Technical Report 116 (366), 47-52, 2016-12-15 A basic expression for the BNN identification calculation can be derived as in the following (Equation 1).
(Equation 1):
y _ijn = c _n ^T M _n ^T B _ij r _ij + min (x) Offset
y _ijn : output of the n-th feature map (output value of coordinates (i, j) of the feature map)
c _n ^T : transposed matrix of scale coefficient vector c _n of n-th feature map M _n ^T : transposed matrix of binary basis matrix of n-th feature map B _ij r _ij : binary feature map (two- _dimensional map after quantization) Value feature map)
min (x): the minimum value of the values of the elements of the n-th feature map Offset: the result obtained in the Offset mode Also, M _n ^T {-1,1} and B _ij r _ij −1 , 1} are binary, and can be calculated by a logical operation and a bit count using the following (Equation 2).
(Equation 2):
M _n ^T B _ij r _ij
= 2 × BITCNT (AND (M _n ^T , B _ij r _ij )) − Norm (z)
z = B _ij r _ij
Norm (z): a function for acquiring the norm of z BITCNT (x): a function for counting the number of bits of “1” in the binary code x In the binary neural network processor 100, the processing of the convolutional layer In the processing of the connection layer, by sharing a part where the same processing is executed, high-speed processing is realized while suppressing an increase in hardware scale.

  Hereinafter, the operation of the binary neural network processor 100 will be described by dividing into “processing of the convolutional layer” and “processing of the fully connected layer”.

  FIG. 3 is a diagram for explaining the process in the Offset mode.

  FIG. 4 is a diagram for explaining processing in the Norm mode.

  FIG. 5 is a diagram for explaining processing in the DP mode (inner product operation processing mode).

  In the binarized neural network processor 100, processing is performed using three modes: (1) Offset mode, (2) Norm mode, and (3) DP mode.

(1.2.1: Convolutional layer processing)
First, the processing of the convolution layer will be described.

The quantization processing unit 2 of the binarized neural network processor 100 _calculates the quantization width Δd between the maximum value and the minimum value in the m-th (m: natural number) feature map z ^l _ijm in the l-th layer (1: natural number). To
^{_{Δd = {max (z l ijm}} ) -min (z l ijm)} / (2 Q -1)
max (x): function for acquiring the maximum value of x min (x): function for acquiring the minimum value of x Q: acquired as the number of quantization bits.

Then, the quantization processing unit 2 shifts the value so that the minimum value of the feature map becomes 0. That is, the quantization processing unit 2
^{_{z l ijm '= {z l}} ijm -min (z l ijm)} / Q
Is performed, and the value obtained by the above equation is rounded off, rounded to an integer value, and quantized. Further, the quantization processing unit 2 obtains a binary code z ^l _ijm ^(b) {0, 1} by _{performing a binarization} process on the value obtained by the rounding quantization.

The binary code z ^l _ijm ^(b) {0, 1} (the feature map B _ij r _ij after the quantization process ⁾ acquired as described above is stored and held in the area BinInT of the internal RAM.

In the processing of the convolutional layer, the following holds.
(1) The feature map B _ij r _ij after the quantization process changes (replaces) for each feature map.
(2) The second term on the right side of (Equation 1), that is, the value of min (x) Offset is constant regardless of the feature map.

In consideration of the above, the binarized neural network processor 100 executes the processing of the convolutional layer by processing corresponding to the following pseudo code.
擬 似 Pseudo code for processing of convolutional layer≫
For (number of outputs)
Operate_offset (); // Offset restoration processing
For (number of feature maps)
Operate_Norm (); // Calculation of norm Processing equivalent to the second term on the right side of (Equation 2)
For (number of outputs)
Operate_dp (); // Dot product calculation Binary neural network processor 100
(1) The above offset restoration process is executed in the process of the Offset mode,
(2) The above-described norm calculation process is executed in a Norm mode process,
(3) The above inner product calculation process is executed in the DP mode (inner product operation processing mode) process.

  Hereinafter, this will be described.

(1.2.1.1: Offset mode processing (convolution layer processing))
The processing in the Offset mode will be described.

  As shown in FIG. 3, the data D2 is input to the selector 32.

  The selector 32 receives the mode signal dp_mode whose signal value is set to “0”. The selector 32 selects the data D2 based on the mode signal dp_mode and outputs the data D2 to the count processing unit 33 as data D5. I do.

  The count processing unit 33 outputs the input data D5 as it is to the ALU 35 as the data D6 in the Offset mode.

The microcode acquisition unit 34 acquires the microcode μCode (Offset_mode) for the Offset mode, and outputs it to the ALU 35. Note that the microcode μCode (Offset_mode) for the Offset mode is, for example, a code that causes the ALU 35 to execute the following processing.
(1) Loading (reading) of min (x)
(2) Multiplication process of data D1 (= c _n ^T ), data D6 (= M _n ^T ), and min (x) Note that min (x) was obtained when the quantization process was performed. For example, the value may be stored and held in the internal RAM R1, and the microcode acquisition unit 34 may read the min (x) data from the internal RAM R1.

The ALU 35 receives data D1 (= c _n ^T ) and data D6 (= M _n ^T ), as shown in FIG. The data D1 (= c _n ^T ) is the scale coefficient vector data c _n ^T stored and held in the area CV of the internal RAM.

  Also, the ALU 35 receives the microcode μCode (Offset_mode) for the Offset mode output from the microcode acquisition unit 34 and performs an operation according to the microcode μCode (Offset_mode) for the Offset mode.

In other words, ALU 35
(1) Loading (reading) of min (x)
(2) Output data Do (= min (x) Offset) is obtained by performing a multiplication process of data D1 (= c _n ^T ), data D6 (= M _n ^T ), and min (x). I do.

  By performing the processing as described above, the second term on the right side of (Equation 1), that is, the value (offset value) of min (x) Offset can be obtained.

  In the processing of the convolutional layer, the above processing (offset restoration processing) is executed for the number of outputs of the convolutional layer.

(1.2.1.2: Norm mode processing (convolutional layer processing))
The processing in the Norm mode will be described.

As shown in FIG. 4, data D3 (= B _ij r _ij ) is input to the AND processing unit 31. The data D3 is a feature map B _ij r _ij after the quantization process, and is stored and held in the area BinInT of the internal RAM.

  In the Norm mode, the AND processing unit 31 outputs the input data D3 to the selector 32 as it is as data D4.

  The selector 32 receives the mode signal dp_mode whose signal value is set to “1”. The selector 32 selects the data D4 based on the mode signal dp_mode, and outputs the data D4 to the count processing unit 33 as data D5. I do.

Counting processing unit 33 executes a counting process (process by BITCNT function) with respect to the input data D5, and outputs the processing result data _{D6 (= BITCNT (B ij r} ij)) as the ALU35.

  The microcode acquisition unit 34 acquires the microcode μCode (Norm_mode) for the Norm mode, and outputs it to the ALU 35. Note that the Norm mode microcode μCode (Norm_mode) is a code that causes the ALU 35 to execute a process of directly outputting the data input from the count processing unit 33.

The ALU 35 receives the data D6 (= BITCNT (B _ij r _ij )) output from the count processing unit 33, as shown in FIG.

  Further, the ALU 35 receives the Norm mode microcode μCode (Norm_mode) output from the microcode acquisition unit 34 and performs an operation according to the Norm mode microcode μCode (Norm_mode).

That is, the ALU 35 performs a process of directly outputting the data input from the count processing unit 33, and outputs data Do (= BITCNT (B _ij r _ij )). BITCNT (B _ij r _ij ) corresponds to the norm of the feature map B _ij r _ij after the quantization processing.

By performing the processing as described above, the second term on the right side of (Equation 2), that is, the value (norm) of Norm (z) (z = B _ij r _ij ) can be obtained.

  In the processing of the convolutional layer, the above processing (norm calculation processing) is executed for the number of feature maps of the convolutional layer to be processed.

(1.2.1.3: DP mode processing (convolutional layer processing))
The processing in the DP mode will be described.

As shown in FIG. 5, data D2 (= M _n ^T ) and data D3 (= B _ij r _ij ) are input to the AND processing unit 31.

The data D2 is data M _n ^T of a binary basis matrix stored and held in the area BinMtx0 / 1 of the internal RAM.

The data D3 is a feature map B _ij r _ij after the quantization processing, and is stored and held in the area BinInT of the internal RAM.

The AND processing unit 31 performs an AND process on the data D2 and the data D3, and outputs the data including the processing result to the selector 32 as data D4 (= AND (M _n ^T , B _ij r _ij )). Note that the AND process is a process in which, when the value of an element is “−1”, the “−1” is replaced with “0” to obtain a logical product.

  The selector 32 receives the mode signal dp_mode whose signal value is set to “1”. The selector 32 selects the data D4 based on the mode signal dp_mode, and outputs the data D4 to the count processing unit 33 as data D5. I do.

Counting processing unit 33 executes a counting process (process by BITCNT function) with respect to the input data D5, the processing result data ^{_{D6 (= BITCNT (AND (M}} n T, B ij r ij))) as an output in ALU35 I do.

The microcode obtaining unit 34 obtains the microcode μCode (DP_mode) for the DP mode and outputs it to the ALU 35. The microcode μCode (DP_mode) for the DP mode is, for example, a code for causing the ALU 35 to execute the following processing.
(1) D6 × 2 processing (processing to shift one bit to the left)
(2) Processing of subtracting the norm from the result of (1) (3) Processing of multiplying the result of (2) by data D1 (= c _n ^T ) The ALU 35, as shown in FIG. _{(= c} ^{n T)} and data ^{_{D6 (= BITCNT (aND (M}} n T, B ij r ij))) inputs the. The data D1 (= c _n ^T ) is the scale coefficient vector data c _n ^T stored and held in the area CV of the internal RAM.

  Further, the ALU 35 receives the microcode μCode (DP_mode) for the DP mode output from the microcode acquisition unit 34 and performs an operation according to the microcode μCode (DP_mode) for the DP mode.

In other words, ALU 35
(1) D6 × 2 processing (processing to shift one bit to the left)
(2) Processing for subtracting the norm from the result of (1) (2 × D6-Norm (z))
(3) A process of multiplying the result of (2) by data D1 (= c _n ^T ) to obtain output data Do (= c _n ^T M _n ^T B _ij r _ij ).

That is, according to the above, processing corresponding to the following is executed.
Do = c _n ^T M _n ^T B _ij r _{ij
^{M n T B ij r ij =}} 2 × BITCNT (AND (M n T, B ij r ij)) - Norm (z)
By performing the processing as described above, the first term on the right side of the above (Equation 1), that is, the value of c _n ^T M _n ^T B _ij r _ij can be obtained.

  In the processing of the convolutional layer, the above-described processing (inner product calculation processing) is executed for each feature map of the convolutional layer to be processed by the number of outputs of the convolutional layer. The above processing result is, for example, stored and held in a predetermined area of the internal RAM R1, or output to the control unit CPU1, and the control unit CPU1 executes a predetermined process using the processing result.

By performing the processing as described above, the binarized neural network processor 100 can execute the processing of the convolutional layer. That is, in the binarized neural network processor 100, the data required to obtain y _ijn of (Equation 1) can be obtained by the processing in the above three modes, and as a result, the processing of the convolutional layer Can be performed.

(1.2.2: Processing of all bonding layers)
Next, the processing of the entire bonding layer will be described.

The quantization processing unit 2 of the binarized neural network processor 100 calculates the quantization width Δd between the maximum value and the minimum value in the input vector z ^l _i to the l-th fully connected layer,
_{^{Δd = {max (z l i}} ) -min (z l i)} / (2 Q -1)
max (x): function for acquiring the maximum value of x min (x): function for acquiring the minimum value of x Q: acquired as the number of quantization bits.

Then, the quantization processing unit 2 shifts the value so that the minimum value of the input vector to the fully connected layer becomes zero. That is, the quantization processing unit 2
^{_{z l i '= {z l}} i -min (z l i)} / Q
Is performed, and the value obtained by the above equation is rounded off, rounded to an integer value, and quantized. Furthermore, the quantization processing unit 2, to the obtained value by the quantization rounding, by binarization processing, to obtain a binary code ^{_{z l i (b) ∈ {}} 0,1}.

The acquired binary code _{^{z l i (b) ∈ {}} 0,1} ( feature vector _B ij _{r ij} after quantization process) as is stored and held in the area BinInT internal RAM.

In the processing of the entire bonding layer, the following holds.
(1) There is only one feature vector after the quantization processing.

In consideration of the above, the binarized neural network processor 100 executes the processing of all the connected layers by processing corresponding to the following pseudo code.
擬 似 Pseudo code for processing all connected layers≫
Operate_Norm (); // Calculation of norm Processing equivalent to the second term on the right side of (Equation 2)
For (number of outputs)
Operate_offset (); // Offset restoration processing
Operate_dp (); // Dot product calculation Binary neural network processor 100
(1) The above-described norm calculation process is executed in a Norm mode process,
(2) The above-described offset restoration processing is executed as processing in the Offset mode,
(3) The above inner product calculation process is executed in the DP mode (inner product operation processing mode) process.

  Hereinafter, this will be described.

(1.2.2.1: Norm mode processing (processing of all connected layers))
The processing in the Norm mode will be described.

As shown in FIG. 4, data D3 (= B _ij r _ij ) is input to the AND processing unit 31. Note that the data D3 is the feature vector B _ij r _ij after the quantization process, and is stored and held in the area BinInT of the internal RAM.

  In the Norm mode, the AND processing unit 31 outputs the input data D3 to the selector 32 as it is as data D4.

  The selector 32 receives the mode signal dp_mode whose signal value is set to “1”. The selector 32 selects the data D4 based on the mode signal dp_mode, and outputs the data D4 to the count processing unit 33 as data D5. I do.

Counting processing unit 33 executes a counting process (process by BITCNT function) with respect to the input data D5, and outputs the processing result data _{D6 (= BITCNT (B ij r} ij)) as the ALU35.

  The microcode acquisition unit 34 acquires the microcode μCode (Norm_mode) for the Norm mode, and outputs it to the ALU 35. Note that the Norm mode microcode μCode (Norm_mode) is a code that causes the ALU 35 to execute a process of directly outputting the data input from the count processing unit 33.

The ALU 35 receives the data D6 (= BITCNT (B _ij r _ij )) output from the count processing unit 33, as shown in FIG.

  Further, the ALU 35 receives the Norm mode microcode μCode (Norm_mode) output from the microcode acquisition unit 34 and performs an operation according to the Norm mode microcode μCode (Norm_mode).

That is, the ALU 35 performs a process of directly outputting the data input from the count processing unit 33, and outputs data Do (= BITCNT (B _ij r _ij )). BITCNT (B _ij r _ij ) corresponds to the norm of the feature vector B _ij r _ij after the quantization processing.

By performing the processing as described above, the second term on the right side of (Equation 2), that is, the value (norm) of Norm (z) (z = B _ij r _ij ) can be obtained.

  In the processing of the all connected layers, the above processing (norm calculation processing) is executed once for all the connected layers to be processed.

(1.2.2.2: Offset mode processing (processing of all connected layers))
The processing in the Offset mode will be described.

  As shown in FIG. 3, the data D2 is input to the selector 32.

  The selector 32 receives the mode signal dp_mode whose signal value is set to “0”. The selector 32 selects the data D2 based on the mode signal dp_mode and outputs the data D2 to the count processing unit 33 as data D5. I do.

  The count processing unit 33 outputs the input data D5 as it is to the ALU 35 as the data D6 in the Offset mode.

The microcode acquisition unit 34 acquires the microcode μCode (Offset_mode) for the Offset mode, and outputs it to the ALU 35. Note that the microcode μCode (Offset_mode) for the Offset mode is, for example, a code that causes the ALU 35 to execute the following processing.
(1) Loading (reading) of min (x)
(2) Multiplication process of data D1 (= c _n ^T ), data D6 (= M _n ^T ), and min (x) Note that min (x) was obtained when the quantization process was performed. For example, the value may be stored and held in the internal RAM R1, and the microcode acquisition unit 34 may read the min (x) data from the internal RAM R1.

The ALU 35 receives data D1 (= c _n ^T ) and data D6 (= M _n ^T ), as shown in FIG. The data D1 (= c _n ^T ) is the scale coefficient vector data c _n ^T stored and held in the area CV of the internal RAM.

  Also, the ALU 35 receives the microcode μCode (Offset_mode) for the Offset mode output from the microcode acquisition unit 34 and performs an operation according to the microcode μCode (Offset_mode) for the Offset mode.

In other words, ALU 35
(1) Loading (reading) of min (x)
(2) Output data Do (= min (x) Offset) is obtained by performing a multiplication process of data D1 (= c _n ^T ), data D6 (= M _n ^T ), and min (x). I do.

  By performing the processing as described above, the second term on the right side of (Equation 1), that is, the value (offset value) of min (x) Offset can be obtained.

  In the process of the fully connected layer, the above process (offset restoration process) is executed for the number of outputs of the fully connected layer.

(1.2.2.3: DP mode processing (processing of all coupled layers))
The processing in the DP mode will be described.

As shown in FIG. 5, data D2 (= M _n ^T ) and data D3 (= B _ij r _ij ) are input to the AND processing unit 31.

The data D2 is data M _n ^T of a binary basis matrix stored and held in the area BinMtx0 / 1 of the internal RAM.

The data D3 is a feature map B _ij r _ij after the quantization processing, and is stored and held in the area BinInT of the internal RAM.

The AND processing unit 31 performs an AND process on the data D2 and the data D3, and outputs the data including the processing result to the selector 32 as data D4 (= AND (M _n ^T , B _ij r _ij )). Note that the AND process is a process in which, when the value of an element is “−1”, the “−1” is replaced with “0” to obtain a logical product.

  The selector 32 receives the mode signal dp_mode whose signal value is set to “1”. The selector 32 selects the data D4 based on the mode signal dp_mode, and outputs the data D4 to the count processing unit 33 as data D5. I do.

Counting processing unit 33 executes a counting process (process by BITCNT function) with respect to the input data D5, the processing result data ^{_{D6 (= BITCNT (AND (M}} n T, B ij r ij))) as an output in ALU35 I do.

The microcode obtaining unit 34 obtains the microcode μCode (DP_mode) for the DP mode and outputs it to the ALU 35. The microcode μCode (DP_mode) for the DP mode is, for example, a code for causing the ALU 35 to execute the following processing.
(1) D6 × 2 processing (processing to shift one bit to the left)
(2) Processing of subtracting the norm from the result of (1) (3) Processing of multiplying the result of (2) by data D1 (= c _n ^T ) The ALU 35, as shown in FIG. _{(= c} ^{n T)} and data ^{_{D6 (= BITCNT (aND (M}} n T, B ij r ij))) inputs the. The data D1 (= c _n ^T ) is the scale coefficient vector data c _n ^T stored and held in the area CV of the internal RAM.

  Further, the ALU 35 receives the microcode μCode (DP_mode) for the DP mode output from the microcode acquisition unit 34 and performs an operation according to the microcode μCode (DP_mode) for the DP mode.

In other words, ALU 35
(1) D6 × 2 processing (processing to shift one bit to the left)
(2) Processing for subtracting the norm from the result of (1) (2 × D6-Norm (z))
(3) A process of multiplying the result of (2) by data D1 (= c _n ^T ) to obtain output data Do (= c _n ^T M _n ^T B _ij r _ij ).

That is, according to the above, processing corresponding to the following is executed.
Do = c _n ^T M _n ^T B _ij r _{ij
^{M n T B ij r ij =}} 2 × BITCNT (AND (M n T, B ij r ij)) - Norm (z)
By performing the processing as described above, the first term on the right side of the above (Equation 1), that is, the value of c _n ^T M _n ^T B _ij r _ij can be obtained.

  In the processing of the fully connected layer, the above processing (inner product calculation processing) is executed for the number of outputs of the fully connected layer. The above processing result is, for example, stored and held in a predetermined area of the internal RAM R1, or output to the control unit CPU1, and the control unit CPU1 executes a predetermined process using the processing result.

By performing the processing as described above, the binarized neural network processor 100 can execute the processing of all the connected layers. That is, in the binarized neural network processor 100, data necessary for obtaining y _ijn of (Equation 1) can be obtained by the processing in the above three modes, and as a result, Processing can be performed.

  As described above, in the binarized neural network processor 100, in the processing of the convolutional layer and the processing of the fully connected layer, the part where the same processing is executed is shared, and the three modes ((1) Offset mode, The processing of (2) Norm mode and (3) DP mode) is executed by processing using microcode corresponding to each mode. In the binarized neural network processor 100, the processing of the different parts in the processing of the convolutional layer and the processing of the fully connected layer is realized by combining the above three modes of processing in an appropriate order. Therefore, the binary neural network processor 100 can execute BNN processing at high speed while suppressing an increase in hardware scale.

[Other embodiments]
In the above embodiment, the case where the binarized neural network processor 100 converts the binarized data from js has been described, but the present invention is not limited to this, and the method of the present invention is applied to multi-valued data. The present invention may be applied to realize a multi-valued neural network processor.

  Also, in the above embodiment, the case where the binarized neural network processor 100 executes processing in three modes ((1) Offset mode, (2) Norm mode, and (3) DP mode) has been described. It is not limited to this. For example, the binarized neural network processor 100 executes processing in (1) Norm mode and (2) DP mode, and the processing in the DP mode includes the processing in the Offset mode described in the above embodiment. You may do so. In addition, the binarized neural network processor 100 obtains a value obtained in the Offset mode by calculation in advance, holds the obtained value, and executes processing using the value when executing the DP mode. It may be. Thereby, in the binarized neural network processor 100, the processing in the Norm mode and the processing in the DP mode can be executed continuously without intervention of the CPU.

  In the above embodiment, the case where the inner product processing unit executes a part of the process of the BNN has been described. However, the present invention is not limited to this. For example, in the inner product processing unit 3 of the arithmetic processing unit PL1, the activation function Processing (for example, processing of a ReLU function) may be executed. The processing of the activation function (for example, the processing of the ReLU function) may be executed by the inner product processing unit 3 and the control unit CPU1.

  In the above embodiment, the number of internal RAMs has been described without particular limitation. However, the internal RAM may be configured by a plurality of RAMs, or may be an external RAM of a binary neural network processor. The processing of the above-described embodiment may be executed using a RAM (for example, a DRAM) provided in the CPU.

  In the above embodiment, the data represented by the scalar, the vector, and the matrix are examples, and are not limited to the above. In accordance with the processing of the BNN, the binarized neural network processor 100 may execute the same processing as described above as scalar, vector, and tensor data.

  Each block (each functional unit) of the binarized neural network processor 100 described in the above embodiment may be individually formed into a single chip by a semiconductor device such as an LSI, or may be configured to include a part or the entirety. It may be made into a chip. Each block (each functional unit) of the binary neural network processor 100 described in the above embodiment may be realized by a semiconductor device such as a plurality of LSIs.

  Here, the LSI is used, but depending on the degree of integration, it may be called an IC, a system LSI, a super LSI, or an ultra LSI.

  Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure connection and setting of circuit cells inside the LSI may be used.

  Further, a part or all of the processing of each functional block in each of the above embodiments may be realized by a program. Part or all of the processing of each functional block in each of the above embodiments is performed by a central processing unit (CPU) in a computer. Further, programs for performing the respective processes are stored in a storage device such as a hard disk or a ROM, and are executed by being read from the ROM or from the RAM.

  Further, each process of the above embodiment may be realized by hardware, or may be realized by software (including a case where it is realized together with an OS (Operating System), middleware, or a predetermined library). Further, it may be realized by mixed processing of software and hardware.

  For example, when each functional unit of the above-described embodiment (including modified examples) is implemented by software, the hardware configuration (for example, a CPU, a ROM, a RAM, an input unit, and an output unit) illustrated in FIG. Each functional unit may be realized by software processing using a connected hardware configuration).

  Further, the execution order of the processing method in the above embodiment is not necessarily limited to the description of the above embodiment, and the execution order can be changed without departing from the gist of the invention.

  A computer program that causes a computer to execute the above-described method and a computer-readable recording medium that records the program are included in the scope of the present invention. Here, examples of the computer-readable recording medium include a flexible disk, hard disk, CD-ROM, MO, DVD, DVD-ROM, DVD-RAM, large-capacity DVD, next-generation DVD, and semiconductor memory. .

  The computer program is not limited to one recorded on the recording medium, and may be transmitted via a telecommunication line, a wireless or wired communication line, a network represented by the Internet, or the like.

  In addition, the word “part” may be a concept including “circuitry”. The circuit may be realized in whole or in part by hardware, software, or a mixture of hardware and software.

  The specific configuration of the present invention is not limited to the above embodiment, and various changes and modifications can be made without departing from the spirit of the invention.

100 Binary Neural Network Processor PL1 Arithmetic Processing Unit 1 DMA Control Unit 2 Quantization Processing Unit R1 Internal RAM
3 Inner product processing unit 34 Microcode acquisition unit 35 ALU

Claims (6)

A neural network processor for executing a multivalued neural network process including a convolutional layer process and a fully connected layer process,
A control unit that sets a scaling coefficient vector that is real number vector data, and sets a multi-value basis matrix having multi-value data as an element;
A quantization processing unit that performs a quantization process on the feature map input to the convolutional layer and the feature vector input to the fully connected layer, the minimum value of the feature map and the minimum value of the feature vector Is set to an offset value smaller than a predetermined value, and the quantization process is performed using a quantization width obtained based on the maximum value and the minimum value of the feature map and the feature vector. The quantization processing unit;
(1) executing a norm mode for calculating a norm of the feature map and the feature vector; and (2) performing an inner product calculation process using the multi-valued base matrix and the feature map or the feature vector after the quantization process. An inner product processing unit that performs processing of the norm mode, and processing of the combination of the inner product operation modes, thereby performing the processing of the convolutional layer and the processing of the fully connected layer. When,
A neural network processor comprising: The inner product processing unit,
A microcode acquisition unit for acquiring the norm mode microcode and the inner product operation mode microcode,
An arithmetic operation processing unit that performs arithmetic operation processing based on microcode;
With
(1) When set to norm mode,
The microcode acquisition unit acquires the norm mode microcode,
The arithmetic processing unit executes the arithmetic processing based on the norm mode microcode,
(2) When the inner product calculation mode is set,
The microcode acquisition unit acquires the inner product operation mode microcode,
The arithmetic operation processing unit executes the arithmetic operation processing based on the inner product operation mode microcode,
The neural network processor according to claim 1. The inner product processing unit,
When performing the processing of the convolution layer,
(1) The norm mode processing is repeatedly executed for the number of feature maps of the convolutional layer to be processed,
(2) The process of the inner product calculation mode is repeatedly executed by the number of outputs of the convolutional layer to be processed each time the process of the norm mode is performed for each feature map.
The neural network processor according to claim 1. The inner product processing unit,
When performing the processing of the fully coupled layer,
(1) The process in the norm mode is executed once for all the connected layers to be processed,
(2) The process of the inner product calculation mode is repeatedly executed for the number of outputs of all the connected layers to be processed.
The neural network processor according to claim 1. A neural network processing method for performing a multi-valued neural network process including a convolutional layer process and a fully connected layer process,
A control step of setting a scaling coefficient vector which is real vector data, and setting a multi-value basis matrix having multi-value data as elements;
A quantization processing step of performing a quantization process on the feature map input to the convolutional layer and the feature vector input to the fully connected layer, the minimum value of the feature map and the minimum value of the feature vector Is set to an offset value smaller than a predetermined value, and the quantization process is performed using a quantization width obtained based on the maximum value and the minimum value of the feature map and the feature vector. Said quantization processing step;
(1) executing a norm mode for calculating a norm of the feature map and the feature vector; and (2) performing an inner product calculation process using the multi-valued base matrix and the feature map or the feature vector after the quantization process. An inner product processing step of performing the processing of the norm mode, and the processing of the combination of the inner product calculation modes, thereby performing the processing of the convolutional layer and the processing of the fully connected layer. When,
A neural network processing method comprising:   A program for causing a computer to execute the neural network processing method according to claim 5.