JP2022170512A - Neural network generation device, neural network operation device, edge device, method for controlling neural network, and software generation program - Google Patents

Abstract

To generate and control a neural network that can be built in such a building-in apparatus as an IoT apparatus and can be operated with high performance.SOLUTION: A neural network generation device is for generating a neural network execution model for operating a neural network. The neural network execution model converts input data to first quantification data quantified by first quantification means and second quantification data quantified by second quantification means different from the first quantification means.SELECTED DRAWING: Figure 14

Description

The present invention relates to a neural network generation device, a neural network arithmetic device, an edge device, a neural network control method, and a software generation program.

In recent years, a convolutional neural network (CNN) has been used as a model for image recognition and the like. A convolutional neural network has a multilayer structure having convolution layers and pooling layers, and requires a large number of operations such as convolution operations. Various calculation methods have been devised for speeding up calculation by a convolutional neural network (Patent Document 1, etc.).

JP 2018-077829 A

On the other hand, image recognition and the like using convolutional neural networks are also used in built-in devices such as IoT devices. In order to efficiently operate a convolutional neural network in an embedded device, it is desired to generate a circuit or a model for performing computations related to the neural network that matches the hardware configuration of the embedded device. Also, a control method for operating these circuits and models with high efficiency and high speed is desired. There is also a demand for a software generation program that generates software that allows these circuits and models to operate efficiently and at high speed.

Based on the above circumstances, the present invention provides a neural network generator that generates circuits and models that perform computations related to neural networks that can be embedded in embedded devices such as IoT devices and that can be operated efficiently and at high speed. Neural network computing equipment that performs computations related to neural networks that can operate efficiently and at high speed, edge devices that include neural network computing devices, and neural networks that operate circuits and models that perform computations related to neural networks efficiently and at high speed. It is an object of the present invention to provide a software generating program for generating software for operating circuits and models for performing calculations related to control methods and neural networks with high efficiency and high speed.

In order to solve the above problems, the present invention proposes the following means.
A neural network generation device according to a first aspect of the present invention is a neural network generation device for generating a neural network execution model for computing a neural network, wherein the neural network execution model converts input data into first quantization means and the input data are converted into second quantized data quantized by a second quantization means different from the first quantization means.

The neural network generation device, neural network arithmetic device, edge device, neural network control method, and software generation program of the present invention can be embedded in embedded devices such as IoT devices, and generate a neural network that can operate at high performance. can be controlled by

1 is a diagram showing a neural network generation device according to a first embodiment; FIG. It is a figure which shows the input-output of the calculating part of the same neural network generation apparatus. 1 is a diagram showing an example of a convolutional neural network; FIG. FIG. 4 is a diagram for explaining convolution operations performed by convolution layers of the same convolutional neural network; It is a figure which shows an example of a neural network execution model. 4 is a timing chart showing an operation example of the same neural network execution model; It is a control flowchart of the same neural network generation device. FIG. 4 is an internal block diagram of a generated convolution operation circuit; FIG. 4 is an internal block diagram of a multiplier of the convolution arithmetic circuit; 3 is an internal block diagram of a sum-of-products operation unit of the same multiplier; FIG. FIG. 4 is an internal block diagram of an accumulator circuit of the same convolution arithmetic circuit; It is an internal block diagram of the accumulator unit of the same accumulator circuit. FIG. 4 is a state transition diagram of a control circuit of the same convolution arithmetic circuit; It is a block diagram of the input conversion part of the same convolution arithmetic circuit. It is a figure which shows the example of a 1st threshold value group and a 2nd threshold value group. It is a figure which shows the data output from the same input conversion part. It is a figure which shows the other example of the same 1st threshold value group and the said 2nd threshold value group. It is a figure which shows the other example of the same 1st threshold value group and the said 2nd threshold value group. It is a figure which shows the other example of the same 1st threshold value group and the said 2nd threshold value group. It is a block diagram of the modification of the same input conversion part. It is a figure which shows the example of the 1st threshold value group, the 2nd threshold value group, and the 3rd threshold value group in the same modification. It is a figure explaining the data division and data expansion|deployment of the same convolution operation.

(First embodiment)
A first embodiment of the present invention will be described with reference to FIGS. 1 to 22. FIG.
FIG. 1 is a diagram showing a neural network generation device 300 according to this embodiment.

[Neural network generation device 300]
The neural network generation device 300 is a device that generates a trained neural network execution model 100 that can be embedded in an embedded device such as an IoT device. The neural network execution model 100 is a software or hardware model generated for operating a convolutional neural network 200 (hereinafter referred to as "CNN 200") in an embedded device.

The neural network generation device 300 is a program-executable device (computer) having a processor such as a CPU (Central Processing Unit) and hardware such as a memory. The functions of neural network generation device 300 are realized by executing a neural network generation program and a software generation program in neural network generation device 300 . The neural network generation device 300 includes a storage unit 310 , a calculation unit 320 , a data input unit 330 , a data output unit 340 , a display unit 350 and an operation input unit 360 .

Storage unit 310 stores hardware information HW, network information NW, learning data set DS, neural network execution model 100 (hereinafter referred to as “NN execution model 100”), and learned parameters PM. Hardware information HW, learning data set DS, and network information NW are input data that are input to neural network generation device 300 . The NN execution model 100 and the learned parameters PM are output data output by the neural network generation device 300 . The "trained NN execution model 100" includes the NN execution model 100 and the learned parameters PM.

The hardware information HW is information about an embedded device that operates the NN execution model 100 (hereinafter referred to as "operation target hardware"). The hardware information HW includes, for example, the device type, device restrictions, memory configuration, bus configuration, operating frequency, power consumption, and manufacturing process type of hardware to be operated. The device type is, for example, a type such as ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array). The device constraint is the upper limit of the number of arithmetic units included in the device to be operated, the upper limit of the circuit scale, and the like. The memory configuration includes memory type, number of memories, memory capacity, and input/output data width. The bus configuration includes the bus type, bus width, bus communication standard, connected devices on the same bus, and the like. Also, when there are multiple variations of the NN execution model 100, the hardware information HW includes information on the variation of the NN execution model 100 to be used.

Network information NW is basic information of CNN 200 . The network information NW is, for example, the network configuration of the CNN 200, input data information, output data information, quantization information, and the like. The input data information includes the type of input data such as image and sound, and the size of the input data.

The learning data set DS has learning data D1 used for learning and test data D2 used for inference testing.

FIG. 2 is a diagram showing inputs and outputs of the calculation unit 320. As shown in FIG.
The calculation unit 320 has an execution model generation unit 321 , a learning unit 322 , an inference unit 323 , a hardware generation unit 324 and a software generation unit 325 . The NN execution model 100 input to the calculation unit 320 may be generated by a device other than the neural network generation device 300 .

The execution model generator 321 generates the NN execution model 100 based on the hardware information HW and network information NW. The NN execution model 100 is a software or hardware model generated to cause the CNN 200 to operate on hardware to be operated. Software includes software that controls the hardware model. The hardware model may be a behavioral level, an RTL (Register Transfer Level), a netlist representing connections between gates and circuit modules, or a combination thereof. .

Learning unit 322 generates learned parameters PM using NN execution model 100 and learning data D1. The inference unit 323 performs an inference test using the NN execution model 100 and the test data D2.

Hardware generator 324 generates neural network hardware model 400 based on hardware information HW and NN execution model 100 . The neural network hardware model 400 is a hardware model that can be implemented in hardware to operate. The neural network hardware model 400 is optimized for operation target hardware based on the hardware information HW. The neural network hardware model 400 may be an RTL (Register Transfer Level), a netlist representing connections between gates and circuit modules, or a combination thereof. The neural network hardware model 400 may be a parameter list and configuration files necessary for implementing the NN execution model 100 on hardware. The parameter list and configuration file are used in combination with the NN execution model 100 generated separately.

In the following description, hardware implemented with the neural network hardware model 400 is referred to as "neural network hardware 600".

Software generator 325 generates software 500 for operating neural network hardware 600 based on network information NW and NN execution model 100 . Software 500 includes software that transfers learned parameters PM to neural network hardware 600 as needed.

The data input unit 330 receives hardware information HW, network information NW, and the like necessary for generating the trained NN execution model 100 . The hardware information HW, network information NW, etc. are input as data described in a predetermined data format, for example. The input hardware information HW, network information NW, etc. are stored in the storage unit 310 . The hardware information HW, network information NW, etc. may be input or changed by the user through the operation input unit 360 .

The generated trained NN execution model 100 is output to the data output unit 340 . For example, the generated NN execution model 100 and learned parameters PM are output to the data output unit 340 .

The display unit 350 has a known monitor such as an LCD display. The display unit 350 can display a GUI (Graphical User Interface) image generated by the calculation unit 320, a console screen for receiving commands, and the like. Further, when the calculation unit 320 requires information input from the user, the display unit 350 can display a message prompting the user to input information from the operation input unit 360 or a GUI image required for information input.

The operation input unit 360 is a device through which the user inputs instructions to the calculation unit 320 and the like. The operation input unit 360 is a known input device such as a touch panel, keyboard, and mouse. An input of the operation input section 360 is transmitted to the calculation section 320 .

All or part of the functions of the arithmetic unit 320 are implemented by one or more processors such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) executing a program stored in a program memory. . However, all or part of the functions of the arithmetic unit 320 are implemented by hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), PLD (Programmable Logic Device), etc. circuitry). Moreover, all or part of the functions of the calculation unit 320 may be realized by a combination of software and hardware.

All or part of the functions of the computing unit 320 may be implemented using an external accelerator such as a CPU or GPU or hardware provided in an external device such as a cloud server. The calculation unit 320 can improve the calculation speed of the calculation unit 320 by using, for example, a GPU with high calculation performance on a cloud server or dedicated hardware.

The storage unit 310 is realized by flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), ROM (Read-Only Memory), RAM (Random Access Memory), or the like. All or part of the storage unit 310 may be provided in an external device such as a cloud server and connected to the calculation unit 320 or the like via a communication line.

[Convolutional Neural Network (CNN) 200]
Next, CNN200 is demonstrated. FIG. 3 is a diagram showing an example of the CNN 200. As shown in FIG. The network information NW of the CNN 200 is information regarding the configuration of the CNN 200 described below. The CNN 200 uses low-bit weight w and quantized input data a, and is easy to incorporate into embedded equipment.

The CNN 200 is a multilayer network including an input layer 205 , a convolution layer 210 that performs convolution operations, a quantization operation layer 220 that performs quantization operations, and an output layer 230 . In at least part of CNN 200, convolutional layers 210 and quantization operation layers 220 are interleaved. CNN200 is a model widely used for image recognition and moving image recognition. The CNN 200 may further have layers with other functions, such as fully connected layers.

When the input data input to the CNN 200 has a different format from the input data a to the convolution layer 210, such as a 32-bit floating point type, the input layer 205 performs type conversion, quantization, data division, data shaping, and the like. Perform data conversion. In the following description, input data that is input to the CNN 200 and has a different format from the input data a to the convolutional layer 210 is referred to as "input data b". The input layer 205 transforms input data b into input data a.

FIG. 4 is a diagram for explaining the convolution operation performed by the convolution layer 210. As shown in FIG.
The convolution layer 210 performs a convolution operation on input data a using weight w. The convolution layer 210 performs a sum-of-products operation with input data a and weight w as inputs.

Input data a (also called activation data or feature map) to the convolutional layer 210 is multidimensional data such as image data. In this embodiment, the input data a is a three-dimensional tensor consisting of elements (x, y, c). The convolution layer 210 of the CNN 200 performs a convolution operation on low-bit input data a. In this embodiment, the elements of the input data a are 2-bit unsigned integers (0, 1, 2, 3). Elements of input data a may be, for example, 4-bit or 8-bit unsigned integers.

The weights w (also called filters, kernels) of the convolutional layer 210 are multidimensional data whose elements are learnable parameters. In this embodiment, the weight w is a 4-dimensional tensor consisting of elements (i,j,c,d). The weight w has d three-dimensional tensors (hereinafter referred to as “weight wo”) each having elements (i, j, c). The weight w in the learned CNN 200 is learned data. Convolutional layer 210 of CNN 200 performs a convolution operation using low-bit weights w. In this embodiment, the elements of the weight w are 1-bit signed integers (0,1), where the value '0' represents +1 and the value '1' represents -1.

The convolution layer 210 performs the convolution operation shown in Equation 1 and outputs output data f. In Equation 1, s indicates stride. A region indicated by a dotted line in FIG. 4 indicates one of the regions ao (hereinafter referred to as “applied region ao”) to which the weight wo is applied to the input data a. Elements of the application area ao are represented by (x+i, y+j, c).

The quantization operation layer 220 performs quantization and the like on the convolution operation output from the convolution layer 210 . The quantization operation layer 220 has a pooling layer 221 , a batch normalization layer 222 , an activation function layer 223 and a quantization layer 224 .

The pooling layer 221 performs operations such as average pooling (equation 2) and MAX pooling (equation 3) on the output data f of the convolutional operation output by the convolutional layer 210 to compress the output data f of the convolutional layer 210. do. In Equations 2 and 3, u indicates the input tensor, v indicates the output tensor, and T indicates the size of the pooling region. In Equation 3, max is a function that outputs the maximum value of u for combinations of i and j contained in T.

The Batch Normalization layer 222 normalizes the data distribution of the output data of the quantization operation layer 220 and the pooling layer 221 by, for example, the operation shown in Equation 4. In Equation 4, u denotes the input tensor, v the output tensor, α the scale, and β the bias. In the trained CNN 200, α and β are trained constant vectors.

The activation function layer 223 computes an activation function such as ReLU (Formula 5) on the outputs of the quantization computation layer 220 , the pooling layer 221 and the batch normalization layer 222 . In Equation 5, u is the input tensor and v is the output tensor. In Expression 5, max is a function that outputs the largest numerical value among the arguments.

The quantization layer 224 quantizes the outputs of the pooling layer 221 and the activation function layer 223 based on the quantization parameter, as shown in Equation 6, for example. The quantization shown in Equation 6 reduces the input tensor u to 2 bits. In Equation 6, q(c) is the vector of quantization parameters. In the trained CNN 200, q(c) is a trained constant vector. The inequality sign “≦” in Equation 6 may be “<”.

The output layer 230 is a layer that outputs the results of the CNN 200 using an identity function, a softmax function, or the like. A layer preceding the output layer 230 may be the convolution layer 210 or the quantization operation layer 220 .

In the CNN 200, the quantized output data of the quantization layer 224 is input to the convolution layer 210, so the convolution operation load of the convolution layer 210 is small compared to other convolutional neural networks that do not perform quantization. .

[Neural network execution model 100 (NN execution model) 100]
Next, the NN execution model 100 will be explained. FIG. 5 is a diagram showing an example of the NN execution model 100. As shown in FIG. The NN execution model 100 is a software or hardware model generated to cause the CNN 200 to operate on hardware to be operated. Software includes software that controls the hardware model. The hardware model may be a behavioral level, an RTL (Register Transfer Level), a netlist representing connections between gates and circuit modules, or a combination thereof. .

The NN execution model 100 includes a first memory 1, a second memory 2, a DMA controller 3 (hereinafter also referred to as "DMAC 3"), a convolution operation circuit 4, a quantization operation circuit 5, and a controller 6. Prepare. The NN execution model 100 is characterized in that a convolution operation circuit 4 and a quantization operation circuit 5 are formed in a loop via a first memory 1 and a second memory 2 .

The first memory 1 is a rewritable memory such as a volatile memory such as an SRAM (Static RAM). Data is written to and read from the first memory 1 via the DMAC 3 and the controller 6 . The first memory 1 is connected to the input port of the convolution operation circuit 4 , and the convolution operation circuit 4 can read data from the first memory 1 . The first memory 1 is also connected to the output port of the quantization arithmetic circuit 5 , and the quantization arithmetic circuit 5 can write data to the first memory 1 . The external processor EP can input/output data to/from the NN execution model 100 by writing data to and reading data from the first memory 1 .

The second memory 2 is, for example, a rewritable memory such as a volatile memory such as an SRAM (Static RAM). Data is written to and read from the second memory 2 via the DMAC 3 and the controller 6 . The second memory 2 is connected to the input port of the quantization arithmetic circuit 5 , and the quantization arithmetic circuit 5 can read data from the second memory 2 . The second memory 2 is also connected to the output port of the convolution circuit 4 , and the convolution circuit 4 can write data to the second memory 2 . The external processor EP can input/output data to/from the NN execution model 100 by writing data to and reading data from the second memory 2 .

The DMAC 3 is connected to the external bus EB and performs data transfer between the external memory EM such as DRAM and the first memory 1 . The DMAC 3 also transfers data between the external memory EM such as a DRAM and the second memory 2 . The DMAC 3 also transfers data between the external memory EM such as DRAM and the convolution circuit 4 . The DMAC 3 also transfers data between the external memory EM such as a DRAM and the quantization arithmetic circuit 5 .

The convolution operation circuit 4 is a circuit that performs convolution operation in the convolution layer 210 of the trained CNN 200 . The convolution operation circuit 4 reads the input data a stored in the first memory 1 and performs a convolution operation on the input data a. The convolution operation circuit 4 writes output data f of the convolution operation (hereinafter also referred to as “convolution operation output data”) to the second memory 2 .

The quantization operation circuit 5 is a circuit that performs at least part of the quantization operation in the quantization operation layer 220 of the trained CNN 200 . The quantization operation circuit 5 reads the output data f of the convolution operation stored in the second memory 2, and performs quantization operations (pooling, batch normalization, activation function, and quantization) on the output data f of the convolution operation. calculation including at least quantization). The quantization operation circuit 5 writes the output data of the quantization operation (hereinafter also referred to as “quantization operation output data”) out to the first memory 1 .

The controller 6 is connected to the external bus EB and operates as a slave to the external processor EP. The controller 6 has registers 61 including parameter registers and status registers. A parameter register is a register that controls the operation of the NN execution model 100 . The state register is a register that indicates the state of the NN execution model 100 including the semaphore S. An external processor EP can access the registers 61 via the controller 6 .

Controller 6 is connected to first memory 1, second memory 2, DMAC 3, convolution circuit 4, and quantization circuit 5 via internal bus IB. An external processor EP can access each block via the controller 6 . For example, the external processor EP can issue commands to the DMAC 3, the convolution circuit 4, and the quantization circuit 5 via the controller 6. FIG. Also, the DMAC 3, the convolution operation circuit 4, and the quantization operation circuit 5 can update the status register (including the semaphore S) of the controller 6 via the internal bus IB. The status register (including the semaphore S) may be configured to be updated via dedicated wiring connected to the DMAC 3, the convolution operation circuit 4, and the quantization operation circuit 5. FIG.

Since the NN execution model 100 has the first memory 1, the second memory 2, etc., it is possible to reduce the number of data transfers of overlapping data in the data transfer by the DMAC 3 from the external memory EM such as DRAM. As a result, power consumption caused by memory access can be greatly reduced.

FIG. 6 is a timing chart showing an operation example of the NN execution model 100. As shown in FIG. The NN execution model 100 performs computations of the CNN 200, which has a multi-layered structure, by circuits formed in loops. The NN execution model 100 can efficiently use hardware resources due to its looped circuit configuration. An operation example of the neural network hardware 600 shown in FIG. 6 will be described below.

The DMAC 3 stores the input data a of layer 1 (see FIG. 3) in the first memory 1 . The DMAC 3 may divide the input data a of the layer 1 according to the order of the convolution operation performed by the convolution operation circuit 4 and transfer the divided data to the first memory 1 .

The convolution operation circuit 4 reads the input data a of layer 1 (see FIG. 3) stored in the first memory 1 . The convolution operation circuit 4 performs a layer 1 convolution operation on layer 1 input data a. The output data f of the layer 1 convolution operation is stored in the second memory 2 .

The quantization arithmetic circuit 5 reads the layer 1 output data f stored in the second memory 2 . A quantization operation circuit 5 performs a layer 2 quantization operation on layer 1 output data f. The output data out of the layer 2 quantization operation are stored in the first memory 1 .

The convolution operation circuit 4 reads the output data of the layer 2 quantization operation stored in the first memory 1 . The convolution operation circuit 4 performs a layer 3 convolution operation using the output data out of the layer 2 quantization operation as input data a. The output data f of the layer 3 convolution operation is stored in the second memory 2 .

The convolution operation circuit 4 reads the output data out of the quantization operation of the layer 2M-2 (M is a natural number) stored in the first memory 1. FIG. The convolution operation circuit 4 performs the convolution operation of the layer 2M-1 using the output data out of the quantization operation of the layer 2M-2 as the input data a. The output data f of the layer 2M-1 convolution operation is stored in the second memory 2. FIG.

The quantization arithmetic circuit 5 reads the layer 2M-1 output data f stored in the second memory 2 . The quantization operation circuit 5 performs a layer 2M quantization operation on the output data f of the 2M-1 layer. The output data out of the layer 2M quantization operation are stored in the first memory 1 .

The convolution operation circuit 4 reads the output data out of the layer 2M quantization operation stored in the first memory 1 . The convolution operation circuit 4 performs a layer 2M+1 convolution operation using the output data out of the layer 2M quantization operation as input data a. The output data f of the layer 2M+1 convolution operation are stored in the second memory 2 .

The convolution calculation circuit 4 and the quantization calculation circuit 5 alternately perform calculations to advance the calculation of the CNN 200 shown in FIG. In the NN execution model 100, the convolution circuit 4 performs the convolution calculations of layer 2M-1 and layer 2M+1 by time division. In addition, in the NN execution model 100, the quantization operation circuit 5 performs quantization operations for layer 2M-2 and layer 2M by time division. Therefore, the NN execution model 100 has a significantly smaller circuit scale than a case where separate convolution operation circuits 4 and quantization operation circuits 5 are implemented for each layer.

[Operation of Neural Network Generation Device 300]
Next, the operation of the neural network generation device 300 (neural network control method) will be described with reference to the control flowchart of the neural network generation device 300 shown in FIG. After executing the initialization process (step S10), the neural network generation device 300 executes step S11.

<Hardware Information Acquisition Step (S11)>
In step S11, the neural network generation device 300 acquires the hardware information HW of the hardware to be operated (hardware information acquisition step). The neural network generation device 300 acquires hardware information HW input to the data input unit 330, for example. The neural network generation device 300 displays a GUI image necessary for inputting the hardware information HW on the display unit 350, and causes the user to input the hardware information HW from the operation input unit 360, thereby acquiring the hardware information HW. may

The hardware information HW specifically has memory types, memory capacities, and input/output data widths of the memories to be allocated as the first memory 1 and the second memory 2 .

The acquired hardware information HW is stored in the storage unit 310 . Next, the neural network generation device 300 executes step S12.

<Network information acquisition step (S12)>
In step S12, the neural network generation device 300 acquires the network information NW of the CNN 200 (network information acquisition step). The neural network generation device 300 acquires network information NW input to the data input unit 330, for example. The neural network generation device 300 may acquire the network information NW by displaying a GUI image necessary for inputting the network information NW on the display unit 350 and having the user input the network information NW from the operation input unit 360. .

Specifically, the network information NW includes the network configuration including the input layer 205 and the output layer 230, the configuration of the convolution layer 210 including the weight w and the bit width of the input data a, and the quantization operation layer including quantization information. 220 configurations.

The acquired network information NW is stored in storage unit 310 . Next, the neural network generation device 300 executes step S13.

<Neural Network Execution Model Generation Step (S13)>
In step S13, the execution model generation unit 321 of the neural network generation device 300 generates the NN execution model 100 based on the hardware information HW and the network information NW (neural network execution model generation step).

The neural network execution model generation step (NN execution model generation step) includes, for example, a convolution operation circuit generation step (S13-1), a quantization operation circuit generation step (S13-2), and a DMAC generation step (S13-3). and have

<Convolution Operation Circuit Generation Step (S13-1)>
The execution model generation unit 321 generates the convolutional operation circuit 4 of the NN execution model 100 based on the hardware information HW and the network information NW (convolutional operation circuit generation step). The execution model generation unit 321 generates a hardware model of the convolution operation circuit 4 from information such as the weight w input as the network information NW and the bit width of the input data a. The hardware model may be a behavioral level, an RTL (Register Transfer Level), a netlist representing connections between gates and circuit modules, or a combination thereof. . An example of the generated hardware model of the convolution operation circuit 4 will be described below.

FIG. 8 is an internal block diagram of the generated convolution operation circuit 4. As shown in FIG.
The convolution arithmetic circuit 4 has a weight memory 41 , a multiplier 42 , an accumulator circuit 43 , a state controller 44 and an input converter 49 . The convolution operation circuit 4 has a dedicated state controller 44 for the multiplier 42 and the accumulator circuit 43, and when an instruction command is input, the convolution operation can be performed without the need for an external controller.

The weight memory 41 is a memory that stores the weight w used in the convolution operation, and is a rewritable memory such as a volatile memory such as an SRAM (Static RAM). The DMAC 3 writes the weight w required for the convolution operation into the weight memory 41 by DMA transfer.

FIG. 9 is an internal block diagram of the multiplier 42. As shown in FIG.
The multiplier 42 multiplies each element of the input data a by each element of the weight w. Each element of the input data a is data obtained by dividing the input data a, and is vector data having Bc elements (for example, "input vector A" described later). Each element of the weight w is data obtained by dividing the weight w, and is matrix data having Bc×Bd elements (for example, a “weight matrix W” described later). The multiplier 42 has Bc×Bd product-sum operation units 47, and can perform multiplication of the input vector A and the weight matrix W in parallel.

The multiplier 42 reads out the input vector A and the weight matrix W required for multiplication from the first memory 1 and the weight memory 41 to carry out the multiplication. The multiplier 42 outputs Bd sum-of-products operation results O(di).

FIG. 10 is an internal block diagram of the sum-of-products operation unit 47. As shown in FIG.
Sum-of-products unit 47 performs multiplication of input vector A element A(ci) with weight matrix W element W(ci, di). Further, the product-sum operation unit 47 adds the multiplication result and the multiplication result S(ci, di) of another product-sum operation unit 47 . The sum-of-products operation unit 47 outputs the addition result S(ci+1, di). ci is an index from 0 to (Bc-1). di is an index from 0 to (Bd-1). Element A(ci) is a 2-bit unsigned integer (0, 1, 2, 3). The element W(ci,di) is a 1-bit signed integer (0,1), where the value "0" represents +1 and the value "1" represents -1.

The sum-of-products operation unit 47 has an inverter (inverter) 47a, a selector 47b, and an adder 47c. The sum-of-products operation unit 47 performs multiplication using only the inverter 47a and the selector 47b without using a multiplier. The selector 47b selects the input of the element A(ci) when the element W(ci, di) is "0". If the element W(ci, di) is "1", the selector 47b selects the complement of the element A(ci) inverted by an inverter. Element W(ci, di) is also input to Carry-in of adder 47c. The adder 47c outputs a value obtained by adding the element A(ci) to S(ci, di) when the element W(ci, di) is "0". The adder 47c outputs a value obtained by subtracting the element A(ci) from S(ci, di) when the element W(ci, di) is "1".

FIG. 11 is an internal block diagram of the accumulator circuit 43. As shown in FIG.
The accumulator circuit 43 accumulates the sum-of-products operation result O(di) of the multiplier 42 in the second memory 2 . The accumulator circuit 43 has Bd accumulator units 48 and can accumulate Bd product-sum operation results O(di) in parallel in the second memory 2 .

FIG. 12 is an internal block diagram of the accumulator unit 48. As shown in FIG.
The accumulator unit 48 has an adder 48a and a mask portion 48b. The adder 48 a adds the element O(di) of the sum-of-products operation result O and the partial sum, which is the intermediate progress of the convolution operation shown in Equation 1, stored in the second memory 2 . The addition result is 16 bits per element. The addition result is not limited to 16 bits per element, and may be, for example, 15 bits or 17 bits per element.

The adder 48a writes the addition result to the same address in the second memory 2. FIG. The mask unit 48b masks the output from the second memory 2 and zeros the addition target for the element O(di) when the initialization signal clear is asserted. The initialization signal clear is asserted when the intermediate partial sum is not stored in the second memory 2 .

When the convolution operation by the multiplier 42 and the accumulator circuit 43 is completed, the output data f(x, y, do) having Bd elements is stored in the second memory.

State controller 44 controls the states of multiplier 42 and accumulator circuit 43 . Also, the state controller 44 is connected to the controller 6 via an internal bus IB. The state controller 44 has an instruction queue 45 and a control circuit 46 .

The instruction queue 45 is a queue in which the instruction command C4 for the convolution operation circuit 4 is stored, and is composed of a FIFO memory, for example. An instruction command C4 is written to the instruction queue 45 via the internal bus IB.

The control circuit 46 is a state machine that decodes the instruction command C4 and controls the multiplier 42 and the accumulator circuit 43 based on the instruction command C4. The control circuit 46 may be implemented by a logic circuit or by a CPU controlled by software.

FIG. 13 is a state transition diagram of the control circuit 46. As shown in FIG.
When the instruction command C4 is input to the instruction queue 45 (Not empty), the control circuit 46 transitions from the idle state S1 to the decode state S2.

The control circuit 46 decodes the instruction command C3 output from the instruction queue 45 in the decode state S2. Also, the control circuit 46 reads the semaphore S stored in the register 61 of the controller 6 and determines whether the operations of the multiplier 42 and the accumulator circuit 43 instructed by the instruction command C4 can be executed. If it is not executable (Not ready), the control circuit 46 waits until it becomes executable (Wait). If it is ready (ready), the control circuit 46 transitions from the decode state S2 to the run state S3.

In the execution state S3, the control circuit 46 controls the multiplier 42 and the accumulator circuit 43 to perform the operation indicated by the instruction command C4. After the operation of the multiplier 42 and the accumulator circuit 43 is finished, the control circuit 34 removes the executed instruction command C4 from the instruction queue 45 and updates the semaphore S stored in the register 61 of the controller 6 . When there is an instruction in the instruction queue 45 (Not empty), the control circuit 46 transitions from the execution state S3 to the decode state S2. When the instruction queue 45 has no instruction (empty), the control circuit 46 transitions from the execution state S3 to the idle state S1.

The execution model generator 321 determines the specifications and sizes (Bc and Bd) of the arithmetic units in the convolution arithmetic circuit 4 from information such as the weight w input as the network information NW and the bit width of the input data a. When the hardware scale of the NN execution model 100 (neural network hardware model 400, neural network hardware 600) to be generated as the hardware information HW is included, the execution model generating unit 321 performs the convolution operation according to the specified scale. The specifications and sizes (Bc and Bd) of the calculator in the circuit 4 are adjusted.

[Input converter 49]
FIG. 14 is a block diagram of the input conversion section 49. As shown in FIG.
The input conversion unit 49 converts the input data b including elements of more bits (for example, 8 bits or more) than the elements of the input data a into the input data a. Input transformer 49 corresponds to at least part of input layer 205 concatenated before convolutional layer 210 of CNN 200 . The input converter 49 has a first converter 491 , a second converter 492 and a threshold memory 495 .

Note that the input conversion unit 49 does not necessarily have to be implemented as hardware. Conversion processing of the input data b may be performed as a pre-processing in the software generation step (S17) described later.

Here, in the description of the input conversion unit 49, for the sake of simplicity, it is assumed that the input data b is image data having one element in the c-axis direction (that is, a two-dimensional image on the xy plane). It is also assumed that the image data comprises a matrix data structure with 8 bits (0-255) for each element in the x and y directions. The input data b is converted by the input conversion unit 49 into input data a that can be input to the convolution circuit.

The first conversion unit 491 converts the input data b into first quantized data a1 using a first threshold value group (first threshold value group) TG1. The first threshold group TG1 is one or more thresholds, and the thresholds are predetermined values within the range (0-255) that the input data b can take. The first conversion unit 491 compares the input data b with the first threshold value group TG1 in the same manner as in Equation 6, and encodes the comparison result to convert the input data b into the first quantized data a1 ( first quantization means).

In this embodiment, the first threshold group TG1 is three thresholds. The first conversion unit 491 quantizes the input data b having 8-bit elements into first quantized data a1 having 2-bit elements.

The second conversion unit 492 converts the input data b into second quantized data a2 using a second threshold value group (second threshold value group) TG2. The second threshold group TG2 is one or more thresholds, and the thresholds are predetermined values within the range (0-255) that the input data b can take. The second conversion unit 492 compares the input data b with the second threshold value group TG2 in the same manner as in Equation 6, and encodes the comparison result to convert the input data b into the second quantized data a2 ( second quantization means).

In this embodiment, the second threshold group TG2 is three thresholds. The second conversion unit 492 quantizes the input data b having 8-bit elements into second quantized data a2 having 2-bit elements.

That is, the input conversion unit 49 converts the input data b into two types of quantized data (first quantized data a1 and second Convert to binary quantized data a2).

The threshold memory 495 is a memory that stores a first threshold value group TG1 used for calculation in the first conversion unit 491 and a second threshold value group TG2 used for calculation in the second conversion unit 492. For example, SRAM (Static A rewritable memory such as a volatile memory such as a RAM). For example, the DMAC3 writes the first threshold group TG1 and the second threshold group TG2 to the threshold memory 495 by DMA transfer.

The second threshold group TG2 is a threshold group different from the first threshold group TG1. The second threshold group TG2 differs from the first threshold group TG1 in at least some thresholds. In the present embodiment, the second threshold group TG2 differs from the first threshold group TG1 in all thresholds.

FIG. 15 is a diagram showing examples of the first threshold value group TG1 and the second threshold value group TG2.
The first threshold group TG1 is threshold 36.4, threshold 109.3 and threshold 182.1. The second threshold group TG2 is threshold 72.8, threshold 145.7, and threshold 218.6. The average value of the thresholds in the second threshold group TG2 is greater than the average value of the thresholds in the first threshold group TG1. Note that these threshold values may be rounded integers.

The thresholds included in the first threshold value group TG1 and the second threshold value group TG2 divide the possible range (0-255) of the input data b into seven areas. The thresholds included in the first threshold group TG1 and the second threshold group TG2 are arranged with the same step width (approximately 36.4 (≈255/7)). The thresholds included in the first threshold value group TG1 and the threshold values included in the second threshold value group TG2 are alternately arranged within the possible range (0-255) of the input data b. At least some of the thresholds included in the first threshold group TG1 and the thresholds included in the second threshold group TG2 may be alternately arranged.

FIG. 16 is a diagram showing data (input data a) output from the input conversion unit 49. As shown in FIG.
The input conversion unit 49 arranges and connects the first quantized data a1 and the second quantized data a2 in the c-axis direction, and outputs the result as input data a. The multiplier 42 performs a convolution operation on the input data a output from the input conversion section 49 .

When the number of elements in the c-axis direction of the input data b is two or more, the input conversion unit 49 performs the same conversion as described above for each element in the c-axis direction, for example. In this case, the number of elements in the c-axis direction of the input data a is twice the number of elements in the c-axis direction of the input data b. As an example, when the input data b includes three elements of RGB, which are color components, as elements in the c-axis direction, the number of elements in the c-axis direction of the input data a is doubled to six. Note that the input conversion unit 49 may convert, for example, selectively increasing only the number of elements in the c-axis direction of the input data a corresponding to one element (for example, G) among the three elements of RGB.

Since the convolution operation circuit 4 uses the quantized input data a as an input for the convolution operation, the size of the configuration of the multiplier 42 and the like can be reduced. On the other hand, since the input data b is quantized into the input data a, the precision of the input data a (the number of gradations in the case of image data) is lowered. However, the convolution operation circuit 4 converts the input data b into two types of quantized data (first quantized data a1 and second Converting to two quantized data a2), the two types of quantized data are connected in the c-axis direction. Therefore, the convolution operation circuit 4 can reduce the influence of the accuracy deterioration of the input data a due to quantization while maintaining the configuration of the multiplier 42 and the like which are reduced in size.

When the number of elements in the c-axis direction in the input data a to be input to the convolution operation circuit 4 is small, hardware resources parallelized (multichannelized) in the convolution operation circuit 4 (for example, the multiplier 42 of the convolution operation circuit 4) are used. may not be used effectively. The convolution operation circuit 4 increases the number of elements in the c-axis direction by arranging and concatenating two types of quantized data in the c-axis direction by the above-described method, and effectively utilizes parallelized (multichannel) hardware resources. can be used to improve speed.

The thresholds included in the first threshold value group TG1 shown in FIG. 15 are arranged with substantially the same step width in the possible range (0-255) of the input data b. The thresholds included in the second threshold value group TG2 shown in FIG. 15 are arranged with substantially the same step width in the possible range (0-255) of the input data b. As a result, regardless of the characteristics of the input data b, the influence of the aforementioned decrease in accuracy of the input data a is reduced. Further, the influence of the decrease in accuracy of the input data a due to quantization is reduced by alternately setting the thresholds included in the first threshold value group TG1 and the threshold values included in the second threshold value group TG2 in an interpolative manner. .

The thresholds included in the first threshold value group TG1 and the threshold values included in the second threshold value group TG2 shown in FIG. 15 are arranged alternately within the possible range (0-255) of the input data b. Therefore, both the first threshold value group TG1 and the second threshold value group TG2 divide the possible range (0-255) of the input data b relatively evenly. As a result, the two types of quantized data (the first quantized data a1 and the second quantized data a2) tend to inherit the original features of the input data b, making it easier to extract features in the convolution operation.

Although the present embodiment exemplified the method of using the threshold value shown in Equation 6 as the quantization means in the input conversion section 49, the quantization means is not limited to this. The quantization means may for example be performed using a plurality of lookup tables.

FIG. 17 is a diagram showing another example of the first threshold value group TG1 and the second threshold value group TG2.
The first threshold group TG1 shown in FIG. 17 is a threshold of 36.4, a threshold of 72.8, and a threshold of 109.3. The second threshold value group TG2 shown in FIG. 17 is a threshold value of 145.7, a threshold value of 182.1, and a threshold value of 218.6. Note that these threshold values may be rounded integers.

In the example shown in FIG. 17 as well, the thresholds included in the first threshold value group TG1 and the second threshold value group TG2 divide the possible range (0-255) of the input data b into seven regions. . Therefore, the influence of the accuracy deterioration of the input data a is reduced in the same manner as the quantization by the threshold group shown in FIG. 15 .

The three threshold values of the first threshold value group TG1 shown in FIG. 17 are closer to the minimum value 0 than the maximum value 255. Therefore, the first quantized data a1 transformed based on the first threshold value group TG1 tends to inherit the characteristics of the data in the area close to the minimum value 0 in the possible range (0-255) of the input data b.

The three threshold values of the second threshold value group TG2 shown in FIG. 17 are closer to the maximum value of 255 than the minimum value of 0. Therefore, the second quantized data a2 transformed based on the second threshold value group TG2 tends to inherit the characteristics of the data in the area close to the maximum value 255 in the possible range (0-255) of the input data b.

FIG. 18 is a diagram showing another example of the first threshold value group TG1 and the second threshold value group TG2.
The first threshold value group TG1 shown in FIG. 18 is a threshold value of 36.4, a threshold value of 72.8, and a threshold value of 145.7. The second threshold group TG2 shown in FIG. 18 is threshold 109.3, threshold 182.1, and threshold 218.6. Note that these threshold values may be rounded integers.

In the example shown in FIG. 18 as well, the thresholds included in the first threshold group TG1 and the second threshold group TG2 divide the possible range (0-255) of the input data b into seven areas. . Therefore, the influence of the accuracy deterioration of the input data a is reduced in the same manner as the quantization by the threshold group shown in FIG. 15 . Further, the influence of the decrease in accuracy of the input data a due to quantization is reduced by alternately setting the thresholds included in the first threshold value group TG1 and the threshold values included in the second threshold value group TG2 in an interpolative manner. there is In other words, by setting the average value of the threshold values included in the first threshold value group TG1 and the average value of the threshold values included in the second threshold value group TG2 to be different, the respective threshold values are alternately set in an interpolative manner. making it possible to

The thresholds included in the first threshold value group TG1 and the second threshold value group TG2 do not have to divide the possible range (0-255) of the input data b into substantially equal parts. The input conversion unit 49 may set threshold values included in the first threshold value group TG1 and the second threshold value group TG2, for example, based on data distribution information obtained from a histogram of the input data b. For example, if the threshold group is set so that the quantized data after data conversion (the first quantized data a1 and the second quantized data a2) are not biased, the influence of the accuracy deterioration of the input data a can be reduced.

FIG. 19 is a diagram showing another example of the first threshold value group TG1 and the second threshold value group TG2.
The thresholds included in the first threshold value group TG1 and the second threshold value group TG2 may not be distributed over the entire range (0-255) that the input data b can take. As shown in FIG. 19, the input conversion unit 49 may set the minimum and maximum values in the possible range (0-255) of the input data b to set the effective range VR in which the thresholds are distributed. The input conversion unit 49 may set, for example, a region in which a large amount of data is distributed as the effective range VR for distributing the threshold values, based on data distribution information obtained from a histogram of the input data b. By setting a range with many data changes (gradation changes in the case of image data) as the effective range VR, the quantized data after data conversion (the first quantized data a1 and the second quantized data a2) becomes the input data. It is easy to inherit the characteristics of b.

When the input data b is image data obtained from a camera, the input conversion unit 49 may exclude a region containing noise such as pattern noise from the effective range VR. The influence of noise contained in the input data b is eliminated, and the influence of deterioration in accuracy of the input data a is reduced. Also, when the black level in the image data is set to a value greater than 0, the input conversion section 49 may exclude the area below the black level from the valid range VR. Furthermore, when performing an additional operation such as an operation of multiplying the image data by a digital gain, the input conversion unit 49 may remove in advance from the effective range VR a region where the gradation is lost due to the additional operation. As an example, when multiplying by a double digital gain in the additional operation, the input conversion unit 49 sets the effective range of the input data b to the first half region (0-127) and excludes the latter half region (128-255) from the effective range VR. may Note that the effective range VR may be a divided area.

Note that the valid range VR may be the same as the possible range (0-255) of the input data b.

Note that the thresholds included in the first threshold value group TG1 and the second threshold value group TG2 may be powers of two. In the input conversion unit 49, the scale of the circuit required for comparison between the threshold value and the input data b and data conversion is reduced.

[Modified example of input converter 49]
FIG. 20 is a block diagram of an input conversion section 49B that is a modification of the input conversion section 49. As shown in FIG.
The input conversion unit 49B converts the input data b including elements of more bits (for example, 8 bits or more) than the elements of the input data a into the input data a. Input transformer 49B corresponds to at least part of input layer 205 connected before convolutional layer 210 of CNN 200 . The input converter 49B has a first converter 491 , a second converter 492 , a third converter 493 and a threshold memory 495 .

The third transform unit 493 transforms the input data b into third quantized data a3 using a third threshold value group (third threshold value group) TG3. The third threshold group TG3 is one or more thresholds, and the thresholds are predetermined values within the range (0-255) that the input data b can take. The third conversion unit 493 compares the input data b with the third threshold value group TG3 in the same manner as in Equation 6, and encodes the comparison result to convert the input data b into the third quantized data a3 ( third quantization means).

In this embodiment, the third threshold group TG3 is three thresholds. The third conversion unit 493 quantizes the input data b having 8-bit elements into third quantized data a3 having 2-bit elements.

That is, the input conversion unit 4B transforms the input data b into three types of quantized data (first Quantized data a1, second quantized data a2 and third quantized data a3).

The threshold memory 495 stores a third threshold group TG3 in addition to a first threshold group TG1 used for calculation in the first conversion unit 491 and a second threshold group TG2 used for calculation in the second conversion unit 492. .

The third threshold group TG3 is a threshold group different from the first threshold group TG1 and a threshold group different from the second threshold group TG2. In this embodiment, the third threshold group TG3 differs in all thresholds from the first threshold group TG1 and the second threshold group TG2.

FIG. 21 is a diagram showing examples of the first threshold group TG1, the second threshold group TG2, and the third threshold group TG3. The first threshold group TG1 is a threshold of 25.5, a threshold of 102.0 and a threshold of 178.5. The second threshold group TG2 is a threshold of 51.0, a threshold of 127.5, and a threshold of 204.5. The third threshold group TG3 is a threshold of 76.5, a threshold of 153.0, and a threshold of 229.5. Note that these threshold values may be rounded integers.

The thresholds included in the first threshold group TG1, the second threshold group TG2, and the third threshold group TG3 substantially equally divide the range (0-255) of the input data b into 10 areas. The threshold values included in the first threshold value group TG1 and the second threshold value group TG2 are arranged with the same step width (25.5.(=255/10)).

The input conversion unit 49B arranges and connects the first quantized data a1, the second quantized data a2, and the third quantized data a3 in the c-axis direction, and outputs them as the input data a. The multiplier 42 performs a convolution operation on the input data a output from the input conversion section 49 .

The thresholds included in the first threshold value group TG1 shown in FIG. 21 are arranged with substantially the same step width in the possible range (0-255) of the input data b. The thresholds included in the second threshold value group TG2 shown in FIG. 21 are arranged with substantially the same step width in the possible range (0-255) of the input data b. The thresholds included in the third threshold value group TG3 shown in FIG. 21 are arranged with substantially the same step width in the possible range (0-255) of the input data b. As a result, regardless of the characteristics of the input data b, the influence of the aforementioned decrease in accuracy of the input data a is reduced.

The convolution operation circuit 4 including the input conversion unit 49B can reduce the influence of the accuracy deterioration of the input data a due to quantization while maintaining the configuration of the multiplier 42 and the like which are reduced in size.

Note that the input conversion unit 49B may convert the input data b into four or more types of quantized data based on four or more different threshold groups. When converting into quantized data based on N different threshold groups, the step width step of the thresholds included in the N threshold groups can be calculated by Equation 7, for example. Here, min_val is the lower limit of the range that the input data b can take, max_val is the upper limit of the range that the input data b can take, and k is the number of bits of the input data a. Note that min_val and max_val may be matched with the lower limit and upper limit of the effective range VR.

<Quantization Operation Circuit Generation Step (S13-2)>
The execution model generation unit 321 generates the quantization arithmetic circuit 5 of the NN execution model 100 based on the hardware information HW and the network information NW (quantization arithmetic circuit generation step). The execution model generation unit 321 generates a hardware model of the quantization arithmetic circuit 5 from the quantization information input as the network information NW. The hardware model may be a behavioral level, an RTL (Register Transfer Level), a netlist representing connections between gates and circuit modules, or a combination thereof. .

<DMAC generation step (S13-3)>
The execution model generation unit 321 generates the DMAC3 of the NN execution model 100 based on the hardware information HW and the network information NW (DMAC generation step). The execution model generation unit 321 generates a hardware model of the DMAC 3 from information input as the network information NW. The hardware model may be a behavioral level, an RTL (Register Transfer Level), a netlist representing connections between gates and circuit modules, or a combination thereof. .

<Learning step (S14)>
In step S14, the learning unit 322 and the inference unit 323 of the neural network generating device 300 learn learning parameters of the generated NN execution model 100 using the learning data set DS (learning step). The learning step (S14) includes, for example, a learned parameter generation step (S14-1) and an inference test step (S14-2).

<Learning Step: Learned Parameter Generation Step (S14-1)>
Learning unit 322 generates learned parameters PM using NN execution model 100 and learning data D1. The learned parameters PM are the learned weight w, the quantization parameter q, and the threshold group (first threshold group TG1 and second threshold group TG2) of the input conversion unit 49 .

For example, when the NN execution model 100 is the execution model of the CNN 200 that performs image recognition, the learning data D1 is a combination of the input image and the teacher data T. An input image is input data a input to the CNN 200 . The teacher data T includes the type of subject captured in the image, the presence or absence of the detection target in the image, the coordinate values of the detection target in the image, and the like.

The learning unit 322 generates a learned parameter PM by supervised learning using a well-known technique such as the error backpropagation method. The learning unit 322 obtains the difference E between the output of the NN execution model 100 for the input image and the teacher data T corresponding to the input image using a loss function (error function). Update the weights w and the quantization parameter q.

For example, when updating weight w, the gradient of the loss function with respect to weight w is used. The slope is calculated, for example, by differentiating the loss function. When using backpropagation, the gradient is calculated by backward propagation.

The learning unit 322 increases the precision of operations related to the convolution operation when calculating the gradient and updating the weight w. Specifically, a 32-bit floating-point weight w that is more accurate than the low-bit weight w (for example, 1 bit) used by the NN execution model 100 is used for learning. Further, the precision of the convolution operation performed in the convolution operation circuit 4 of the NN execution model 100 is improved.

The learning unit 322 increases the precision of calculations related to the activation function when calculating the gradient and updating the weight w. Specifically, a Sigmond function, which is more accurate than an activation function such as a ReLU function implemented in the quantization arithmetic circuit 5 of the NN execution model 100, is used for learning.

On the other hand, when calculating the output data for the input image by forward propagation, the learning unit 322 does not increase the precision of the convolution operation and the operation related to the activation function, and performs the operation based on the NN execution model 100. to implement. The high-precision weight w used when updating the weight w is reduced in bits by a lookup table or the like.

When the gradient is calculated and the weight w is updated, the learning unit 322 increases the accuracy of the convolution operation and the operation related to the activation function, thereby preventing the accuracy of the intermediate data in the operation from decreasing and enabling high inference. A learned parameter PM that can achieve accuracy can be generated.

On the other hand, when calculating the output data for the input image, the learning unit 322 performs calculation based on the NN execution model 100 without increasing the accuracy of the forward propagation calculation. Therefore, the output data calculated by the learning unit 322 and the output data of the NN execution model 100 using the generated learned parameter PM match.

The learning unit 322 learns the threshold value group of the input conversion unit 49 in addition to the weight w and the quantization parameter q. The learning unit 322 obtains the difference E between the output of the NN execution model 100 for the input data and the teacher data T corresponding to the input data by supervised learning such as the error backpropagation method, using a loss function (error function), At least one or more thresholds included in the threshold group are updated so that the difference E becomes smaller. As shown in FIG. 15 and the like, the initial values of the threshold value group of the input conversion unit 49 before learning are alternately interpolated between the threshold values included in the first threshold value group TG1 and the threshold values included in the second threshold value group TG2. is set. Each threshold in the threshold group is updated by repeating learning. It should be noted that each threshold value in the threshold value group may have a format including a decimal point at the time of learning, and may be a rounded integer at the end of learning.

Note that the gradient of the loss function gradually disappears from the output layer toward the input layer. In this embodiment, the input converter 49 is closest to the input layer. Therefore, the amount of change in the gradient during learning in the threshold value group of the input conversion unit 49 is small compared to other parameters to be learned. Therefore, when the initial values of the threshold value group of the input conversion unit 49 are appropriately set, the learning unit 322 does not need to set the threshold value group of the input conversion unit 49 as learning targets.

In the present embodiment, an example in which the learning unit 322 learns the threshold value group of the input conversion unit 49 is shown. may be used as learning targets.

Note that the learning unit 322 may learn the effective range VR by estimating the possible range of the input data b at the time of inference from the input data b used at the time of learning. The input data b used during learning preferably has the same conditions as the input data b used during inference. Therefore, the learning unit 322 can predict to some extent the possible range of the input data b at the time of inference from the input data b used at the time of learning.

<Learning Step: Inference Test Step (S14-2)>
The inference unit 323 performs an inference test using the learned parameters PM generated by the learning unit 322, the NN execution model 100, and the test data D2. For example, if the NN execution model 100 is the execution model of the CNN 200 that performs image recognition, the test data D2 is a combination of the input image and the teacher data T, similar to the learning data D1.

The inference unit 323 displays the progress and results of the inference test on the display unit 350 . The result of the reasoning test is, for example, the accuracy rate for the test data D2.

<Confirmation step (S15)>
In step S<b>15 , the inference unit 323 of the neural network generation device 300 causes the display unit 350 to display a message prompting the user to input confirmation regarding the result from the operation input unit 360 and a GUI image necessary for inputting information. The user inputs from the operation input unit 360 whether to accept the result of the inference test. When an input indicating that the user accepts the result of the inference test is input from the operation input unit 360, the neural network generation device 300 next performs step S16. When the user inputs an input from the operation input unit 360 indicating that the result of the inference test is not acceptable, the neural network generator 300 performs step S12 again. Incidentally, the neural network generation device 300 may return to step S11 and allow the user to re-input the hardware information HW.

<Output step (S16)>
In step S<b>16 , hardware generation unit 324 of neural network generation device 300 generates neural network hardware model 400 based on hardware information HW and NN execution model 100 .

<Software generation step (S17)>
In step S17, the software generation unit 325 of the neural network generation device 300 generates the neural network hardware 600 (the neural network hardware model 400 implemented in the operation target hardware) based on the network information NW and the NN execution model 100. ) is generated. Software 500 includes software that transfers learned parameters PM to neural network hardware 600 as needed.

The software generation step (S17) includes, for example, an input data conversion step (S17-1), an input data division step (S17-2), a network division step (S17-3), and an allocation step (S17-4). , have

<Input data conversion step (S17-1)>
If the input conversion unit 49 is not implemented as hardware in the convolution arithmetic circuit 4, the software generation unit 325 converts the input data b in advance to generate converted input data a as preprocessing. The conversion method of the input data a in the input data conversion step is the same as the conversion method in the input conversion section 49 .

<Input Data Division Step (S17-2): Data Division>
The software generation unit 325 partially converts the input data a for the convolution operation of the convolution layer 210 based on the memory capacity of the memory allocated as the first memory 1 and the second memory 2, the specifications and sizes (Bc and Bd) of the calculator, and the like. Split into tensors. The method of division into partial tensors and the number of divisions are not particularly limited. A partial tensor is formed, for example, by splitting the input data a(x+i, y+j, c) into a(x+i, y+j, co).

FIG. 22 is a diagram for explaining data division and data development in a convolution operation.
In the data division of the convolution operation, the variable c in Equation 1 is divided into blocks of size Bc as shown in Equation 8. Also, the variable d in Equation 1 is divided into blocks of size Bd, as shown in Equation 9. In Equation 8, co is the offset and ci is the index from 0 to (Bc-1). In Equation 9, do is the offset and di is the index from 0 to (Bd-1). Note that the size Bc and the size Bd may be the same.

Input data a(x+i, y+j, c) in Equation 1 is divided by size Bc in the c-axis direction and represented by divided input data a(x+i, y+j, co). In the following description, the divided input data a is also referred to as "divided input data a".

The weight w (i, j, c, d) in Equation 1 is divided by the size Bc in the c-axis direction and the size Bd in the d-axis direction, and is represented by the divided weight w (i, j, co, do) be. In the following description, the divided weight w is also referred to as "divided weight w".

The output data f(x, y, do) divided by the size Bd is obtained by Equation (10). By combining the divided output data f(x, y, do), the final output data f(x, y, d) can be calculated.

<Input data division step (S17-3): data expansion>
The software generator 325 develops the divided input data a and weight w in the convolution operation circuit 4 of the NN execution model 100 .

Divided input data a(x+i, y+j, co) is developed into vector data having Bc elements. Elements of the divided input data a are indexed by ci (0≤ci<Bc). In the following description, divided input data a developed into vector data for each i and j is also referred to as "input vector A". Input vector A has elements from divided input data a(x+i, y+j, co×Bc) to divided input data a(x+i, y+j, co×Bc+(Bc−1)).

The division weight w(i, j, co, do) is developed into matrix data having Bc×Bd elements. The elements of the division weight w developed into matrix data are indexed by ci and di (0≦di<Bd). In the following description, the divided weight w developed into matrix data for each i and j is also referred to as "weight matrix W". The weight matrix W includes division weights w(i, j, co×Bc, do×Bd) to division weights w(i, j, co×Bc+(Bc−1), do×Bd+(Bd−1)). element.

By multiplying the input vector A and the weight matrix W, vector data is calculated. Output data f(x, y, do) can be obtained by shaping the vector data calculated for each of i, j, and co into a three-dimensional tensor. By developing such data, the convolution operation of the convolution layer 210 can be performed by multiplying the vector data and the matrix data.

<Allocation step (S17-4)>
The software generation unit 325 generates the software 500 that allocates the divided operations to the neural network hardware 600 for execution (allocation step). The generated software 500 includes an instruction command C4. If the input data b has been converted in the input data conversion step (S17-1), the software 500 includes the converted input data a.

As described above, according to the neural network generation device 300, the neural network control method, and the software generation program according to the present embodiment, the neural network can be embedded in an embedded device such as an IoT device and can be operated with high performance. Can generate and control networks.

As described above, the first embodiment of the present invention has been described in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and design changes and the like are included within the scope of the present invention. . Also, the constituent elements shown in the above-described embodiment and modifications can be combined as appropriate.

(Modification 1-1)
In the above embodiment, the first memory 1 and the second memory 2 are different memories, but the aspect of the first memory 1 and the second memory 2 is not limited to this. The first memory 1 and the second memory 2 may be, for example, a first memory area and a second memory area in the same memory.

(Modification 1-2)
For example, the data input to the NN execution model 100 and the neural network hardware 600 described in the above embodiments are not limited to a single format, and can be composed of still images, moving images, voices, characters, numerical values, and combinations thereof. It is possible to The data input to the NN execution model 100 and the neural network hardware 600 can be mounted on the edge device where the neural network hardware 600 is provided, such as an optical sensor, a thermometer, a Global Positioning System (GPS) measuring instrument, an angular velocity It is not limited to the measurement result of a physical quantity measuring instrument such as a measuring instrument or an anemometer. Peripheral information such as base station information, vehicle/vessel information, weather information, and congestion information received from peripheral devices via wired or wireless communication, and different information such as financial information and personal information may be combined.

(Modification 1-3)
Edge devices provided with the neural network hardware 600 are assumed to be communication devices such as mobile phones driven by batteries, smart devices such as personal computers, mobile devices such as digital cameras, game devices, and robot products. It is not limited. Unprecedented effects can also be obtained by using power on Ethernet (PoE), etc., to limit the peak power that can be supplied, reduce product heat generation, or use it for products that require long-time operation. For example, by applying it to in-vehicle cameras installed in vehicles and ships, surveillance cameras installed in public facilities and roads, etc., it is possible not only to realize long-time shooting, but also to contribute to weight reduction and durability. . Similar effects can be obtained by applying the present invention to display devices such as televisions and displays, medical equipment such as medical cameras and surgical robots, and work robots used at manufacturing sites and construction sites.

It may be realized by recording the program in the above-described embodiment in a computer-readable recording medium and causing a computer system to read and execute the program recorded in the recording medium. It should be noted that the "computer system" referred to here includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. Furthermore, "computer-readable recording medium" refers to a program that dynamically retains programs for a short period of time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It may also include something that holds the program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in that case. Further, the program may be for realizing part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in the computer system.

Also, the effects described herein are merely illustrative or exemplary, and are not limiting. In other words, the technology according to the present disclosure can produce other effects that are obvious to those skilled in the art from the description of this specification, in addition to or instead of the above effects.

The present invention can be applied to the generation of neural networks.

300 Neural network generator 200 Convolutional neural network (CNN)
100 neural network execution model (NN execution model)
400 neural network hardware model 500 software 600 neural network hardware (neural network arithmetic unit)
1 first memory 2 second memory 3 DMA controller (DMAC)
4 convolution operation circuit 42 multiplier 43 accumulator circuit 49 input conversion unit 491 first conversion unit 492 second conversion unit 493 third conversion unit 495 threshold memory 5 quantization operation circuit 6 controller PM learned parameter DS learning data set HW Hardware Information NW Network information TG1 First threshold group TG2 Second threshold group TG3 Third threshold group

Claims (16)

A neural network generation device for generating a neural network execution model for computing a neural network,
The neural network execution model includes first quantized data obtained by quantizing input data by a first quantizing means, and second quantized data obtained by quantizing the input data by a second quantizing means different from the first quantizing means. converted to readable data,
Neural network generator. the number of bits of the first quantized data and the number of bits of the second quantized data are equal,
The neural network execution model performs a convolution operation on data in which the first quantized data and the second quantized data are concatenated.
The neural network generator according to claim 1. The first quantization means converts the input data into the first quantized data according to a first threshold group,
The second quantization means converts the input data into the second quantized data using a second threshold group at least partially different from the first threshold group.
3. The neural network generator according to claim 1 or 2. The thresholds of the first threshold group and the second threshold group are values included in the valid range set to the possible range of the input data.
4. The neural network generator according to claim 3. The thresholds of the first threshold group and the second threshold group are values that substantially evenly divide the effective range.
5. The neural network generation device according to claim 4. The thresholds of the first threshold group and the thresholds of the second threshold group are values arranged alternately in the effective range,
6. The neural network generator according to claim 4 or 5. The thresholds of the first threshold group are values arranged with substantially the same step width,
The neural network generator according to any one of claims 3 to 6. The thresholds of the first threshold group and the second threshold group are values set based on the distribution information of the input data.
5. The neural network generator according to claim 3 or 4. Input conversion for converting input data into first quantized data obtained by quantizing input data by a first quantizing means and second quantized data obtained by quantizing the input data by a second quantizing means different from the first quantizing means. Department and
a convolution operation circuit that receives as input data in which the first quantized data and the second quantized data are concatenated;
comprising
Neural network arithmetic unit. The first quantization means converts the input data into the first quantized data according to a first threshold group,
10. The neural network operation device according to claim 9, wherein said second quantization means converts said input data into said second quantized data using a second threshold value group at least partially different from said first threshold value group. The thresholds of the first threshold group and the second threshold group are values included in the valid range set to the possible range of the input data,
The thresholds of the first threshold group and the second threshold group are values that substantially evenly divide the effective range.
The neural network operation device according to claim 10. a neural network operation device according to any one of claims 9 to 11;
a power supply for operating the neural network arithmetic device;
edge device. A method of controlling neural network hardware that operates a neural network, comprising:
A conversion step of converting input data into first quantized data obtained by quantizing input data by a first quantizing means and second quantized data obtained by quantizing said input data by a second quantizing means different from said first quantizing means. When,
an operation step of performing a convolution operation on the first quantized data and the second quantized data;
comprising
Neural network control method. the transforming step is preprocessed by a device other than the neural network hardware;
The neural network control method according to claim 13. A program for generating software for controlling neural network hardware that operates a neural network,
A conversion step of converting input data into first quantized data obtained by quantizing input data by a first quantizing means and second quantized data obtained by quantizing said input data by a second quantizing means different from said first quantizing means. When,
generating the software comprising an operation step of performing a convolution operation on the first quantized data and the second quantized data;
Software generation program. A program for generating software for controlling neural network hardware that operates a neural network,
Convolution using first quantized data obtained by quantizing input data by a first quantizing means and second quantized data obtained by quantizing the input data by a second quantizing means different from the first quantizing means generating said software comprising computation steps to perform computation;
Software generation program.

Images