JP2023006509A - Software generation device and software generation method - Google Patents

Abstract

To provide a software generation device and a software generation method capable of generating and controlling a neural network which can be incorporated into an embedded device such as an IoT device and can be operated at high performance.SOLUTION: Disclosed is a neural network generation device for generating the software for controlling a neural network execution model (NN execution model) and including a software generation part which includes the steps of: analyzing information on a model including a plurality of layers operating in an NN execution model; determining a lifetime corresponding to a plurality of layers included in the model based on an analysis result of the analysis means; and generating the software based on the lifetime.SELECTED DRAWING: Figure 2

Description

The present invention relates to a software generation device and software generation method.

In recent years, a convolutional neural network (CNN) has been used as a model for image recognition and the like. A convolutional neural network has a multilayer structure having convolution layers and pooling layers, and requires a large number of operations such as convolution operations. Various calculation methods have been devised for speeding up calculation by a convolutional neural network (Patent Document 1, etc.).

JP 2018-077829 A

On the other hand, image recognition and the like using convolutional neural networks are also used in built-in devices such as IoT devices. In order to efficiently operate a convolutional neural network in an embedded device, it is desired to generate a circuit or a model for performing computations related to the neural network that matches the hardware configuration of the embedded device. There is also a demand for a software generation method that allows these circuits and models to operate efficiently and at high speed.

In view of the above circumstances, the present invention provides a software generation apparatus and a neural network for generating circuits and models for performing calculations related to neural networks that can be embedded in embedded devices such as IoT devices and can be operated at high efficiency and high speed. It is an object of the present invention to provide a software generation method for operating a circuit or model that performs calculations related to high efficiency and high speed.

In order to solve the above problems, the present invention proposes the following means.
A neural network generation device according to a first aspect of the present invention is a software generation device for generating software for controlling a neural network circuit, wherein information on a model including a plurality of layers operating in the neural network circuit is generated by: analysis means for analyzing; determination means for determining lifetimes corresponding to the plurality of layers included in the model based on the analysis results of the analysis means; and generation means for generating the software based on the lifetimes Prepare.

INDUSTRIAL APPLICABILITY The software generation device and software generation method of the present invention can be embedded in embedded equipment such as IoT equipment, and can generate and control a neural network that can operate with high performance.

1 is a diagram showing a neural network generation device according to an embodiment; FIG. It is a figure which shows the input-output of the calculating part of the same neural network generation apparatus. 1 is a diagram showing an example of a convolutional neural network; FIG. FIG. 4 is a diagram for explaining convolution operations performed by convolution layers of the same convolutional neural network; It is a figure which shows an example of a neural network execution model. 4 is a timing chart showing an operation example of the same neural network execution model; It is a control flowchart of the same neural network generation device. FIG. 4 is an internal block diagram of a generated convolution operation circuit; FIG. 4 is an internal block diagram of a multiplier of the convolution arithmetic circuit; 3 is an internal block diagram of a sum-of-products operation unit of the same multiplier; FIG. FIG. 4 is an internal block diagram of an accumulator circuit of the same convolution arithmetic circuit; It is an internal block diagram of the accumulator unit of the same accumulator circuit. FIG. 4 is a state transition diagram of a control circuit of the same convolution arithmetic circuit; 3 is an internal block diagram of a generated quantization arithmetic circuit; FIG. 3 is an internal block diagram of a vector operation circuit and a quantization circuit of the same quantization operation circuit; FIG. 4 is a block diagram of an arithmetic unit of the same vector arithmetic circuit; FIG. 4 is an internal block diagram of a quantization unit of the same quantization circuit; FIG. 4 is an internal block diagram of a generated DMAC; FIG. 3 is an internal block diagram of an edge device; FIG. It is a figure which shows an example of a neural network and an example of the conventional allocation. It is a figure explaining an allocation process. It is a figure which shows an example of a neural network, and an example of allocation.

(embodiment)
An embodiment of the present invention will be described with reference to FIGS. 1 to 22. FIG.
FIG. 1 is a diagram showing a neural network generation device 300 according to this embodiment.

[Neural network generation device 300]
The neural network generation device 300 is a device that generates a trained neural network execution model 100 that can be embedded in an embedded device such as an IoT device. The neural network execution model 100 is a software or hardware model generated for operating a convolutional neural network 200 (hereinafter referred to as "CNN 200") in an embedded device.

The neural network generation device 300 is a program-executable device (computer) having a processor such as a CPU (Central Processing Unit) and hardware such as a memory. The functions of neural network generation device 300 are realized by executing a neural network generation program in neural network generation device 300 . The neural network generation device 300 includes a storage unit 310 , a calculation unit 320 , a data input unit 330 , a data output unit 340 , a display unit 350 and an operation input unit 360 .

Storage unit 310 stores hardware information HW, network information NW, learning data set DS, neural network execution model 100 (hereinafter referred to as “NN execution model 100”), and learned parameters PM. Hardware information HW, learning data set DS, and network information NW are input data that are input to neural network generation device 300 . The NN execution model 100 and the learned parameters PM are output data output by the neural network generation device 300 . The "trained NN execution model 100" includes the NN execution model 100 and the learned parameters PM.

The hardware information HW is information about an embedded device that operates the NN execution model 100 (hereinafter referred to as "operation target hardware"). The hardware information HW includes, for example, the device type, device restrictions, memory configuration, bus configuration, operating frequency, power consumption, and manufacturing process type of hardware to be operated. The device type is, for example, a type such as ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array). The device constraint is the upper limit of the number of arithmetic units included in the device to be operated, the upper limit of the circuit scale, and the like. The memory configuration includes memory type, number of memories, memory capacity, and input/output data width. The bus configuration includes the bus type, bus width, bus communication standard, connected devices on the same bus, and the like. Also, when there are multiple variations of the NN execution model 100, the hardware information HW includes information on the variation of the NN execution model 100 to be used.

Network information NW is basic information of CNN 200 . The network information NW is, for example, the network configuration of the CNN 200, input data information, output data information, quantization information, and the like. The input data information includes the type of input data such as image and sound, and the size of the input data.

The learning data set DS has learning data D1 used for learning and test data D2 used for inference testing.

FIG. 2 is a diagram showing inputs and outputs of the calculation unit 320. As shown in FIG.
The calculation unit 320 has an execution model generation unit 321 , a learning unit 322 , an inference unit 323 , a hardware generation unit 324 and a software generation unit 325 . The NN execution model 100 input to the calculation unit 320 may be generated by a device other than the neural network generation device 300 .

The execution model generator 321 generates the NN execution model 100 based on the hardware information HW and network information NW. The NN execution model 100 is a software or hardware model generated to cause the CNN 200 to operate on hardware to be operated. Software includes software that controls the hardware model. The hardware model may be a behavioral level, an RTL (Register Transfer Level), a netlist representing connections between gates and circuit modules, or a combination thereof. .

Learning unit 322 generates learned parameters PM using NN execution model 100 and learning data D1. The inference unit 323 performs an inference test using the NN execution model 100 and the test data D2.

Hardware generator 324 generates neural network hardware model 400 based on hardware information HW and NN execution model 100 . The neural network hardware model 400 is a hardware model that can be implemented in hardware to operate. The neural network hardware model 400 is optimized for operation target hardware based on the hardware information HW. The neural network hardware model 400 may be an RTL (Register Transfer Level), a netlist representing connections between gates and circuit modules, or a combination thereof. The neural network hardware model 400 may be a parameter list and configuration files necessary for implementing the NN execution model 100 on hardware. The parameter list and configuration file are used in combination with the NN execution model 100 generated separately.

In the following description, hardware implemented with the neural network hardware model 400 is referred to as "neural network hardware 600".

Software generator 325 generates software 500 for operating neural network hardware 600 based on network information NW or NN execution model 100 . Software 500 includes software that transfers learned parameters PM to neural network hardware 600 as needed. The format of the software 500 may be not only the source code format but also the binary format.

The data input unit 330 receives hardware information HW, network information NW, and the like necessary for generating the trained NN execution model 100 . The hardware information HW, network information NW, etc. are input as data described in a predetermined data format, for example. The input hardware information HW, network information NW, etc. are stored in the storage unit 310 . The hardware information HW, network information NW, etc. may be input or changed by the user through the operation input unit 360 .

The generated trained NN execution model 100 is output to the data output unit 340 . For example, the generated NN execution model 100 and learned parameters PM are output to the data output unit 340 .

The display unit 350 has a known monitor such as an LCD display. The display unit 350 can display a GUI (Graphical User Interface) image generated by the calculation unit 320, a console screen for receiving commands, and the like. Further, when the calculation unit 320 requires information input from the user, the display unit 350 can display a message prompting the user to input information from the operation input unit 360 or a GUI image required for information input.

The operation input unit 360 is a device through which the user inputs instructions to the calculation unit 320 and the like. The operation input unit 360 is a known input device such as a touch panel, keyboard, and mouse. An input of the operation input section 360 is transmitted to the calculation section 320 .

All or part of the functions of the arithmetic unit 320 are implemented by one or more processors such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) executing a program stored in a program memory. . However, all or part of the functions of the arithmetic unit 320 are implemented by hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), PLD (Programmable Logic Device), etc. circuitry). Moreover, all or part of the functions of the calculation unit 320 may be realized by a combination of software and hardware.

All or part of the functions of the computing unit 320 may be implemented using an external accelerator such as a CPU or GPU or hardware provided in an external device such as a cloud server. The calculation unit 320 can improve the calculation speed of the calculation unit 320 by using, for example, a GPU with high calculation performance on a cloud server or dedicated hardware.

The storage unit 310 is realized by flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), ROM (Read-Only Memory), RAM (Random Access Memory), or the like. All or part of the storage unit 310 may be provided in an external device such as a cloud server and connected to the calculation unit 320 or the like via a communication line.

[Convolutional Neural Network (CNN) 200]
Next, CNN200 is demonstrated. FIG. 3 is a diagram showing an example of the CNN 200. As shown in FIG. The network information NW of the CNN 200 is information regarding the configuration of the CNN 200 described below. The CNN 200 uses low-bit weight w and quantized input data a, and is easy to incorporate into embedded equipment.

The CNN 200 is a multi-layered network including a convolution layer 210 that performs convolution operations, a quantization operation layer 220 that performs quantization operations, and an output layer 230 . In at least part of CNN 200, convolutional layers 210 and quantization operation layers 220 are interleaved. CNN200 is a model widely used for image recognition and moving image recognition. The CNN 200 may further have layers with other functions, such as fully connected layers.

FIG. 4 is a diagram for explaining the convolution operation performed by the convolution layer 210. As shown in FIG.
The convolution layer 210 performs a convolution operation on input data a using weight w. The convolution layer 210 performs a sum-of-products operation with input data a and weight w as inputs.

Input data a (also called activation data or feature map) to the convolutional layer 210 is multidimensional data such as image data. In this embodiment, the input data a is a three-dimensional tensor consisting of elements (x, y, c). The convolution layer 210 of the CNN 200 performs a convolution operation on low-bit input data a. In this embodiment, the elements of the input data a are 2-bit unsigned integers (0, 1, 2, 3). Elements of input data a may be, for example, 4-bit or 8-bit unsigned integers.

If the input data input to the CNN 200 has a different format from the input data a to the convolutional layer 210, such as a 32-bit floating point type, the CNN 200 has an input layer that performs type conversion and quantization before the convolutional layer 210. You may have more.

The weights w (also called filters, kernels) of the convolutional layer 210 are multidimensional data whose elements are learnable parameters. In this embodiment, the weight w is a 4-dimensional tensor consisting of elements (i,j,c,d). The weight w has d three-dimensional tensors (hereinafter referred to as “weight wo”) each having elements (i, j, c). The weight w in the learned CNN 200 is learned data. Convolutional layer 210 of CNN 200 performs a convolution operation using low-bit weights w. In this embodiment, the elements of the weight w are 1-bit signed integers (0,1), where the value '0' represents +1 and the value '1' represents -1.

The convolution layer 210 performs the convolution operation shown in Equation 1 and outputs output data f. In Equation 1, s indicates stride. A region indicated by a dotted line in FIG. 4 indicates one of the regions ao (hereinafter referred to as “applied region ao”) to which the weight wo is applied to the input data a. Elements of the application area ao are represented by (x+i, y+j, c).

The quantization operation layer 220 performs quantization and the like on the convolution operation output from the convolution layer 210 . The quantization operation layer 220 has a pooling layer 221 , a batch normalization layer 222 , an activation function layer 223 and a quantization layer 224 .

The pooling layer 221 performs operations such as average pooling (equation 2) and MAX pooling (equation 3) on the output data f of the convolutional operation output by the convolutional layer 210 to compress the output data f of the convolutional layer 210. do. In Equations 2 and 3, u indicates the input tensor, v indicates the output tensor, and T indicates the size of the pooling region. In Equation 3, max is a function that outputs the maximum value of u for combinations of i and j contained in T.

The Batch Normalization layer 222 normalizes the data distribution of the output data of the quantization operation layer 220 and the pooling layer 221 by, for example, the operation shown in Equation 4. In Equation 4, u denotes the input tensor, v the output tensor, α the scale, and β the bias. In the trained CNN 200, α and β are trained constant vectors.

The activation function layer 223 computes an activation function such as ReLU (Formula 5) on the outputs of the quantization computation layer 220 , the pooling layer 221 and the batch normalization layer 222 . In Equation 5, u is the input tensor and v is the output tensor. In Expression 5, max is a function that outputs the largest numerical value among the arguments.

The quantization layer 224 quantizes the outputs of the pooling layer 221 and the activation function layer 223 based on the quantization parameter, as shown in Equation 6, for example. The quantization shown in Equation 6 reduces the input tensor u to 2 bits. In Equation 6, q(c) is the vector of quantization parameters. In the trained CNN 200, q(c) is a trained constant vector. The inequality “≦” in Equation 6 may be “<”.

The output layer 230 is a layer that outputs the results of the CNN 200 using an identity function, a softmax function, or the like. A layer preceding the output layer 230 may be the convolution layer 210 or the quantization operation layer 220 .

In the CNN 200, the quantized output data of the quantization layer 224 is input to the convolution layer 210, so the convolution operation load of the convolution layer 210 is small compared to other convolutional neural networks that do not perform quantization. .

[Neural network execution model 100 (NN execution model) 100]
Next, the NN execution model 100 will be explained. FIG. 5 is a diagram showing an example of the NN execution model 100. As shown in FIG. The NN execution model 100 is a software or hardware model generated to cause the CNN 200 to operate on hardware to be operated. Software includes software that controls the hardware model. The hardware model may be a behavioral level, an RTL (Register Transfer Level), a netlist representing connections between gates and circuit modules, or a combination thereof. .

The NN execution model 100 includes a first memory 1, a second memory 2, a DMA controller 3 (hereinafter also referred to as "DMAC 3"), a convolution operation circuit 4, a quantization operation circuit 5, and a controller 6. Prepare. The NN execution model 100 is characterized in that a convolution operation circuit 4 and a quantization operation circuit 5 are formed in a loop via a first memory 1 and a second memory 2 .

The first memory 1 is a rewritable memory such as a volatile memory such as an SRAM (Static RAM). Data is written to and read from the first memory 1 via the DMAC 3 and the controller 6 . The first memory 1 is connected to the input port of the convolution operation circuit 4 , and the convolution operation circuit 4 can read data from the first memory 1 . The first memory 1 is also connected to the output port of the quantization arithmetic circuit 5 , and the quantization arithmetic circuit 5 can write data to the first memory 1 . The external host CPU can input/output data to/from the NN execution model 100 by writing/reading data to/from the first memory 1 .

The second memory 2 is, for example, a rewritable memory such as a volatile memory such as an SRAM (Static RAM). Data is written to and read from the second memory 2 via the DMAC 3 and the controller 6 . The second memory 2 is connected to the input port of the quantization arithmetic circuit 5 , and the quantization arithmetic circuit 5 can read data from the second memory 2 . The second memory 2 is also connected to the output port of the convolution circuit 4 , and the convolution circuit 4 can write data to the second memory 2 . The external host CPU can input/output data to/from the NN execution model 100 by writing/reading data to/from the second memory 2 .

The DMAC 3 is connected to the external bus EB and performs data transfer between an external memory such as a DRAM and the first memory 1 . The DMAC 3 also transfers data between an external memory such as a DRAM and the second memory 2 . The DMAC 3 also transfers data between an external memory such as a DRAM and the convolution circuit 4 . The DMAC 3 also transfers data between an external memory such as a DRAM and the quantization arithmetic circuit 5 .

The convolution operation circuit 4 is a circuit that performs convolution operation in the convolution layer 210 of the trained CNN 200 . The convolution operation circuit 4 reads the input data a stored in the first memory 1 and performs a convolution operation on the input data a. The convolution operation circuit 4 writes output data f of the convolution operation (hereinafter also referred to as “convolution operation output data”) to the second memory 2 .

The quantization operation circuit 5 is a circuit that performs at least part of the quantization operation in the quantization operation layer 220 of the trained CNN 200 . The quantization operation circuit 5 reads the output data f of the convolution operation stored in the second memory 2, and performs quantization operations (pooling, batch normalization, activation function, and quantization) on the output data f of the convolution operation. calculation including at least quantization). The quantization operation circuit 5 writes the output data of the quantization operation (hereinafter also referred to as “quantization operation output data”) out to the first memory 1 .

The controller 6 is connected to the external bus EB and operates as a slave of an external host CPU. The controller 6 has registers 61 including parameter registers and status registers. A parameter register is a register that controls the operation of the NN execution model 100 . The state register is a register that indicates the state of the NN execution model 100 including the semaphore S. An external host CPU can access the register 61 via the controller 6 .

Controller 6 is connected to first memory 1, second memory 2, DMAC 3, convolution circuit 4, and quantization circuit 5 via internal bus IB. An external host CPU can access each block via the controller 6 . For example, the external host CPU can issue commands to the DMAC 3, the convolution circuit 4, and the quantization circuit 5 via the controller 6. FIG. Also, the DMAC 3, the convolution operation circuit 4, and the quantization operation circuit 5 can update the status register (including the semaphore S) of the controller 6 via the internal bus IB. The status register (including the semaphore S) may be configured to be updated via dedicated wiring connected to the DMAC 3, the convolution operation circuit 4, and the quantization operation circuit 5. FIG.

Since the NN execution model 100 has the first memory 1, the second memory 2, etc., it is possible to reduce the number of data transfers of overlapping data in the data transfer by the DMAC 3 from the external memory such as DRAM. As a result, power consumption caused by memory access can be greatly reduced.

FIG. 6 is a timing chart showing an operation example of the NN execution model 100. As shown in FIG. The NN execution model 100 performs computations of the CNN 200, which has a multi-layered structure, by circuits formed in loops. The NN execution model 100 can efficiently use hardware resources due to its looped circuit configuration. An operation example of the neural network hardware 600 shown in FIG. 6 will be described below.

The DMAC 3 stores the input data a of layer 1 (see FIG. 3) in the first memory 1 . The DMAC 3 may divide the input data a of the layer 1 according to the order of the convolution operation performed by the convolution operation circuit 4 and transfer the divided data to the first memory 1 .

The convolution operation circuit 4 reads the input data a of layer 1 (see FIG. 3) stored in the first memory 1 . The convolution operation circuit 4 performs a layer 1 convolution operation on layer 1 input data a. The output data f of the layer 1 convolution operation is stored in the second memory 2 .

The quantization arithmetic circuit 5 reads the layer 1 output data f stored in the second memory 2 . A quantization operation circuit 5 performs a layer 2 quantization operation on layer 1 output data f. The output data out of the layer 2 quantization operation are stored in the first memory 1 .

The convolution operation circuit 4 reads the output data of the layer 2 quantization operation stored in the first memory 1 . The convolution operation circuit 4 performs a layer 3 convolution operation using the output data out of the layer 2 quantization operation as input data a. The output data f of the layer 3 convolution operation is stored in the second memory 2 .

The convolution operation circuit 4 reads the output data out of the quantization operation of the layer 2M-2 (M is a natural number) stored in the first memory 1. FIG. The convolution operation circuit 4 performs the convolution operation of the layer 2M-1 using the output data out of the quantization operation of the layer 2M-2 as the input data a. The output data f of the layer 2M-1 convolution operation is stored in the second memory 2. FIG.

The quantization arithmetic circuit 5 reads the layer 2M-1 output data f stored in the second memory 2 . The quantization operation circuit 5 performs a layer 2M quantization operation on the output data f of the 2M-1 layer. The output data out of the layer 2M quantization operation are stored in the first memory 1 .

The convolution operation circuit 4 reads the output data out of the layer 2M quantization operation stored in the first memory 1 . The convolution operation circuit 4 performs a layer 2M+1 convolution operation using the output data out of the layer 2M quantization operation as input data a. The output data f of the layer 2M+1 convolution operation are stored in the second memory 2 .

The convolution calculation circuit 4 and the quantization calculation circuit 5 alternately perform calculations to advance the calculation of the CNN 200 shown in FIG. In the NN execution model 100, the convolution circuit 4 performs the layer 2M-1 convolution and the layer 2M+1 by time division. In addition, in the NN execution model 100, the quantization operation circuit 5 performs layer 2M-2 convolution operation and layer 2M by time division. Therefore, the NN execution model 100 has a significantly smaller circuit scale than a case where separate convolution operation circuits 4 and quantization operation circuits 5 are implemented for each layer.

Although FIG. 6 shows an example in which the convolution operation circuit 4 and the quantization operation circuit 5 alternately perform operations to advance the operation of the CNN 200 shown in FIG. 3, the present invention is not limited to this. For example, the convolution operation circuit 4 may be controlled to perform the layer 2M+1 in parallel with the quantization operation circuit 5 performing the layer 2M quantization operation by time division. This operation makes it possible to further improve the computational efficiency. As another example, parallel operations may be performed on two discontinuous layers.

[Operation of Neural Network Generation Device 300]
Next, the operation of the neural network generation device 300 (neural network control method) will be described with reference to the control flowchart of the neural network generation device 300 shown in FIG. After executing the initialization process (step S10), the neural network generation device 300 executes step S11.

<Hardware Information Acquisition Step (S11)>
In step S11, the neural network generation device 300 acquires the hardware information HW of the hardware to be operated (hardware information acquisition step). The neural network generation device 300 acquires hardware information HW input to the data input unit 330, for example. The neural network generation device 300 displays a GUI image necessary for inputting the hardware information HW on the display unit 350, and causes the user to input the hardware information HW from the operation input unit 360, thereby acquiring the hardware information HW. may

The hardware information HW specifically has memory types, memory capacities, and input/output data widths of the memories to be allocated as the first memory 1 and the second memory 2 .

The acquired hardware information HW is stored in the storage unit 310 . Next, the neural network generation device 300 executes step S12.

<Network information acquisition step (S12)>
In step S12, the neural network generation device 300 acquires the network information NW of the CNN 200 (network information acquisition step). The neural network generation device 300 acquires network information NW input to the data input unit 330, for example. The neural network generation device 300 may acquire the network information NW by displaying a GUI image necessary for inputting the network information NW on the display unit 350 and having the user input the network information NW from the operation input unit 360. .

Specifically, the network information NW includes the network configuration including the input layer and the output layer 230, the configuration of the convolution layer 210 including the weight w and the bit width of the input data a, and the quantization operation layer 220 including quantization information. and a configuration of

The acquired network information NW is stored in storage unit 310 . Next, the neural network generation device 300 executes step S13.

<Neural Network Execution Model Generation Step (S13)>
In step S13, the execution model generation unit 321 of the neural network generation device 300 generates the NN execution model 100 based on the hardware information HW and the network information NW (neural network execution model generation step).

The neural network execution model generation step (NN execution model generation step) includes, for example, a convolution circuit generation step (S13-1), a quantization circuit generation step (S13-2), a DMAC generation step (S13-3), have As the NN execution model generation process, a part or all of the process may be omitted by using a part or all of the circuit generated in advance and stored in the storage unit 310 or the like. Alternatively, the NN execution model generation process may be realized by selecting from circuits prepared in advance based on the hardware information HW or the network information NW.

<Convolution Circuit Generation Step (S13-1)>
The execution model generator 321 generates the convolution circuit 4 of the NN execution model 100 based on the hardware information HW and the network information NW (convolution circuit generation step). The execution model generation unit 321 generates a hardware model of the convolution operation circuit 4 from information such as the weight w input as the network information NW and the bit width of the input data a. The hardware model may be a behavioral level, an RTL (Register Transfer Level), a netlist representing connections between gates and circuit modules, or a combination thereof. . An example of the generated hardware model of the convolution operation circuit 4 will be described below.

FIG. 8 is an internal block diagram of the generated convolution operation circuit 4. As shown in FIG.
The convolution circuit 4 has a weight memory 41 , a multiplier 42 , an accumulator circuit 43 and a state controller 44 . The convolution operation circuit 4 has a dedicated state controller 44 for the multiplier 42 and the accumulator circuit 43, and when an instruction command is input, the convolution operation can be performed without the need for an external controller.

The weight memory 41 is a memory that stores the weight w used in the convolution operation, and is a rewritable memory such as a volatile memory such as an SRAM (Static RAM). The DMAC 3 writes the weight w required for the convolution operation into the weight memory 41 by DMA transfer.

FIG. 9 is an internal block diagram of the multiplier 42. As shown in FIG.
Multiplier 42 multiplies input vector A and weight matrix W. FIG. The input vector A is data obtained by dividing the input data a, and is vector data having Bc elements. The weight matrix W is data obtained by dividing the weight w, and is matrix data having Bc×Bd elements. The multiplier 42 has Bc×Bd product-sum operation units 47, and can perform multiplication of the input vector A and the weight matrix W in parallel.

The multiplier 42 reads out the input vector A and the weight matrix W required for multiplication from the first memory 1 and the weight memory 41 to carry out the multiplication. The multiplier 42 outputs Bd sum-of-products operation results O(di).

FIG. 10 is an internal block diagram of the sum-of-products operation unit 47. As shown in FIG.
Sum-of-products unit 47 performs multiplication of input vector A element A(ci) with weight matrix W element W(ci, di). Further, the product-sum operation unit 47 adds the multiplication result and the multiplication result S(ci, di) of another product-sum operation unit 47 . The sum-of-products operation unit 47 outputs the addition result S(ci+1, di). ci is an index from 0 to (Bc-1). di is an index from 0 to (Bd-1). Element A(ci) is a 2-bit unsigned integer (0, 1, 2, 3). The element W(ci,di) is a 1-bit signed integer (0,1), where the value "0" represents +1 and the value "1" represents -1.

The sum-of-products operation unit 47 has an inverter (inverter) 47a, a selector 47b, and an adder 47c. The sum-of-products operation unit 47 performs multiplication using only the inverter 47a and the selector 47b without using a multiplier. The selector 47b selects the input of the element A(ci) when the element W(ci, di) is "0". If the element W(ci, di) is "1", the selector 47b selects the complement of the element A(ci) inverted by an inverter. Element W(ci, di) is also input to Carry-in of adder 47c. The adder 47c outputs a value obtained by adding the element A(ci) to S(ci, di) when the element W(ci, di) is "0". The adder 47c outputs a value obtained by subtracting the element A(ci) from S(ci, di) when W(ci, di) is "1".

FIG. 11 is an internal block diagram of the accumulator circuit 43. As shown in FIG.
The accumulator circuit 43 accumulates the sum-of-products operation result O(di) of the multiplier 42 in the second memory 2 . The accumulator circuit 43 has Bd accumulator units 48 and can accumulate Bd product-sum operation results O(di) in parallel in the second memory 2 .

FIG. 12 is an internal block diagram of the accumulator unit 48. As shown in FIG.
The accumulator unit 48 has an adder 48a and a mask portion 48b. The adder 48 a adds the element O(di) of the sum-of-products operation result O and the partial sum, which is the intermediate progress of the convolution operation shown in Equation 1, stored in the second memory 2 . The addition result is 16 bits per element. The addition result is not limited to 16 bits per element, and may be, for example, 15 bits or 17 bits per element.

The adder 48a writes the addition result to the same address in the second memory 2. FIG. The mask unit 48b masks the output from the second memory 2 and zeros the addition target for the element O(di) when the initialization signal clear is asserted. The initialization signal clear is asserted when the intermediate partial sum is not stored in the second memory 2 .

When the convolution operation by the multiplier 42 and the accumulator circuit 43 is completed, the output data f(x, y, do) having Bd elements is stored in the second memory.

State controller 44 controls the states of multiplier 42 and accumulator circuit 43 . Also, the state controller 44 is connected to the controller 6 via an internal bus IB. The state controller 44 has an instruction queue 45 and a control circuit 46 .

The instruction queue 45 is a queue in which the instruction command C4 for the convolution operation circuit 4 is stored, and is composed of a FIFO memory, for example. An instruction command C4 is written to the instruction queue 45 via the internal bus IB.

The control circuit 46 is a state machine that decodes the instruction command C4 and controls the multiplier 42 and the accumulator circuit 43 based on the instruction command C4. The control circuit 46 may be implemented by a logic circuit or by a CPU controlled by software.

FIG. 13 is a state transition diagram of the control circuit 46. As shown in FIG.
When the instruction command C4 is input to the instruction queue 45 (Not empty), the control circuit 46 transitions from the idle state S1 to the decode state S2.

The control circuit 46 decodes the instruction command C3 output from the instruction queue 45 in the decode state S2. Also, the control circuit 46 reads the semaphore S stored in the register 61 of the controller 6 and determines whether the operations of the multiplier 42 and the accumulator circuit 43 instructed by the instruction command C4 can be executed. If it is not executable (Not ready), the control circuit 46 waits until it becomes executable (Wait). If it is ready (ready), the control circuit 46 transitions from the decode state S2 to the run state S3.

In the execution state S3, the control circuit 46 controls the multiplier 42 and the accumulator circuit 43 to perform the operation indicated by the instruction command C4. After the operation of the multiplier 42 and the accumulator circuit 43 is finished, the control circuit 34 removes the executed instruction command C4 from the instruction queue 45 and updates the semaphore S stored in the register 61 of the controller 6 . When there is an instruction in the instruction queue 45 (Not empty), the control circuit 46 transitions from the execution state S3 to the decode state S2. When the instruction queue 45 has no instruction (empty), the control circuit 46 transitions from the execution state S3 to the idle state S1.

The execution model generator 321 determines the specifications and sizes (Bc and Bd) of the arithmetic units in the convolution arithmetic circuit 4 from information such as the weight w input as the network information NW and the bit width of the input data a. When the hardware scale of the NN execution model 100 (neural network hardware model 400, neural network hardware 600) to be generated as the hardware information HW is included, the execution model generating unit 321 performs the convolution operation according to the specified scale. The specifications and sizes (Bc and Bd) of the calculator in the circuit 4 are adjusted.

<Quantization Circuit Generation Step (S13-2)>
The execution model generation unit 321 generates the quantization circuit 5 of the NN execution model 100 based on the hardware information HW and the network information NW (quantization circuit generation step). The execution model generation unit 321 generates a hardware model of the quantization circuit 5 from the quantization information input as the network information NW. The hardware model may be a behavioral level, an RTL (Register Transfer Level), a netlist representing connections between gates and circuit modules, or a combination thereof. . An example of the generated hardware model of the quantization circuit 5 will be described below.

FIG. 14 is an internal block diagram of the generated quantization arithmetic circuit 5. As shown in FIG.
Quantization operation circuit 5 has quantization parameter memory 51 , vector operation circuit 52 , quantization circuit 53 , and state controller 54 . It has a dedicated state controller 54, and when an instruction command is input, it can perform quantization operations without the need for an external controller.

The quantization parameter memory 51 is a memory that stores the quantization parameter q used in the quantization calculation, and is a rewritable memory such as a volatile memory such as an SRAM (Static RAM). The DMAC 3 writes the quantization parameter q required for the quantization calculation into the quantization parameter memory 51 by DMA transfer.

FIG. 15 is an internal block diagram of the vector operation circuit 52 and the quantization circuit 53. As shown in FIG.
The vector computation circuit 52 computes the output data f(x, y, do) stored in the second memory 2 . The vector operation circuit 52 has Bd number of operation units 57 and performs SIMD operations on output data f(x, y, do) in parallel.

FIG. 16 is a block diagram of the arithmetic unit 57. As shown in FIG.
The arithmetic unit 57 has, for example, an ALU 57a, a first selector 57b, a second selector 57c, a register 57d, and a shifter 57e. The arithmetic unit 57 may further include other calculators and the like that a known general-purpose SIMD arithmetic circuit has.

The vector operation circuit 52 performs a pooling layer 221 in the quantization operation layer 220, a batch normalization layer 222, and a At least one operation among the operations of the activation function layer 223 is performed.

The arithmetic unit 57 can add the data stored in the register 57d and the element f(di) of the output data f(x, y, do) read from the second memory 2 by the ALU 57a. The arithmetic unit 57 can store the addition result by the ALU 57a in the register 57d. The arithmetic unit 57 can initialize the addition result by inputting "0" to the ALU 57a instead of the data stored in the register 57d by selecting the first selector 57b. For example, when the pooling area is 2×2, the shifter 57e can output the average value of the addition result by shifting the output of the ALU 57a to the right by 2 bits. The vector operation circuit 52 can perform the average pooling operation shown in Equation 2 by repeating the above operations and the like by the Bd number of operation units 57 .

The arithmetic unit 57 can compare the data stored in the register 57d with the element f(di) of the output data f(x, y, do) read from the second memory 2 by the ALU 57a.
The arithmetic unit 57 can control the second selector 57c according to the result of comparison by the ALU 57a to select the larger one of the data stored in the register 57d and the element f(di). The arithmetic unit 57 can initialize the comparison target to the minimum value by inputting the minimum value of the possible values of the element f(di) to the ALU 57a by selecting the first selector 57b. Since the element f(di) is a 16-bit signed integer in this embodiment, the minimum possible value of the element f(di) is "0x8000". The vector operation circuit 52 can implement the MAX pooling operation of Equation 3 by repeating the above operations and the like by the Bd number of operation units 57 . Note that the shifter 57e does not shift the output of the second selector 57c in the MAX pooling calculation.

The arithmetic unit 57 can subtract the data stored in the register 57d and the element f(di) of the output data f(x, y, do) read from the second memory 2 by the ALU 57a. Shifter 57e can left shift (ie, multiply) or right shift (ie, divide) the output of ALU 57a. The vector operation circuit 52 can perform the operation of Batch Normalization of Equation 4 by repeating the above operation and the like by the Bd number of operation units 57 .

The arithmetic unit 57 can compare the element f(di) of the output data f(x, y, do) read from the second memory 2 with "0" selected by the first selector 57b by the ALU 57a. The arithmetic unit 57 can select and output either the element f(di) or the constant value "0" previously stored in the register 57d according to the comparison result by the ALU 57a. The vector operation circuit 52 can perform the ReLU operation of Equation 5 by repeating the above operations and the like by the Bd number of operation units 57 .

The vector operation circuit 52 can perform average pooling, MAX pooling, batch normalization, activation function operations, and combinations of these operations. Since vector arithmetic circuit 52 is capable of performing general-purpose SIMD operations, it may also perform other operations required for operations in quantization operations layer 220 . Also, the vector operation circuit 52 may perform operations other than the operations in the quantization operation layer 220 .

Note that the quantization arithmetic circuit 5 may not have the vector arithmetic circuit 52 . If the quantization operation circuit 5 does not have the vector operation circuit 52 , the output data f(x, y, do) are input to the quantization circuit 53 .

A quantization circuit 53 quantizes the output data of the vector operation circuit 52 . As shown in FIG. 15, the quantization circuit 53 has Bd quantization units 58 and performs operations on the output data of the vector operation circuit 52 in parallel.

FIG. 17 is an internal block diagram of quantization unit 58. As shown in FIG.
A quantization unit 58 quantizes the element in(di) of the output data of the vector operation circuit 52 . Quantization unit 58 comprises a comparator 58a and an encoder 58b. The quantization unit 58 performs the operation (formula 6) of the quantization layer 224 in the quantization operation layer 220 on the output data (16 bits/element) of the vector operation circuit 52 . The quantization unit 58 reads the necessary quantization parameters q (th0, th1, th2) from the quantization parameter memory 51, and the comparator 58a compares the input in(di) with the quantization parameter q. Quantization unit 58 quantizes the result of comparison by comparator 58a to 2 bits/element by encoder 58b. Since α(c) and β(c) in Equation 4 are different parameters for each variable c, the quantization parameter q(th0, th1, th2) reflecting α(c) and β(c) is in( d) different parameters for each;

Quantization unit 58 divides input in(di) into four regions (eg, in≦th0, th0<in≦th1, th1<in ≤th2, th2<in), and the classification result is encoded into 2 bits and output. The quantization unit 58 can also perform batch normalization and calculation of an activation function together with quantization by setting quantization parameters q (th0, th1, th2).

The quantization unit 58 performs quantization by setting the threshold th0 as β(c) in Equation 4 and threshold differences (th1−th0) and (th2−th1) as α(c) in Equation 4, The Batch Normalization operation shown in Equation 4 can be performed together with quantization. α(c) can be reduced by increasing (th1-th0) and (th2-th1). α(c) can be increased by decreasing (th1-th0) and (th2-th1).

Quantization unit 58 may perform a ReLU operation of the activation function in conjunction with quantization of the input in(di). For example, quantization unit 58 saturates the output values in regions where in(di)≤th0 and th2<in(di). Quantization unit 58 may perform activation function computation in conjunction with quantization by setting the quantization parameter q such that the output is non-linear.

State controller 54 controls the states of vector operation circuit 52 and quantization circuit 53 . Also, the state controller 54 is connected to the controller 6 via an internal bus IB. The state controller 54 has an instruction queue 55 and a control circuit 56 .

The instruction queue 55 is a queue in which the instruction command C5 for the quantization arithmetic circuit 5 is stored, and is composed of a FIFO memory, for example. The instruction command C5 is written to the instruction queue 55 via the internal bus IB.

The control circuit 56 is a state machine that decodes the instruction command C5 and controls the vector operation circuit 52 and the quantization circuit 53 based on the instruction command C5. The control circuit 56 has the same configuration as the control circuit 46 of the state controller 44 of the convolution circuit 4 .

The quantization operation circuit 5 writes quantization operation output data having Bd elements into the first memory 1 . Formula 7 shows a suitable relationship between Bd and Bc. In Equation 7, n is an integer.

Based on the quantization information input as the network information NW, the execution model generation unit 321 determines the presence or absence and type of pooling operation (average pooling, MAX pooling, etc.) and the presence and type of Batch Normalization operation in the quantization operation circuit 5. , the presence or absence and method of operation of the activation function (ReLU operation, etc.), the method of quantization (number of bits, etc.), and the presence or absence of other operations. When the hardware scale of the NN execution model 100 (neural network hardware model 400, neural network hardware 600) to be generated as the hardware information HW is included, the execution model generator 321 quantizes according to the designated scale. The configuration of the computing units in the computing circuit 5 is adjusted.

<DMAC generation step (S13-3)>
The execution model generation unit 321 generates the DMAC3 of the NN execution model 100 based on the hardware information HW and the network information NW (DMAC generation step). The execution model generation unit 321 generates a hardware model of the DMAC 3 from information input as the network information NW. The hardware model may be a behavioral level, an RTL (Register Transfer Level), a netlist representing connections between gates and circuit modules, or a combination thereof. . An example of the generated DMAC3 hardware model will be described below.

FIG. 18 is an internal block diagram of the generated DMAC3.
The DMAC 3 has a data transfer circuit 31 and a state controller 32 . The DMAC 3 has a dedicated state controller 32 for the data transfer circuit 31, and when an instruction command is input, DMA data transfer can be performed without the need for an external controller.

The data transfer circuit 31 is connected to the external bus EB and performs DMA data transfer between an external memory such as a DRAM and the first memory 1 . The data transfer circuit 31 also performs DMA data transfer between an external memory such as a DRAM and the second memory 2 . A data transfer circuit 31 transfers data between an external memory such as a DRAM and the convolution circuit 4 . A data transfer circuit 31 transfers data between an external memory such as a DRAM and the quantization arithmetic circuit 5 . The number of DMA channels of the data transfer circuit 31 is not limited. For example, each of the first memory 1 and the second memory 2 may have a dedicated DMA channel.

State controller 32 controls the state of data transfer circuit 31 . Also, the state controller 32 is connected to the controller 6 via an internal bus IB. The state controller 32 has an instruction queue 33 and a control circuit 34 .

The instruction queue 33 is a queue in which the instruction command C3 for the DMAC 3 is stored, and is composed of, for example, a FIFO memory. One or more instruction commands C3 are written to the instruction queue 33 via the internal bus IB.

The control circuit 34 is a state machine that decodes the instruction command C3 and sequentially controls the data transfer circuit 31 based on the instruction command C3. The control circuit 34 has the same configuration as the control circuit 46 of the state controller 44 of the convolution circuit 4 .

The execution model generation unit 321 determines the number of DMA channels, data bus width, etc. in the DMAC 3 from the information input as the network information NW.

For example, the execution model generator 321 generates the DMAC 3 with specifications (data bus width, etc.) matching the specifications of the external bus EB on the host side. By increasing the data bus width and the number of DMA channels, the data transmission speed between the external memory and the first memory 1 or the second memory 2 can be improved.

<Learning step (S14)>
In step S14, the learning unit 322 and the inference unit 323 of the neural network generating device 300 learn learning parameters of the generated NN execution model 100 using the learning data set DS (learning step). The learning step (S14) includes, for example, a learned parameter generation step (S14-1) and an inference test step (S14-2).

<Learning Step: Learned Parameter Generation Step (S14-1)>
Learning unit 322 generates learned parameters PM using NN execution model 100 and learning data D1. The learned parameters PM are the learned weight w, the quantization parameter q, and the like.

For example, when the NN execution model 100 is the execution model of the CNN 200 that performs image recognition, the learning data D1 is a combination of the input image and the teacher data T. An input image is input data a input to the CNN 200 . The teacher data T includes the type of subject captured in the image, the presence or absence of the detection target in the image, the coordinate values of the detection target in the image, and the like.

The learning unit 322 generates a learned parameter PM by supervised learning using a well-known technique such as the error backpropagation method. The learning unit 322 obtains the difference E between the output of the NN execution model 100 for the input image and the teacher data T corresponding to the input image using a loss function (error function), and calculates the weight w and the quantum update the optimization parameter q.

For example, when updating weight w, the gradient of the loss function with respect to weight w is used. The slope is calculated, for example, by differentiating the loss function. When using backpropagation, the gradient is calculated by backward propagation.

The learning unit 322 increases the precision of operations related to the convolution operation when calculating the gradient and updating the weight w. Specifically, a 32-bit floating-point weight w that is more accurate than the low-bit weight w (for example, 1 bit) used by the NN execution model 100 is used for learning. Further, the precision of the convolution operation performed in the convolution operation circuit 4 of the NN execution model 100 is improved.

The learning unit 322 increases the precision of calculations related to the activation function when calculating the gradient and updating the weight w. Specifically, a Sigmond function, which is more accurate than an activation function such as a ReLU function implemented in the quantization arithmetic circuit 5 of the NN execution model 100, is used for learning.

On the other hand, when calculating the output data for the input image by forward propagation, the learning unit 322 does not increase the precision of the convolution operation and the operation related to the activation function, and performs the operation based on the NN execution model 100. to implement. The high-precision weight w used when updating the weight w is reduced in bits by a lookup table or the like.

When the gradient is calculated and the weight w is updated, the learning unit 322 increases the accuracy of the convolution operation and the operation related to the activation function, thereby preventing the accuracy of the intermediate data in the operation from decreasing and enabling high inference. A learned parameter PM that can achieve accuracy can be generated.

On the other hand, when calculating the output data for the input image, the learning unit 322 performs calculation based on the NN execution model 100 without increasing the accuracy of the forward propagation calculation. Therefore, the output data calculated by the learning unit 322 and the output data of the NN execution model 100 using the generated learned parameter PM match.

<Learning Step: Inference Test Step (S14-2)>
The inference unit 323 performs an inference test using the learned parameters PM generated by the learning unit 322, the NN execution model 100, and the test data D2. For example, if the NN execution model 100 is the execution model of the CNN 200 that performs image recognition, the test data D2 is a combination of the input image and the teacher data T, similar to the learning data D1.

The inference unit 323 displays the progress and results of the inference test on the display unit 350 . The result of the reasoning test is, for example, the accuracy rate for the test data D2.

<Confirmation step (S15)>
In step S<b>15 , the inference unit 323 of the neural network generation device 300 causes the display unit 350 to display a message prompting the user to input confirmation regarding the result from the operation input unit 360 and a GUI image necessary for inputting information. The user inputs from the operation input unit 360 whether to accept the result of the inference test. When an input indicating that the user accepts the result of the inference test is input from the operation input unit 360, the neural network generation device 300 next performs step S16. When the user inputs an input from the operation input unit 360 indicating that the result of the inference test is not acceptable, the neural network generator 300 performs step S12 again. Incidentally, the neural network generation device 300 may return to step S11 and allow the user to re-input the hardware information HW.

<Output step (S16)>
In step S<b>16 , hardware generation unit 324 of neural network generation device 300 generates neural network hardware model 400 and the like based on hardware information HW and NN execution model 100 .

<Software generation step (S17)>
In step S17, the software generation unit 325 of the neural network generation device 300 generates the neural network hardware 600 (the neural network hardware model 400 implemented in the operation target hardware) based on the network information NW and the NN execution model 100. ) is generated. Software 500 includes software that transfers learned parameters PM to neural network hardware 600 as needed. In addition, the software 500 may include multiple forms of software for operating or controlling the neural network hardware 600, and may also include instructions and commands for use in operating or controlling the neural network hardware 600 and some operations of the NN execution model 100. may include software for

The software generation step (S17) includes, for example, a division step of dividing the input data according to the unit to be calculated, a network division step of dividing a part of the CNN 200 in order to improve the calculation efficiency, and a memory for the input data. It has an allocation process such as placing on top.

<Input data division process>
The software generation unit 325 partially converts the input data a for the convolution operation of the convolution layer 210 based on the memory capacity of the memory allocated as the first memory 1 and the second memory 2, the specifications and sizes (Bc and Bd) of the calculator, and the like. Split into tensors. The method of division into partial tensors and the number of divisions are not particularly limited. A partial tensor is formed, for example, by splitting the input data a(x+i, y+j, c) into a(x+i, y+j, co).

Furthermore, the software generator 325 develops the divided input data a and weight w in the convolution circuit 4 of the NN execution model 100 .

Further, the software generation unit 325 divides the input data a into partial tensors so that a plurality of divided input data a (2*X*Y*Bc bits) are stored in the first memory 1, for example. The software generation unit 325 divides the input data a for each layer. The units that can be easily calculated by the neural network hardware 600 are based on the number of parallel calculations that can be performed by the neural network hardware 600, the capacity and bandwidth of the first memory 1 or the second memory 2, the amount of power consumption, the calculation frequency, etc. decide. For example, when the number of parallel operations is large, it is preferable to reduce the number of divisions.

<Network division process>
The software generation unit 325 divides the network (layer) of the CNN 200 and maps it to the looped convolution operation circuit 4 and the quantization operation circuit 5 (network division step).

The software generator 325 divides the network (layer) of the CNN 200 for the entire CNN 200 . The software generator 325 divides the network (layer) of the CNN 200 so that memory transfers between the first memory 1 and the external memory by the DMAC 3 are reduced as much as possible.

Also, when the CNN 200 includes an operation for changing the tensor shape of the input data a, the network (layer) is divided before the operation. The operation to change the tensor shape of the input data a is, for example, an operation to shorten the depth direction (c direction) of the input data a and expand it in the plane direction (xy direction), or an operation to integrate the tensors (data). and so on.

<Allocation process>
The software generation unit 325 generates the software 500 that allocates the divided operations to the neural network hardware 600 for execution (allocation step). The software generation unit 325 generates arrangement information for arranging data on the memory in the allocation process.

Here, the software 500 generated by the software generation unit 325 includes a command group for controlling and operating the neural network hardware 600 . In this case, the command group includes a plurality of instruction commands for operating each circuit shown in FIG. 5 etc. at appropriate timing based on the NN execution model 100 etc. Also included are commands or software running on a processor other than neural network hardware 600, such as an external host CPU. The command group is stored in the external memory and used by the external host CPU to control the DMAC 3, the convolution circuit 4, and the quantization circuit 5 directly or via the controller 6. FIG.

FIG. 19 is a block diagram representing an example of an edge device in which neural network hardware 600 is provided. The edge device includes an external host CPU 700, a buffer memory 710 included in the external host CPU 700, an external memory 720, and neural network hardware 600, and each block is connected by an external bus EB. The command group is stored in the external memory 720, and the external host CPU 700 or via the controller 6 included in the neural network hardware 600 controls the DMAC 3, the convolution operation circuit 4, and the quantization operation circuit 5. will be In other words, the command group corresponds to part of the neural network hardware model 400 . Therefore, the NN execution model 100 can be appropriately controlled on the edge device by executing the command group at appropriate timing.

Also, in FIG. 19, it is assumed that the input data processed by the CNN 200 is multidimensional data such as image data. In order to hold such a large amount of data on the neural network hardware 600, a large memory is required, which is not preferable. Therefore, input data of each layer included in CNN 200 is held in external memory 720 at the start of calculation. By appropriately reading the divided input data, the circuit scale of the neural network hardware 600 can be suppressed. Furthermore, output data output from each layer included in CNN 200 is similarly held in external memory 720 as a calculation result. Note that the output data held in the external memory 720 becomes input data in the next layer.

In this embodiment, in the operation of inputting input data to the neural network hardware 600 and the operation of acquiring output data from the neural network hardware 600, each data is sent via the buffer memory 710 included in the external host CPU 700. It is read from and written to external memory 720 . More specifically, the external host CPU 700 acquires from the external memory 720 the input data corresponding to the layer on which the operation is to be performed, and temporarily stores it in the buffer memory 710 when the operation starts. The buffer memory 710 is composed of, for example, an SRAM or the like that can be read and written at high speed. In addition, the external host CPU 700 temporarily saves the data output as the calculation result in the target layer in the buffer memory 710 and then writes the data in the external memory 720 . As described above, the buffer memory 710 is configured by, for example, an SRAM that can be read and written at high speed.

When the input data processed by the neural network hardware 600 has a multidimensional data structure such as image data, the amount of data processed in each layer is relatively large. Furthermore, since CNN 200 includes multiple layers, neural network hardware 600 is repeatedly fed with data. As a result, reading and writing to and from the buffer memory 710 are repeated in order to obtain repeated calculation results.

Each data used for computation is held in a buffer memory 710 that can be read and written at high speed, and can be used repeatedly for efficient computation. In order to increase the efficiency of the operations to be executed, it is preferable to retain a large amount of data in the buffer memory 710 for a long period of time. This has led to an increase in the amount of buffer memory 710 used. Therefore, conventionally, the larger the number of layers included in the CNN 200, the larger the capacity of the buffer memory 710 required.

An increase in the capacity of the high-performance buffer memory 710 raises the cost and power consumption of the final product, so in order to effectively utilize limited computing resources such as edge devices, it is necessary not only to increase the efficiency of computation, but also to reduce the amount of buffer memory used. I had to hold back.

FIG. 20 shows an example of the CNN 200 (upper diagram) and a conventional allocation example (lower diagram) for the buffer memory 710 in the edge device in which it operates. The CNN 200 of this embodiment includes 7 layers and 1 layer of post-processing.

Input data of the CNN 200 in FIG. 20 is image data, and each layer includes the convolution operation and quantization operation shown in FIG. Input data is first input to layer _L0 , and operations are performed in each layer. Then, the output data of the final layer _L6 becomes the input of the post-processing P. Here, the post-processing P includes image processing for outputting a resultant image as output data. Examples of image processing include black level adjustment, gain adjustment, color adjustment, and the like. In addition, processing of bounding boxes may be included as processing other than image processing. As for the post-processing P, it is not necessary for all of the processing to be performed on the neural network hardware 600, and may be configured to be performed on a plurality of external processors such as the external host CPU 700. FIG. When the post-processing P is executed by the external host CPU 700, the degree of freedom of processing can be improved.

In order to execute processing in each layer of CNN 200 in FIG. 20, external host CPU 700 supplies input data to neural network hardware 600 or the like via buffer memory 710 at appropriate timing. Specifically, first, data D ₀ necessary for the calculation of layer L ₀ is read out from the external memory 720 and stored in the buffer memory 710 at the start of the calculation of the layer L ₀ . Then, the data _D0 is held in the buffer memory 710 in order to supply the data _D0 in accordance with the execution period of the arithmetic processing in the layer _L0 .

Next, the computation in layer _L1 is started in parallel with the computation in layer _L0 . Therefore, in accordance with this start, the data _D1 necessary for the calculation of the layer _L1 is read out from the external memory 720 and stored in the buffer memory 710 . Then, the data _D1 is held in the buffer memory 710 in order to supply the data _D1 in accordance with the execution period of the arithmetic processing in the layer _L1 . In the present embodiment, at _{least a} part of the operations in layer _L0 and the operations in layer _L1 are performed in parallel. Duplicate. The output data of layer _L0 may be overwritten in the used area of data _D0 , or may be directly written to the external memory 720. FIG.

As shown in FIG. 20, the operation of reading the corresponding data on the buffer memory 710 in accordance with each layer calculation and holding it until the calculation is completed is performed for all layers of the CNN 200 . Specifically, the data Dp is held until the post-processing P, which is the processing after the final layer L6, is completed, and the calculation result of the post _- processing P is written into the external memory 720, thereby completing the processing of the CNN 200.

In controlling the buffer memory ₇₁₀ corresponding to _each layer of the _{CNN 200} shown in FIG. It must be held on the buffer memory together with the data Dn ₊₁ . In FIG. 20, the timing for reading data _Dn is indicated by a dotted line, and the period during which data _Dn is held in the buffer memory 710 (hereinafter referred to as "lifetime") is indicated by a rectangle. In the CNN 200 shown in FIG. 20, operations are performed in parallel between consecutive layers, so the lifetime of data _Dn and the lifetime of data Dn ₊₁ temporally overlap. The lifetime of each data corresponds to the calculation period of each layer.

Since the data _Dn and the data Dn ₊₁ are different data, they cannot be held in the same memory area of the buffer memory 710 . Therefore, it is necessary to provide a dedicated area corresponding to each layer. According to conventional allocation, as shown in FIG. The required memory capacity of the buffer memory 710 increases depending on the

Next, allocation according to this embodiment will be described in detail. In the allocation process included in the process of generating the software 500, the software generator 325 performs allocation on the buffer memory based on temporally overlapping lifetimes in order to reduce the required amount of usage of the buffer memory 710. .

FIG. 21 is a flow chart for explaining the steps included in the allocation process in this embodiment. Each step of the flowchart is executed by the software generation unit 325 .

When the process is started, the software generation unit 325 acquires information about the target CNN 200 based on the network information NW, the NN execution model 100, or the like in step S21. Specifically, each layer in the CNN 200 is determined, and the calculation content/order, the type/size of the input data, the input order of the input data, and the like regarding each determined layer are acquired. Then the process moves to the next step.

At step S22, the software generation unit 325 determines the lifetime for each layer in the CNN 200 based on the information on the CNN 200 acquired at step S21. As a specific determination method, for example, the software generation unit 325 creates a directed acyclic graph corresponding to the CNN 200 based on a plurality of pieces of acquired information, with each layer treated as one node. As an example, a directed acyclic graph is created with one or more operations divided in the network division step as one node. Then, determine the topological order for each node. In the directed acyclic graph created, the topological order considers the direction of the graph, arbitrarily selects one node that has no incoming edge to the node, gives it a number, and connects this node and this node. It can be determined by repeating processing such as excluding branches that are present. In this embodiment, the multiple pieces of information acquired by the software generation unit 325 correspond to information regarding neural network models, and the software generation unit 325 corresponds to an analysis unit that analyzes the information.

The software generation unit 325 further uses the topological order in the directed acyclic graph corresponding to the NN execution model 100 and the information about the CNN 200 acquired in step S21 to determine the start time and lifetime of processing in each layer. do. Then the process moves to the next step. The start time of the processing of this embodiment is determined based on the timing of loading necessary data onto the buffer memory 710 or acquiring from the neural network hardware 600 . The start times need not be absolute timings, but may be defined as relative orders between nodes. Also, the lifetime is determined based on the start time of processing and the input timing of inputting predetermined data (when inputting multiple times, the last input timing). As an example, when a predetermined node requires G data inputs, the lifetime can be determined as _G ×T _G , where TG is the unit period required for data input. In other words, the lifetime includes at least one timing for inputting data, and can be determined from the order of operations between layers and the period required for the operations. Also, the lifetime end time can be determined from the start time and the duration of the lifetime.

In step S23, the software generation unit 325 allocates groups based on the lifetime for each layer of the NN execution model 100 determined in step S22. As an example of group assignment, lifetime periods that do not overlap can be grouped together and assigned to a plurality of target lifetimes. More specifically, when two lifetimes are arbitrarily specified, the end time of one lifetime is compared with the start time of the other lifetime. In this case, if the end time of one lifetime arrives earlier than the start time of the other lifetime, these two lifetimes are grouped. This process is performed for all lifetimes (step S24). As a result, multiple groups are formed, and all lifetimes are assigned to one of the groups.

Here, as an example of a method of allocating lifetimes to groups, the end time and start time of two arbitrarily specified lifetimes are focused. If the end time of one lifetime arrives earlier than the start time of the other lifetime, a branch can be extended between these two lifetimes. The result of doing this for all lifetimes can be regarded as a directed graph without cycles. In other words, there is a one-to-one correspondence between a set of non-overlapping lifetimes and a directed path on the directed graph. Therefore, lifetime group allocation corresponds to covering all vertices of a directed graph by directed paths that do not intersect each other on the directed graph (hereinafter referred to as path coverage). Then, the grouping with the minimum number of lifetime groups can be obtained from the minimum path coverage of the directed graph.

In step S25, the software generation unit 325 determines the processing order of each group determined by the allocation up to step S24. In this determination, it is preferable to determine the processing order in order from the group including the layer with the earliest start time, but it is not limited to this. For example, it may be determined by the number of allocated layers or the presence or absence of post-processing. Then, when the allocation ends, the processing related to this flowchart ends. Note that the reduction in memory usage obtained as an effect of the present embodiment mainly depends on the number of groups, so it is less dependent on the order of the groups. Therefore, the processing order between groups can be changed as appropriate. As an example, the number of groups also depends on the number of branches R within the neural network. Specifically, by performing lifetime-based allocation according to the present embodiment, the maximum number of groups can be suppressed to within twice the number R of branches.

FIG. 22 shows an example of the CNN 200 (upper diagram) and an allocation example (lower diagram) according to the present embodiment regarding use of the buffer memory 710 in the edge device on which it operates. To simplify the explanation, the CNN 200 is assumed to have the same configuration as that shown in FIG.

In the present embodiment, as shown in FIG. 22, the software generation unit 325 allocates to a plurality of groups (for example, group 1 and group 2) in the allocation process, taking into account temporally overlapping lifetimes. Allocate on the buffer memory. More specifically, each lifetime is analyzed and the start and end times of each lifetime are compared and grouped as solutions to the path coverage problem. As _an example, since the end time of data _D0 comes before the start time of data D2, these _two are allocated to the same group, and the end time of data _D0 comes before the start time of data D1. Since it never arrives, the two are assigned to different groups.

In this embodiment, data D ₀ , data D ₂ , data D ₄ and data D ₆ are grouped as group 1, and data D ₁ , data D ₃ , data D ₅ , data D ₇ and data D _p are grouped as group 2. is assigned.

As shown in FIG. 22, the memory usage of buffer memory 710 depends on the number of groups, and can be greatly reduced compared to the conventional example of FIG. Furthermore, as an effect of the allocation according to the present embodiment, memory usage can be suppressed without depending on an increase in the number of layers by performing allocation to groups based on lifetime. Note that the lifetime-based allocation process in this embodiment is not limited to the form of the CNN 200 shown in FIG. can.

In the conventional allocation process for CNN 200 with multiple layers, what kind of allocation is performed depends on the heuristic allocation due to the complexity of repeated operations in the neural network. Therefore, it has been difficult to allocate an appropriate amount for operations related to the neural network, and it has been difficult to estimate the amount of buffer usage at the time of design. However, by calculating the minimum path coverage using temporally overlapping lifetimes as nodes in the present invention, it is possible not only to perform appropriate allocation in polynomial time, but also to estimate the buffer usage at the design time. be able to.

In this embodiment, the software generation unit 325 allocates the divided operations to the neural network hardware 600 to generate the software 500 for execution. is not limited to For example, divided data obtained by dividing one layer may be targeted.

In this embodiment, the target memory is the buffer memory 710 provided in the external host CPU 700, but other memory may be the target of the allocation process. For example, when a plurality of memories are provided on the external host CPU 700, or when a memory is provided in a circuit that relays the external host CPU 700 and the neural network hardware 600, the memory may be targeted.

As described above, the neural network generation device 300 according to the present embodiment can generate and control a neural network that can be embedded in an embedded device such as an IoT device and that can operate with high performance.

As described above, an example of the embodiment of the present invention has been described in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and design changes and the like are included within the scope of the present invention. . Also, the constituent elements shown in the above-described embodiment and modifications can be combined as appropriate.

(Modification 1)
In the above embodiment, the first memory 1 and the second memory 2 are different memories, but the aspect of the first memory 1 and the second memory 2 is not limited to this. The first memory 1 and the second memory 2 may be, for example, a first memory area and a second memory area in the same memory.

(Modification 2)
For example, the data input to the NN execution model 100 described in the above embodiment is not limited to a single format, and can be composed of still images, moving images, sounds, characters, numerical values, and combinations thereof. . It should be noted that the data input to the NN execution model 100 can be mounted on the edge device in which the neural network hardware model 400 is provided, such as an optical sensor, a thermometer, a Global Positioning System (GPS) measuring instrument, an angular velocity measuring instrument, an anemometer It is not limited to the measurement result in a physical quantity measuring instrument such as Peripheral information such as base station information, vehicle/vessel information, weather information, and congestion information received from peripheral devices via wired or wireless communication, and different information such as financial information and personal information may be combined.

(Modification 3)
Edge devices provided with the NN execution model 100 are assumed to be communication devices such as mobile phones driven by batteries, smart devices such as personal computers, mobile devices such as digital cameras, game devices, and robot products, but are limited to these. It is not something that can be done. Unprecedented effects can also be obtained by using power on Ethernet (PoE), etc., to limit the peak power that can be supplied, reduce product heat generation, or use it for products that require long-time operation. For example, by applying it to in-vehicle cameras installed in vehicles and ships, surveillance cameras installed in public facilities and roads, etc., it is possible not only to realize long-time shooting, but also to contribute to weight reduction and durability. . Similar effects can be obtained by applying the present invention to display devices such as televisions and displays, medical equipment such as medical cameras and surgical robots, and work robots used at manufacturing sites and construction sites.

It may be realized by recording the program in the above-described embodiment in a computer-readable recording medium and causing a computer system to read and execute the program recorded in the recording medium. It should be noted that the "computer system" referred to here includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. Furthermore, "computer-readable recording medium" means a medium that dynamically retains a program for a short period of time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It may also include something that holds the program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in that case. Further, the program may be for realizing part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in the computer system.

Also, the effects described herein are merely illustrative or exemplary, and are not limiting. In other words, the technology according to the present disclosure can produce other effects that are obvious to those skilled in the art from the description of this specification, in addition to or instead of the above effects.

The present invention can be applied to the generation of neural networks.

300 Neural network generator 200 Convolutional neural network (CNN)
100 neural network execution model (NN execution model)
400 neural network hardware model 500 software 600 neural network hardware 1 first memory 2 second memory 3 DMA controller (DMAC)
4 convolution operation circuit 42 multiplier 43 accumulator circuit 5 quantization operation circuit 52 vector operation circuit 53 quantization circuit 6 controller 61 register PM learned parameter DS learning data set HW hardware information NW network information

Claims (7)

A software generation device for generating software for controlling a neural network circuit,
analysis means for analyzing information about a model including multiple layers operating in the neural network circuit;
determining means for determining lifetimes corresponding to the plurality of layers included in the model based on the analysis result of the analyzing means;
generating means for generating the software based on the lifetime;
A software generation device comprising: 2. The software generator according to claim 1, wherein said software includes a plurality of commands for controlling said neural network circuit. 3. The software generating apparatus according to claim 1, wherein said lifetime includes one or more timings for inputting data to said neural network circuit in a corresponding layer. the software includes arrangement information of the data held on the external memory;
4. The software generating apparatus according to claim 1, wherein said generating means generates said allocation information based on a group to which a plurality of lifetimes are assigned. the lifetime corresponds to a computation period in the corresponding layer;
5. The software generating apparatus according to any one of claims 1 to 4, wherein at least some of the corresponding computation periods in the plurality of layers overlap in time. The neural network circuit includes a convolution operation circuit that performs a convolution operation and a quantization operation circuit that performs a subquantitative operation based on the result of the convolution operation,
6. The software generating apparatus according to claim 1, wherein said convolution operation circuit and quantization operation circuit are configured in a loop shape. A software generation method for generating software for controlling a neural network circuit,
an analysis step of analyzing information about a model including multiple layers operating in the neural network circuit;
a determination step of determining lifetimes corresponding to the plurality of layers included in the model based on the analysis result of the analysis means;
a generating step of generating the software based on the lifetime;
A software generation method, comprising:

Images