JP7040771B2 - Neural network processing equipment, communication equipment, neural network processing methods, and programs - Google Patents

Description

The present invention relates to a neural network processing apparatus and a neural network processing method.

In recent years, a convolutional neural network (CNN) has been attracting attention as a deep neural network for classifying images into a plurality of categories. CNNs are characterized by having a convolutional layer in a deep neural network. The convolution layer applies a filter to the input data. More specifically, in the convolution layer, the window of the filter is slid with a constant stride, the element of the filter is multiplied by the corresponding element of the input data, and the product-sum operation is performed to obtain the sum.

FIG. 11 is a diagram showing a general CNN signal processing flow. The CNN has an input layer, an intermediate layer, and an output layer (see, for example, Non-Patent Document 1). In the intermediate layer, a convolution operation is performed in which the input signal is weighted. As shown in FIG. 11, in the intermediate layer, an activation function such as ReLU is applied to the result of the convolution operation as needed, and further, normalization and pooling processing are performed in some cases. Further, the characteristics of the input signal extracted through the convolution operation are applied to a classifier composed of a fully connected layer, and the classification result is output from the output layer. As described above, one of the features of a neural network such as CNN is that the product-sum operation is repeatedly performed.

Here, the input value and weight of the input data used for CNN may include a decimal point, but in the product-sum operation of a conventional neural network such as CNN, the "input signal" and "weight" in FIG. 11 And, as shown in each value of "convolution operation", the operation process is performed in a form that secures the number of digits of the operation result.

Hideki Asoh, et al., "Deep Learning Deep Learning", Modern Science Co., Ltd., November 2015

However, when a conventional neural network such as CNN is implemented by embedded hardware such as FPGA (Field Programmable Gate Array) or a microcomputer, a decrease in processing speed due to a product-sum operation having a large number of digits has been a problem.

The present invention has been made to solve the above-mentioned problems, and to provide a neural network processing device, a communication device, a neural network processing method , and a program capable of suppressing a decrease in the processing speed of a neural network. The purpose is to provide.

In order to solve the above-mentioned problems, the neural network processing apparatus according to the present invention includes a first memory for storing an input signal given to the multi- layered neural network, a second memory for storing the weights of the multi- layered neural network, and the above -mentioned. A processor that performs a convolution operation of the multi- layer neural network including a product-sum operation of the input signal and the weight stored in the first memory and the second memory, respectively , and the operation result of the convolution operation and the multi-layer. The convolution operation is not an unquantized operation result with built-in quantization that reduces the bit accuracy of the data, with a buffer that stores the operation result used for the next operation in the neural network. It is a convolutional convolution operation in which convolution and quantization are performed collectively so that only the quantized operation result is stored in the buffer, and the weight used in the product-sum operation is stored in the second memory. The first function that realizes the quantization convolution operation by being quantized to 1-bit bit accuracy in advance before being performed, or quantized to 1-bit bit accuracy by the processor after being stored in the second memory. The processor further includes a third memory for storing the data, reads the first function from the third memory, and performs the quantization convolution operation using the read first function.

Further, the processor may perform processing using the activation function on the quantized calculation result. The processor may perform quantization to reduce bit precision with respect to the bias used in the quantization convolution operation. The multi-layer neural network has a plurality of convolution layers in which the quantization convolution operation is sequentially performed by the processor, and the processor is the output of each of the convolution layers other than the final layer among the plurality of convolution layers. The quantized calculation result input to the next convolution layer may be stored in the auxiliary storage device.

Further, a fourth memory for storing the second function for quantizing the weight is further provided, and the processor reads the second function from the fourth memory and uses the read second function to store the weight. It may be quantized. The processor may alternately read the second function and the first function to perform the quantization of the weight and the quantized convolution operation in each of the plurality of convolution layers.

Further, the neural network further includes an input buffer for storing the preprocessed image signal as the input signal, and the preprocessing applied to the image signal is one of monochrome conversion, contrast adjustment, and luminance adjustment. May include one. Further, the quantization of the weight may be performed on the FPGA.

In order to solve the above-mentioned problems, the communication device according to the present invention is a communication device including the neural network processing device, and further includes an interface circuit for communicating with an external electronic device via the communication network. , The data used for the quantization convolution operation is communicated from the external electronic device via the communication network. Further, the weights quantized in advance may be received from the external electronic device via the communication network. Further, the first function may be received from the external electronic device via the communication network.

In order to solve the above-mentioned problems, the neural network processing method according to the present invention has a first step of storing an input signal given to a multilayer neural network in a first memory and a weight of the multilayer neural network in a second memory. A second step of storage and a third step in which the processor performs a convolution operation of the multilayer neural network including a product-sum operation of the input signal and the weight stored in the first memory and the second memory, respectively. The convolution operation comprises a fourth step of storing the operation result of the convolution operation by the buffer and used for the next operation in the multi-layer neural network, and the convolution operation is a quantum that reduces the bit accuracy of the data. It is a quantized convolution operation that performs convolution and quantization collectively so that only the quantized operation result is stored in the buffer, not the non-quantized operation result with built-in conversion. The weights used in the calculation are either quantized to a bit precision of 1 bit in advance before being stored in the second memory, or stored in the second memory and then stored in the second memory up to a bit precision of 1 bit by the processor. It is quantized, and in the third step, the processor reads the function from a third memory that stores a function that realizes the quantization convolution operation, and performs the quantization convolution operation using the read function. To prepare for .

In order to solve the above-mentioned problems, the program according to the present invention is a computer including a first memory for storing an input signal given to a multi-layer neural network and a second memory for storing the weights of the multi-layer neural network. In the arithmetic processing that performs the convolution operation of the multi-layer neural network including the product-sum operation of the input signal and the weight stored in the first memory and the second memory, respectively, in the processor, and the operation result of the convolution operation. Yes The convolution operation incorporates quantization that reduces the bit accuracy of the data, which is to execute a storage process that stores the operation result used for the next operation in the multi-layer neural network in a buffer. It is a quantized convolution operation that performs convolution and quantization collectively so that only the quantized operation result is stored in the buffer, not the operation result that has not been performed, and the weight used in the product-sum operation is The computer is either quantized to a bit precision of 1 bit in advance before being stored in the second memory, or is quantized to a bit precision of 1 bit by the processor after being stored in the second memory. A third memory for storing a function for realizing the quantization convolution operation is further provided, and the calculation process performs a process of reading the function from the third memory and a process of reading the function to perform the quantization convolution operation. Processing and including. The program may cause the processor to execute a process using the activation function on the quantized operation result. The processor may be made to perform a process of performing quantization that reduces bit accuracy with respect to the bias used in the quantization convolution operation. The multi-layer neural network has a plurality of convolution layers in which the quantization convolution operation is sequentially performed by the processor, and the output of each of the convolution layers other than the final layer among the plurality of convolution layers is output to the processor. The process of storing the quantized calculation result input to the next convolution layer in the auxiliary storage device may be executed.

According to the present invention, it is possible to suppress a decrease in the processing speed of the neural network.

FIG. 1 is a block diagram illustrating a function of the CNN processing apparatus according to the first embodiment of the present invention. FIG. 2 is a block diagram showing a hardware configuration of the CNN processing apparatus according to the first embodiment. FIG. 3 is a diagram for explaining the flow of the CNN processing method according to the first embodiment. FIG. 4 is a diagram for explaining the CNN processing method according to the first embodiment. FIG. 5 is a block diagram illustrating the function of the CNN processing apparatus according to the second embodiment. FIG. 6 is a diagram for explaining the flow of the CNN processing method according to the second embodiment. FIG. 7 is a diagram for explaining the CNN processing method according to the second embodiment. FIG. 8 is a block diagram illustrating the function of the CNN processing apparatus according to the third embodiment. FIG. 9 is a diagram for explaining the flow of the CNN processing method according to the third embodiment. FIG. 10 is a diagram for explaining the CNN processing method according to the third embodiment. FIG. 11 is a diagram for explaining a conventional CNN arithmetic process.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to FIGS. 1 to 10. In the following, a case where CNN is used as the "neural network" will be described as an example.
[First Embodiment]
First, the configuration of the CNN processing device (neural network processing device) 1 according to the first embodiment of the present invention will be described. FIG. 1 is a block diagram showing a functional configuration of the CNN processing device 1.

The CNN processing device 1 according to the present embodiment is a calculation processing device that performs a product-sum calculation of an input signal given to a CNN and a weight of a CNN and outputs a calculation result. This arithmetic processing includes a product-sum operation of convolutional layers constituting the intermediate layer of the CNN (hereinafter, may be referred to as “convolutional arithmetic”). The calculation result obtained by the CNN processing device 1 performing the convolutional operation is applied to an activation function such as ReLU to obtain an output of one convolutional layer. In the following, for the sake of simplicity, it is assumed that the result of the product-sum operation of the convolution layer, that is, the result of the convolution operation is used as the input signal of the next convolution layer. The CNN processing device 1 repeatedly performs a product-sum calculation of the input signal and the weight, and executes a product-sum calculation a number of times according to the number of convolutional layers possessed by the preset CNN model.

[Functional block of CNN processing device]
The CNN processing device 1 described above includes an input buffer (first memory) 10, a weight buffer (second memory) 11, a convolution calculation unit 12, a calculation result buffer 13, a quantization processing unit 14, an output buffer 15, and a storage unit 16. To prepare for.

The input buffer 10 stores an input signal given to the CNN. More specifically, the input buffer 10 stores, for example, an input signal such as image data given from the outside. Further, the input buffer 10 outputs an input signal such as image data to the quantization processing unit 14. The input signal supplied to the input buffer 10 may be image data that has been preprocessed in advance. Examples of preprocessing include monochrome conversion, contrast adjustment, and luminance adjustment. Further, the input signal may be reduced to a bit depth set according to the CNN model preset in the CNN processing device 1. As the value of the input signal supplied to the input buffer 10, for example, a value including a decimal point represented by a 32-bit or 16-bit precision floating-point array is used.

The weight buffer 11 stores the weight of the CNN. More specifically, the weight buffer 11 is loaded with CNN weight parameters stored in advance in a server (not shown) installed outside the CNN processing device 1, a storage unit 16, or the like. In the present embodiment, as the weight value, a value including a decimal point represented by a 32-bit or 16-bit precision floating-point array is used. The weight buffer 11 outputs the buffered weight to the quantization processing unit 14 described later.

The convolution calculation unit 12 performs a CNN convolution operation including a product-sum operation of an input signal stored in the input buffer 10 and a weight stored in the weight buffer 11 (first process). More specifically, the convolutional calculation unit 12 performs a product-sum calculation of the convolutional layers constituting the CNN model preset in the CNN processing device 1. In the present embodiment, the convolution calculation unit 12 performs the convolution calculation based on the input signal and the weight value quantized by the quantization processing unit 14 described later. The calculation result output by the convolution calculation unit 12 is supplied to the calculation result buffer 13.

The calculation result buffer 13 buffers the result of the convolution operation by the convolution calculation unit 12, and supplies the calculation result to the quantization processing unit 14.

The quantization processing unit 14 performs quantization that reduces the bit precision of at least a part of the data used in the convolution operation processing (first processing) (second processing). More specifically, the quantization processing unit 14 quantizes at least a part of the data of the input signal, the weight, and the operation result of the convolution operation. In the present embodiment, the quantization processing unit 14 performs the quantization processing on all the data of the input signal, the weight, and the calculation result of the convolution operation.

Specifically, the quantization processing unit 14 quantizes the input signal read from the input buffer 10, quantizes the weight read from the weight buffer 11, and quantizes the result of the convolution operation read from the calculation result buffer 13. I do. Further, the quantization processing unit 14 converts the input signal, the weight, and the calculation result stored in the input buffer 10, the weight buffer 11, and the calculation result buffer 13, respectively, into the quantized input signal, the weight, and the calculation result. Update to the value of.

The quantization process performed by the quantization processing unit 14 includes well-known rounding processes such as rounding, rounding up, rounding down, and nearest rounding, and is used for each value of the input signal, weight, and convolution operation result. On the other hand, restrictions such as converting values including decimal points into integers are applied. The improvement in processing speed associated with the quantization process and the accuracy of the processing result are in a trade-off relationship with each other. The quantization of each value performed by the quantization processing unit 14 is processing speed and accuracy by reducing the number of bits according to the number of bits that can be handled by the processor 102 included in the CNN processing device 1 such as 8 bits and 16 bits at one time. Can be achieved at the same time. For example, in the case of FPGA, each value may be reduced to 1 bit. The quantization processing unit 14 may be used only for a part of the plurality of convolutional layers connected in multiple stages constituting the CNN handled by the CNN processing apparatus 1.

The output buffer 15 temporarily stores the result of the convolution operation quantized by the quantization processing unit 14.

The storage unit 16 stores the result of the quantized convolution operation temporarily stored in the output buffer 15. The result of the convolution operation stored in the storage unit 16 is stored as the output value of the convolution layer of the CNN in the present embodiment and the input value of the next convolution layer. Further, the storage unit 16 stores in advance information indicating a rounding method used by the quantization processing unit 14 to quantize the input signal, the weight, and the value as a result of the convolution operation.

[Hardware configuration of CNN processing device]
Next, an example of the hardware configuration of the CNN processing device 1 having the above-mentioned functions will be described with reference to the block diagram of FIG.

As shown in FIG. 2, the CNN processing device 1 includes, for example, a computer including a processor 102, a main storage device 103, a communication interface 104, an auxiliary storage device 105, and an input / output device 106 connected via a bus 101, and these. It can be realized by a program that controls the hardware resources of.

The main storage device 103 stores in advance a program for the processor 102 to perform various controls and operations. The processor 102 and the main storage device 103 realize each function of the CNN processing device 1 including the convolution calculation unit 12 and the quantization processing unit 14 shown in FIG.

The main storage device 103 realizes the input buffer 10, the weight buffer 11, the calculation result buffer 13, and the output buffer 15 described with reference to FIG.

The communication interface 104 is an interface circuit for communicating with various external electronic devices via the communication network NW. Input signals such as image data used by the CNN processing device 1 and weights may be received from an external server or the like via the communication interface 104.

The auxiliary storage device 105 is composed of a readable / writable storage medium and a drive device for reading / writing various information such as programs and data to the storage medium. A semiconductor memory such as a hard disk or a flash memory can be used as the storage medium in the auxiliary storage device 105.

The auxiliary storage device 105 has a storage area for storing input data and weights acquired from the outside, and a program storage area for storing a program for the CNN processing device 1 to perform CNN arithmetic processing such as a convolution operation. The auxiliary storage device 105 realizes the storage unit 16 described with reference to FIG. Further, for example, it may have a backup area for backing up the above-mentioned data, programs, and the like.

The input / output device 106 is composed of an I / O terminal that inputs a signal from an external device and outputs a signal to the external device. A display device (not shown) may be provided via the input / output device 106 to display the calculation result output by the CNN processing device 1.

Here, the program stored in the program storage area of the auxiliary storage device 105 may be a program in which processing is performed in chronological order according to the order of the CNN processing method described in the present specification, and in parallel, Alternatively, it may be a program in which processing is performed at a required timing such as when a call is made. Further, the program may be processed by one computer or may be distributed processed by a plurality of computers.

[CNN processing method]
Next, the operation of the CNN processing apparatus 1 having the above-described configuration will be described with reference to FIGS. 3 and 4. First, the input buffer 10 and the weight buffer 11 temporarily store the input signal A and the weight U given from the server installed outside the CNN processing device 1, respectively (steps S1 and S3).

As shown in FIG. 4, the input signal A is stored in the input buffer 10. The input signal A is vectorized input image data and has vertical and horizontal dimensions. The value of the input signal A is represented by a value including a decimal point. The rectangle shown by the thick line in FIG. 4 represents the filter. Further, the weight U is stored in the weight buffer 11. The weight U is an element of the kernel represented by a matrix and is a parameter that is adjusted and updated by the learning of CNN and finally determined. The value of the weight U also has vertical and horizontal dimensions, and each element is represented by a value including a decimal point.

Next, the quantization processing unit 14 reads the input signal A from the input buffer 10 and quantizes the value of each element (step S2). Further, the quantization processing unit 14 reads the weight U from the weight buffer 11 and quantizes the value of each element (step S4).

More specifically, as shown in FIG. 4, the value of the input signal A including the decimal point is rounded off, for example, by the quantization processing unit 14. The quantization processing unit 14 updates the value of the input signal A of the input buffer 10 with the value of the quantized input signal A'.

Further, as shown in FIG. 4, the value of the weight U including the decimal point is rounded off, for example, by the quantization processing unit 14. The quantization processing unit 14 updates the value of the weight U of the weight buffer 11 with the value of the quantized weight U'.

Returning to FIG. 3, the convolution calculation unit 12 reads the quantized input signal A'and the quantized weight U'from the input buffer 10 and the weight buffer 11, respectively, and performs the convolution calculation (step S5). More specifically, the convolution unit 12 multiplies the vector of the quantized input signal A'and the matrix of the quantized weight U'.

Specifically, as shown in FIG. 4, the CNN filter window (in the example of FIG. 4, the thick line square shown by 2 × 2) is slid with a predetermined stride. The convolution unit 12 multiplies the element of the quantized weight U'and the corresponding element of the quantized input signal A'at each location of the filter and obtains the sum.

The convolution calculation unit 12 stores the calculation result B of the convolution calculation by the product-sum calculation in the corresponding location of the calculation result buffer 13 (step S6). Specifically, as shown in FIG. 4, the operation result buffer 13 stores the product-sum result “37” not including the decimal point.

After that, the quantization processing unit 14 reads the calculation result of the convolution operation obtained in step S5 from the calculation result buffer 13 and performs the quantization process (step S7). Specifically, as shown in FIG. 4, the quantization processing unit 14 rounds off the operation result “37” of the convolution operation, for example, and quantizes it into “40”. The quantization processing unit 14 updates the data in the calculation result buffer 13 with the quantized calculation result B'.

Next, the processor 102 of the CNN processing device 1 reads the operation result B'of the convolution operation quantized from the operation result buffer 13 and applies an activation function such as ReLU (step S8). Specifically, when the calculation result B'is a negative value, the processor 102 outputs 0 through the ReLU function, and the positive calculation result B'outputs the value as it is.

Next, the processor 102 performs a well-known pooling process on the value output in step S8, and compresses the result of the convolution operation (step S9). The pooling process in step S9 may be performed as needed. Further, the processor 102 may perform normalization on the output of the activation function (ReLU) obtained in step S8 (see Non-Patent Document 1).

The pooled quantized calculation result B'is stored in the output buffer 15, read out by the processor 102, and output (step S10). The output value is input to the fully coupled layer constituting the subsequent classifier (not shown) as the output of the feature extraction unit of the CNN, and the image data of the input signal A is discriminated.

As described above, according to the CNN processing device 1 according to the first embodiment, the input signal of the CNN, the weight, and the calculation result of the convolution operation are quantized. Therefore, when the embedded hardware is used. Even so, it is possible to suppress a decrease in the processing speed of CNN.

Further, according to the CNN processing device 1, in particular, since the calculation result of the convolution operation is also quantized, the calculation load of the entire CNN composed of a large number of layers can be reduced, and the signal processing can be speeded up. ..

In the embodiment described, the addition of the bias is omitted in the description of the multiply-accumulate operation for the sake of simplicity. However, the vector of the input signal may be multiplied by the weight matrix, and the bias value added thereafter may be quantized in the same manner.

Further, in the described embodiment, the case where the quantization processing unit 14 quantizes all the data of the input signal, the weight, and the operation result of the convolution operation has been described. However, the quantization processing unit 14 may be configured to quantize the data contained in any one or two of the input signal, the weight, and the calculation result of the convolution operation.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the following description, the same reference numerals will be given to the same configurations as those of the first embodiment described above, and the description thereof will be omitted.

In the first embodiment, the case where the quantization processing unit 14 performs the quantization processing on the input signal A, the weight U, and the operation result B of the convolution operation in the CNN has been described. On the other hand, in the CNN processing apparatus 1A according to the second embodiment, the weight U'quantized in advance is used. Further, the CNN processing apparatus 1A includes a quantized convolution calculation unit 12A that performs a convolution calculation incorporating a quantization process.

[Functional block of CNN processing device]
FIG. 5 is a block diagram showing a functional configuration of the CNN processing device 1A according to the second embodiment. The CNN processing device 1A includes an input buffer 10, a quantization weight buffer (second memory) 11A, a quantization convolution calculation unit 12A, a calculation result buffer 13, an output buffer 15, and a storage unit 16. Hereinafter, a configuration different from that of the first embodiment will be mainly described.

The quantization weight buffer 11A stores the weight (quantization weight) in which the weight is quantized in advance. The quantized weight buffer 11A temporarily stores pre-quantized weights acquired from, for example, a server (not shown) installed outside the CNN processing device 1A. For example, the CNN processing device 1A may acquire the learned weight of the CNN from a server or the like installed outside in advance via the communication network NW and store it in the storage unit 16 in advance. In this case, the quantized weight buffer 11A reads the quantized weight from the storage unit 16 and temporarily stores it. Here, the quantized weight is a weight reduced to a bit accuracy according to the processing capacity of the CNN processing device 1A, as in the first embodiment, and for example, the weight including the decimal point is converted into an integer. Is included.

The quantized convolution calculation unit 12A reads the input signal and the quantized weight from the input buffer 10 and the quantized weight buffer 11A, respectively, and performs the convolution calculation. More specifically, the quantized convolution calculation unit 12A also performs signal quantization processing when performing the convolution calculation. For example, the quantization convolution calculation unit 12A performs a process of limiting the number of digits of the calculation result of the convolution operation to an integer in advance, or a calculation of performing a bit shift before the calculation when the number of digits of the calculation result is limited. Is executed when executing the convolution operation. The calculation result by the quantized convolution calculation unit 12A is temporarily stored in the calculation result buffer 13.

[CNN processing method]
Next, the operation of the CNN processing apparatus 1A having the above-described configuration will be described with reference to FIGS. 6 and 7. First, it is assumed that the weight U'quantized by an external server or the like is acquired in the CNN processing device 1 via the communication network NW, and the quantized weight U'is stored in the storage unit 16.

First, the input buffer 10 temporarily stores the input signal A acquired from a server or the like installed outside the CNN processing device 1 (step S20). Further, the quantized weight buffer 11A reads the pre-quantized weight U'from the storage unit 16 and temporarily stores it (step S21).

As shown in FIG. 7, the input signal A is stored in the input buffer 10. The input signal A is vectorized input image data and has vertical and horizontal dimensions. The value of the input signal A is represented by a value including a decimal point. The rectangle shown by the thick line in FIG. 7 represents the filter. Further, the quantized weight U'is stored in the quantized weight buffer 11A. The weight U is an element of the kernel represented by a matrix and is a parameter that is adjusted and updated by the learning of CNN and finally determined. The quantized weight U'is a weight expressed by converting the weight U composed of each element including the value of the decimal point into an integer, for example, and reducing the number of bits. The quantized weight U'has vertical and horizontal dimensions as well as the weight U.

Next, the quantized convolution calculation unit 12A reads the input signal A from the input buffer 10 and the quantized weight U'from the quantization weight buffer 11A, and performs a convolution operation incorporating the quantization process (step S22). ). For example, the quantized convolution calculation unit 12A performs a process of limiting the number of digits of the calculation result of the convolution calculation to an integer. Further, for example, the quantized convolution calculation unit 12A performs a bit shift on the input signal A and the quantized weight U'in advance to perform processing on the number of digits of the data before performing the convolution calculation.

More specifically, as shown in the example of FIG. 7, the quantized convolution calculation unit 12A performs a process of truncating the decimal point at the time of the convolution calculation (C = floor (conv (A, U')). Further, the quantized convolution unit 12A multiplies the element of the quantized weight U'and the corresponding element of the input signal A at each location of the filter, and obtains the sum thereof. The operation result C of the quantized convolution operation by 12A is stored in the operation result buffer 13 (step S23).

After that, the processor 102 of the CNN processing device 1A reads the operation result C of the convolution operation quantized from the operation result buffer 13 and applies an activation function such as ReLU (step S24). Specifically, when the calculation result C is a negative value, the processor 102 outputs 0 through the ReLU function, and the positive calculation result C outputs the value as it is.

Next, the processor 102 performs a well-known pooling process on the value output in step S24, and compresses the result of the convolution operation (step S25). The pooling process in step S25 may be performed as needed. Further, the processor 102 may perform normalization on the output of the activation function (ReLU) obtained in step S24 (see Non-Patent Document 1).

The operation result C of the pooled quantized convolution operation is stored in the output buffer 15, further read by the processor 102, and output to the outside (step S26). The output value is input to the fully coupled layer constituting the subsequent classifier (not shown) as the output of the feature extraction unit of the CNN, and the image data of the input signal A is discriminated.

As described above, according to the CNN processing apparatus 1A according to the second embodiment, it is possible to reduce the arithmetic processing in the CNN convolution operation by using the weight quantized in advance. Further, since the CNN processing apparatus 1A incorporates the quantization processing when performing the convolution calculation, it is possible to reduce the calculation processing associated with the quantization of the input signal and the weight.

[Third Embodiment]
Next, a third embodiment of the present invention will be described. In the following description, the same components as those of the first and second embodiments described above are designated by the same reference numerals, and the description thereof will be omitted.

In the second embodiment, the case where the quantized convolution calculation unit 12A performs the convolution calculation incorporating the quantization process by using the weight quantized in advance has been described. On the other hand, in the third embodiment, when the weight is quantized and the quantized convolution operation is performed, the function stored in advance in the storage unit 16B is read out and the operation is performed.

[Functional block of CNN processing device]
FIG. 8 is a block diagram showing a functional configuration of the CNN processing device 1B according to the third embodiment. The CNN processing device 1B includes an input buffer 10, a quantization weight buffer 11A, a convolution calculation unit 12, a calculation result buffer 13, a quantization processing unit 14, an output buffer 15, and a storage unit (third memory, fourth memory) 16B. Be prepared.

The storage unit 16B stores a predefined weight quantization function ( second function) 160 and a convolution operation quantization function ( first function) 161.
The weight quantization function 160 is a function that realizes the quantization of weights. More specifically, in the weight quantization function 160, the quantization processing unit 14 performs rounding processing preset for the weight U, for example, integerization of the weight U including a decimal point, and reduces the number of data. It is a function that realizes the quantization process.

The convolution operation quantization function 161 is a function that realizes a convolution operation in which a quantization process is incorporated. More specifically, the convolution operation quantization function 161 sets a limit on the number of digits of the operation result when the convolution operation unit 12 performs the convolution operation, or performs a bit shift of a value such as an input signal in advance. Is a function that realizes with a convolution operation.

The quantization processing unit 14 calls the weight quantization function 160 from the storage unit 16B to execute the quantization of the weight U. The value of the weight U to be quantized is stored in the storage unit 16B in advance. The weight U'quantized by the quantization processing unit 14 using the weight quantization function 160 is temporarily stored in the quantization weight buffer 11A.

The convolution unit 12 calls the convolution operation quantization function 161 stored in the storage unit 16B to execute the quantization convolution operation. More specifically, the convolution calculation unit 12 reads the quantized weight U'from the quantized weight buffer 11A. Further, the convolution calculation unit 12 reads the input signal A from the input buffer 10. Then, the convolution calculation unit 12 performs a convolution operation in which the quantization process is incorporated by using the called convolution operation quantization function 161 based on the input signal A and the quantized weight U'. The calculation result using the function by the convolution calculation unit 12 is stored in the calculation result buffer 13.

[CNN processing method]
Next, the operation of the CNN processing apparatus 1B having the above-described configuration will be described with reference to FIGS. 9 and 10.

First, the quantization processing unit 14 calls the weight quantization function 160 stored in advance in the storage unit 16B to quantize the weight U (step S30). The quantized weight U'is temporarily stored in the quantized weight buffer 11A (step S31). For example, as shown in FIG. 10, the weight quantization function 160 corresponding to the quantized weight U'is called from the storage unit 16B by the quantization processing unit 14. In the example of FIG. 10, the function represented by "weight quantization function 1" is assigned to each element as the weight quantization function 160.

Next, the input buffer 10 temporarily stores the input signal A acquired from a server or the like installed outside the CNN processing device 1 (step S32).

Next, the convolution calculation unit 12 calls the convolution operation quantization function 161 from the storage unit 16B to perform a convolution operation in which the quantization process is incorporated (step S33). More specifically, the convolution calculation unit 12 reads the input signal A from the input buffer 10 and the quantized weight U'from the quantized weight buffer 11A. The convolution calculation unit 12 performs a convolution operation in which the quantization process is incorporated by using the convolution operation quantization function 161 based on the input signal A and the quantized weight U'. For example, the convolution calculation unit 12 performs a process of limiting the number of digits of the calculation result of the convolution calculation to an integer according to the called function. Further, for example, the quantized convolution calculation unit 12A performs a bit shift on the input signal A and the quantized weight U'in advance to perform processing on the number of digits of the data before performing the convolution calculation.

More specifically, as shown in the example of FIG. 9, the convolution calculation unit 12 performs a quantization process of truncating after the decimal point at the time of the convolution operation according to the function 1 of the convolution operation quantization function 161 (floor (floor). conv)). Further, the convolution calculation unit 12 multiplies the element of the quantized weight U'and the corresponding element of the input signal A at each location of the filter according to the convolution calculation quantization function 161 and obtains the sum thereof. .. The calculation result X of the quantized convolution operation by the quantized convolution calculation unit 12A is temporarily stored in the calculation result buffer 13 (step S34).

After that, the processor 102 of the CNN processing device 1B reads the operation result X of the convolution operation quantized from the operation result buffer 13 and applies an activation function such as ReLU (step S35). Specifically, when the calculation result X is a negative value, the processor 102 outputs 0 through the ReLU function, and the positive calculation result X outputs the value as it is.

Next, the processor 102 performs a well-known pooling process on the value output in step S35, and compresses the result of the convolution operation (step S36). The pooling process in step S35 may be performed as needed. Further, the processor 102 may perform normalization on the output of the activation function (ReLU) obtained in step S35 (see Non-Patent Document 1).

The operation result X of the pooled quantized convolution operation is temporarily stored in the output buffer 15, further read by the processor 102, and output to the outside (step S37). The output value is input to the fully coupled layer constituting the subsequent classifier (not shown) as the output of the feature extraction unit of the CNN, and the image data of the input signal A is discriminated.

As described above, according to the CNN processing apparatus 1B according to the third embodiment, the weight quantization and the quantum using the predefined weight quantization function 160 and the convolution operation quantization function 161. Performs a deconvolution operation. Therefore, in the operation of the convolution layer connected in multiple stages, all the convolution operations can be defined by exchanging the functions, the amount of the program can be suppressed, and the decrease in the processing speed of the CNN can be suppressed.

The weight quantization function 160 and the convolution operation quantization function 161 according to the embodiment described may be programs that can be incorporated into hardware. For example, FPGA source code, microcomputer firmware, etc. fall under this category, and in this case, it can be separately provided in the form of hardware IP or the like.

Further, the weight quantization function 160 and the convolution operation quantization function 161 configured as programs that can be incorporated into hardware may be stored in the memory in the same hardware, or may be stored in other network equipment or the like. You may. By exchanging the above programs according to the form of the neural network, it is possible to realize hardware that flexibly realizes a neural network having a desired processing function.

Although the embodiments of the neural network processing apparatus and the neural network processing method of the present invention have been described above, the present invention is not limited to the described embodiments and is within the scope of the invention described in the claims. It is possible to make various modifications that can be imagined by a person skilled in the art.

For example, in the described embodiment, CNN has been described as an example of the neural network, but the neural network adopted by the neural network processing device is not limited to CNN.

The various functional blocks, modules, and circuits described in connection with the embodiments disclosed herein include general-purpose processors, GPUs, digital signal processors (DSPs), application-specific integrated circuits (ASICs), and FPGAs. Alternatively, it may be performed using other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or any combination of the above designed to achieve the functions described above.

Although it is possible to use a microprocessor as a general-purpose processor, it is also possible to use a conventional processor, controller, microcontroller, or state device instead. The processor can also be realized, for example, as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors connected to a DSP core, or a combination of computing devices having such an arbitrary configuration. Is.

1 ... CNN processing device, 10 ... input buffer, 11 ... weight buffer, 12 ... convolution calculation unit, 13 ... calculation result buffer, 14 ... quantization processing unit, 15 ... output buffer, 16 ... storage unit, 101 ... bus, 102 ... Processor, 103 ... Main storage, 104 ... Communication interface, 105 ... Auxiliary storage, 106 ... Input / output device, NW ... Communication network, U ... Weight, U'... Quantified weight, A ... Input signal, A '… Quantized input signal.

Claims (16)

The first memory that stores the input signal given to the multi-layer neural network,
A second memory that stores the weights of the multi-layer neural network,
A processor that performs a convolution operation of the multi-layer neural network including a product-sum operation of the input signal and the weight stored in the first memory and the second memory, respectively.
A buffer for storing the operation result of the convolution operation and used for the next operation in the multi-layer neural network is provided.
The convolution operation integrates convolution and quantization so that only the quantized operation result is stored in the buffer, not the non-quantized operation result, which incorporates the quantization that reduces the bit accuracy of the data. It is a quantized convolution operation performed by
The weights used in the product-sum operation are either quantized to a bit precision of 1 bit in advance before being stored in the second memory, or are stored in the second memory and then stored in the second memory for 1 bit by the processor. Quantized to bit precision ,
Further, a third memory for storing the first function for realizing the quantized convolution operation is provided.
The processor reads the first function from the third memory, and uses the read first function to perform the quantization convolution operation.
A neural network processing device characterized by this. In the neural network processing apparatus according to claim 1,
The processor is a neural network processing apparatus characterized in that processing using an activation function is performed on the quantized calculation result. In the neural network processing apparatus according to claim 1 or 2.
The processor is a neural network processing device characterized by performing quantization that reduces bit accuracy with respect to a bias used in the quantization convolution operation. The neural network processing apparatus according to any one of claims 1 to 3.
The multi-layer neural network has a plurality of convolution layers in which the quantized convolution operation is sequentially performed by the processor.
The processor stores the quantized calculation result, which is the output of each of the convolution layers other than the final layer among the plurality of convolution layers and is input to the next convolution layer, in the auxiliary storage device.
A neural network processing device characterized by this . In the neural network processing apparatus according to claim 4,
Further, a fourth memory for storing the second function for quantizing the weight is provided.
The processor reads the second function from the fourth memory and quantizes the weight using the read second function .
A neural network processing device characterized by this. In the neural network processing apparatus according to claim 5,
The processor is a neural network processing apparatus characterized in that the second function and the first function are alternately read out to perform the quantization of the weight and the quantized convolution operation in each of the plurality of convolution layers. .. The neural network processing apparatus according to any one of claims 1 to 6.
The multi-layer neural network further includes an input buffer for storing the preprocessed image signal as the input signal.
The preprocessing applied to the image signal includes one of monochrome conversion, contrast adjustment, and luminance adjustment .
A neural network processing device characterized by this. The neural network processing apparatus according to any one of claims 1 to 7.
A neural network processing device characterized in that the quantization of the weight is performed on the FPGA. A communication device including the neural network processing device according to any one of claims 1 to 8.
Further equipped with an interface circuit for communicating with external electronic devices via a communication network,
Data used for the quantized convolution operation is communicated from the external electronic device via the communication network.
A communication device characterized by that. The communication device according to claim 9.
A communication device characterized by receiving prequantized weights from the external electronic device via the communication network. The communication device according to claim 9 or 10.
A communication device characterized by receiving the first function from the external electronic device via the communication network. The first step of storing the input signal given to the multi-layer neural network in the first memory,
The second step of storing the weight of the multi-layer neural network in the second memory,
A third step in which the processor performs a convolution operation of the multi-layer neural network including a product-sum operation of the input signal and the weight stored in the first memory and the second memory, respectively.
The buffer comprises a fourth step of storing the operation result of the convolution operation and the operation result used for the next operation in the multi-layer neural network.
The convolution operation integrates convolution and quantization so that only the quantized operation result is stored in the buffer, not the non-quantized operation result, which incorporates the quantization that reduces the bit accuracy of the data. It is a quantized convolution operation performed by
The weight used in the product-sum operation is either quantized to a bit precision of 1 bit in advance before being stored in the second memory, or 1 bit by the processor after being stored in the second memory. Quantified to the bit precision of
In the third step, the processor reads the function from a third memory that stores a function that realizes the quantized convolution operation, and performs the quantized convolution operation using the read function.
A neural network processing method characterized by this. A computer processor including a first memory for storing an input signal given to a multi-layer neural network and a second memory for storing the weights of the multi-layer neural network.
An arithmetic process for performing a convolution operation of the multi-layer neural network including a product-sum operation of the input signal and the weight stored in the first memory and the second memory, respectively.
The storage process of storing the operation result of the convolution operation and the operation result used for the next operation in the multi-layer neural network in the buffer is executed.
The convolution operation integrates convolution and quantization so that only the quantized operation result is stored in the buffer, not the non-quantized operation result, which incorporates the quantization that reduces the bit accuracy of the data. It is a quantized convolution operation performed by
The weights used in the product-sum operation are either quantized to a bit precision of 1 bit in advance before being stored in the second memory, or are stored in the second memory and then stored in the second memory for 1 bit by the processor. Quantized to bit precision,
The computer further comprises a third memory for storing a function that realizes the quantized convolution operation.
The arithmetic process includes a process of reading the function from the third memory and a process of performing the quantized convolution operation using the read function.
program. The program according to claim 13.
Have the processor execute a process using the activation function on the quantized operation result.
program. The program according to claim 13 or 14.
Have the processor execute a process of performing quantization that reduces bit accuracy with respect to the bias used in the quantization convolution operation.
program. The program according to any one of claims 13 to 15.
The multi-layer neural network has a plurality of convolution layers in which the quantized convolution operation is sequentially performed by the processor.
The processor executes a process of storing the quantized calculation result, which is the output of each of the convolution layers other than the final layer among the plurality of convolution layers and is input to the next convolution layer, in the auxiliary storage device. Let,
program.

Images