JP7279921B2 - Neural network circuit device, neural network processing method, and neural network execution program - Google Patents

Description

The present invention relates to a neural network circuit device, a neural network processing method, and a neural network execution program.

In recent years, robotics, ADAS (advanced driver assistance system), image recognition and automatic translation for drones, etc., have attracted attention as new methods such as convolutional neural networks (CNNs) (NNs where layers are not fully connected) and recursive neural networks. Neural networks (bi-directional propagation) are emerging.
A neural network has an "input layer," a "hidden layer," and an "output layer," and each layer has a plurality of "nodes" connected by "edges." A "hidden layer" can have a plurality of layers, and a network having a particularly deep hidden layer is called a deep neural network (DNN). Convolutional Neural Network (CNN) is a multi-layered “hidden layer” that alternately repeats not only a fully connected layer but also a convolution layer and a pooling layer for about 20 to 30 layers. It has a laminated structure of A CNN consists of operations such as convolution, pooling and full connections, and feature maps.

Normally, a neural network is virtually constructed on a computer, and arithmetic processing of each layer is executed by software. However, in CNN, a large number of layers (approximately 20 to 30 layers) for executing such complicated processing are superimposed, so the amount of computation is enormous. Therefore, it is often the case that arithmetic processing of a neural network is executed using an arithmetic device using a GPU (Graphics Processing Unit) instead of an ordinary computer using a general-purpose CPU or the like.

Patent Document 1 discloses a memory control means for allocating a partial area of a memory possessed by an arithmetic processing unit to each of a plurality of processing nodes, and a specification for sequentially designating a processing node to execute arithmetic processing from among the plurality of processing nodes. means and an execution control means for causing arithmetic processing by a plurality of processing nodes to be executed in a specified order. In the arithmetic processing device described in Patent Document 1, the memory control means circulates the data write destination within the partial area in units of memory areas having a size corresponding to the data amount of the calculation result, and the execution control means obtains the data. The partial area is used as a ring buffer by writing the operation result of each processing node assigned to the allocated partial area.

Patent Document 2 describes a plurality of convolution processing means for performing convolution processing on an accepted image, a plurality of rectification processing means for performing rectification processing on the processing results of the convolution processing means, a plurality of normalization processing means for performing normalization processing on the processing result of the rectification processing means; An image processing apparatus is described which includes a plurality of feature extracting means for extracting, and an identifying means for identifying the image based on the feature quantity extracted by the feature extracting means. In the image processing device described in Patent Document 2, the normalization processing means normalizes the difference between the processing results of the two rectification processing means by dividing the difference by a value based on the processing result as a divisor.

The existing CNN is composed of short-precision (multi-bit) product-sum operation circuits, and requires a large number of multiplication circuits. For this reason, there was a drawback that the area and power consumption were large. Therefore, a circuit has been proposed that configures a CNN using only binarized precision, that is, +1 and -1 (or 0 and 1) (see Non-Patent Documents 1 to 4, for example).

In the techniques of Non-Patent Documents 1 to 4, the recognition accuracy of the CNN is also lowered by lowering the accuracy to binary. To avoid this and maintain the accuracy of the binarized CNN, a batch normalization circuit is required.

Japanese Patent No. 5184824 Japanese Patent No. 5772442

M. Courbariaaux, I. Hubara, D. Soudry, R.E. Yaniv, Y. Bengio, "Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1," Computer Research Repository (CoRR), "2 Algorithm of Valued NN", [online], March 2016, [searched on October 5, 2018], <URL:http://arxiv.org/pdf/1602.02830v3.pdf> Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi, "XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks," Computer Vision and Pattern recognition, "Algorithms for Binarized Neural Networks," [online], March 2016, [Retrieved on October 5, 2018], <URL: https://arxiv.org/pdf/1603.05279v4> Hiroki Nakahara, Haruyoshi Yonekawa, Tsutomu Sasao, Hisashi Iwamoto and Masato Motomura, ”A Memory-Based Realization of a Binarized Deep Convolutional Neural Network,” Proc. of the 2016 International Conference on Field-Programmable Technology (FPT), Xi'an, China, Dec 2016 (To Appear). Eriko Nurvitadhi, David Sheffield, Jaewoong Sim, Asit Mishra, Ganesh Venkatesh, Debbie Marr,”Accelerating Binarized Neural Networks: Comparison of FPGA, CPU, GPU, and ASIC,” Proc. of the 2016 International Conference on Field-Programmable Technology (FPT) ), Xi'an, China, Dec 2016 (To Appear).

In CNN, in order to recognize an object and its position, it is necessary to perform convolution operations of observed images and weight matrices over multiple layers an enormous number of times. For this reason, there is a problem that the computer power, the calculation time, and the memory capacity for storing the weight matrix (parameters) are enormous. Moreover, since a large amount of sum-of-products operations and their parameters are stored, it is difficult to implement in embedded devices.

SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances, and a neural network circuit device, neural network processing method, and execution of a neural network capable of reducing convolution calculation time and memory without degrading recognition accuracy. The task is to provide a program.

ことを特徴とする。 In order to solve the above-described problems, a neural network circuit device according to the present invention is a neural network circuit device including at least an input layer, one or more intermediate layers, and an output layer, wherein the intermediate layer performs a convolution operation . and a noise convolution circuit for performing a noise convolution operation, the noise convolution circuit receiving an input value X to be convolved and noise, and receiving the input value An arithmetic circuit that adds the noise to X, and a multiplier circuit that receives the weight W and multiplies the noise convoluted value of the arithmetic circuit and the weight W ,
The noise is
Expected value: E(n _c )=E(Σ _Nc ε _i W _i ′)=0, and
Variance: E(n _c ² )=E(Σ _Nc ε _i W _i ') ² = 2σ ² δ'
When using a noise n _c such that
Output y′ of the convolution operation by the convolution operation circuit;
The output y' by the noise convolution operation by the noise convolution operation circuit is represented by the following formula and is statistically equivalent

It is characterized by

ことを特徴とする。
その他の手段については、発明を実施するための形態のなかで説明する。 Further, a neural network circuit device according to the present invention is a neural network circuit device including at least an input layer, one or more intermediate layers, and an output layer, wherein the intermediate layers are n layers (n is any natural number). and a convolution circuit that performs convolution operation in k (k is a natural number that satisfies nk) layer, and a noise convolution operation that performs noise convolution operation in (nk) layer a circuit, wherein the noise convolution arithmetic circuit receives an input value X to be convolved, an arithmetic circuit that receives noise and adds the noise to the input value X, a weight W, and the noise of the arithmetic circuit a multiplication circuit that multiplies the convolution operation value and the weight W ,
The noise is
Expected value: E(n _c )=E(Σ _Nc ε _i W _i ′)=0, and
Variance: E(n _c ² )=E(Σ _Nc ε _i W _i ') ² = 2σ ² δ'
When using a noise n _c such that
Output y′ of the convolution operation by the convolution operation circuit;
The output y' by the noise convolution operation by the noise convolution operation circuit is represented by the following formula and is statistically equivalent

It is characterized by
Other means are described in the detailed description.

According to the present invention, it is possible to provide a neural network circuit device, a neural network processing method, and a neural network execution program capable of shortening the convolution calculation time and reducing the memory without lowering the recognition accuracy.

4 is a diagram showing the configuration of a noise convolution circuit included in the neural network circuit according to the embodiment of the present invention; FIG. 3 is a block diagram of a noise convolution circuit included in the neural network circuit according to the embodiment of the present invention; FIG. 1 is a schematic configuration diagram of a CNN for explaining the basic principle of the present invention, (a) is a module configuration diagram of the CNN, (b) is a histogram statistically showing parameters (weights) after learning, (c) is A histogram statistically showing parameters (weights) after learning, and (d) is a histogram statistically showing parameters (weights) after learning. FIG. 2 is a diagram showing histograms of noise for explaining the basic principle of the present invention, where (a) is a diagram statistically showing noise, and (b) is a histogram obtained by statistically analyzing the noise in (a). 4 is a diagram showing the configuration of a noise convolution circuit included in the neural network circuit according to the embodiment of the present invention; FIG. 4 is a diagram showing the configuration of a noise convolution circuit included in the neural network circuit according to the embodiment of the present invention; FIG. 4 is a diagram showing the configuration of a noise convolution circuit included in the neural network circuit according to the embodiment of the present invention; FIG. 4 is a diagram showing the configuration of a noise convolution circuit included in the neural network circuit according to the embodiment of the present invention; FIG. 4 is a diagram showing the configuration of a noise convolution circuit included in the neural network circuit according to the embodiment of the present invention; FIG. 4 is a diagram showing the configuration of a noise convolution circuit included in the neural network circuit according to the embodiment of the present invention; FIG. 4 is a diagram showing the configuration of a noise convolution circuit included in the neural network circuit according to the embodiment of the present invention; FIG. 4 is a diagram showing the configuration of a noise convolution circuit included in the neural network circuit according to the embodiment of the present invention; FIG. 4 is a diagram showing the configuration of a noise convolution circuit included in the neural network circuit according to the embodiment of the present invention; FIG. FIG. 2 is a diagram showing a comparison between a noise convolution circuit included in a neural network circuit according to an embodiment of the present invention and a convolution circuit of a comparative example, and (a) is a schematic configuration diagram of the convolution circuit of the comparative example; , (b) is a schematic configuration diagram of a noise convolution arithmetic circuit; FIG. 4 is a diagram showing a comparison of the effects of a noise convolution circuit included in a neural network circuit according to an embodiment of the present invention and a convolution circuit of a comparative example; FIG. 10B is a diagram for explaining an input image and convolution when noise is not added, and FIG. 1B is a diagram for explaining an input image and 1×1 convolution when noise is added in the noise convolution arithmetic circuit; FIG. 2 is a diagram for explaining features of a noise convolution arithmetic circuit included in a neural network circuit according to an embodiment of the present invention, (a) is a histogram showing the value of each pixel of an input image and its frequency distribution, and (b) is a CNN 3 is a histogram showing the value of each pixel of the output of the first convolutional layer of , and its frequency distribution. Recognition of results of learning image recognition tasks CIFAR-10 and CIFAR-100 using noise convolution CNN and VGG16 model of existing convolution CNN when applied to input image of neural network circuit according to the embodiment of the present invention It is a figure explaining precision. 3 is a block diagram showing the configuration of a noise convolution CNN included in the neural network circuit according to the embodiment of the present invention; FIG. FIG. 4 is a table showing experimental results of the convolution CNN layer number k of the noise convolution CNN provided in the neural network circuit according to the embodiment of the present invention, the recognition accuracy (%), the ratio, and the weight of all layers (MB). is. FIG. 5 is a table showing learning time and inference time of the noise convolution CNN included in the neural network circuit according to the embodiment of the present invention; FIG. 10 is a diagram showing a table showing the result of comparing the noise convolution operation circuit included in the neural network circuit according to the embodiment of the present invention on an FPGA and comparing the convolution operation with the existing convolution CNN. It is a figure explaining the implementation example of the noise convolution CNN with which the neural network circuit based on embodiment of this invention is provided. It is a module block diagram of CNN. It is a figure explaining an example of the structure of a deep neural network (DNN). It is a figure which shows an example of a structure of the neural network circuit of a comparative example. FIG. 5 is a diagram showing the configuration of a convolution circuit of a comparative example; FIG. 5 is a diagram showing the configuration of a convolution circuit of a comparative example; FIG. 5 is a diagram showing the configuration of a convolution circuit of a comparative example; FIG. 5 is a diagram showing the configuration of a convolution circuit of a comparative example; FIG. 5 is a diagram showing the configuration of a convolution circuit of a comparative example;

Hereinafter, a deep neural network in a mode for carrying out the present invention (hereinafter referred to as "this embodiment") will be described with reference to the drawings.
(Background explanation)
[CNN]
FIG. 23 is a module configuration diagram of CNN. Take the architecture of deep CNN for image recognition as an example.
The CNN shown in FIG. 23 is composed of a laminate of an input layer INPUT, convolutional layers C1 to C5 for converting an input volume into an output volume, fully connected layers A1 to A2, and an output layer OUTPUT.
The input layer INPUT is, for example, raster-scanned image data, and inputs detection target image data (here, the front image of CAR) which is input data to the CNN calculation.
Each layer C1-C5, A1-A2 of the CNN has neurons three-dimensionally arranged in width, height and depth.

The neurons inside the convolutional layers C1-C5 and the fully connected layer A1 are connected only to the nodes of a small area called the receptive field of the previous layer.
The CNN shown in FIG. 23 includes seven layers with weights: convolutional layers C1-C5 and fully connected fully connected layers A1-A2. All connected layers A1-A2 include dropouts.
The neurons of the convolutional layers C1-C5 and the fully connected layer A1 are connected to the receptive fields of the previous layer, and the neurons of the fully connected layer A2 are connected to all neurons of the previous layer.

A batch normalization layer is placed after the convolutional layers C1-C5. Each batch normalization layer is followed by a pooling layer that performs respective pooling (eg, max pooling).

<Processing content of the convolutional layer>
In the CNN shown in FIG. 23, the hidden layers are composed of convolutional layers C1 to C5 and pooling layers. The convolutional layers C1-C5 filter to nodes that are nearby in the previous layers to generate feature maps.
The inputs to the convolutional layers C1 to C5 are in the format of N sheets (N channels) each having a lengthwise size of S×S pixels. In CNN, the number of channels in the first input layer (convolutional layer C1) is N=1 if the input image is grayscale, and N=3 (3 RGB channels) if the input image is color.
In the convolution layers C1-C5, calculations are performed to convolve filters (hereinafter referred to as "kernels") to the inputs. Specifically, the input size S×S pixels of each channel k (k=1 to N) are convolved with a two-dimensional filter of size L×L, and the result is applied to all channels k=1 to N. to add. The result of this calculation is in the form of a one-channel image uij.

<Processing content of the pooling layer>
The pooling layers exist in pairs with the convolutional layers C1-C5. The outputs of the convolutional layers C1-C5 are the inputs to the pooling layers.
In the CNN shown in FIG. 23, the pooling layer further reduces the feature maps output from the convolutional layers C1 to C5 to generate new feature maps.
The purpose of the pooling layer is to achieve invariance of the response to small feature changes by throwing away some information about where in the image the filter response was strong. In CNN, either average pooling or max pooling may be employed. In the max pooling, a method of taking the maximum value within a certain area in the image is used to absorb some deviation of the image and to obtain movement invariance with respect to the positional movement of the image.
The size of the local receptive field of the pooling layer is set independently of that (filter size) of the convolutional layers C1-C5.
Note that in the pooling layer, unlike the convolutional layers C1 to C5, there are no weights that change due to learning, and no activation function is applied.

<Rectified Linear Unit>
The CNN shown in FIG. 23 uses piecewise linear activation functions called Rectified Linear Units as nonlinear mapping functions for the convolutional layers C1-C5 and the fully connected layers A1-A2.

The CNN shown in FIG. 23 will be described in detail.
FIG. 23 shows an example of a CNN in which the first to fourth convolutional layers C1 to C4 have three features, and the fifth convolutional layer C5 has one feature. The image data 1001 is input data to the CNN calculation, and is raster-scanned image data, for example. A reference image region 1002 is a reference image region necessary for convolution calculation of a convolution filter (feature extraction filter) (described later).
The first convolutional layer C1 to the fifth convolutional layer C5 generate feature maps. A feature map is an image data surface showing a processing result obtained by performing an operation while scanning data of the previous layer with a predetermined feature extraction filter (cumulative sum of convolution operation and non-linear processing). Since the feature map is the detection result for raster-scanned image data, the detection result is also represented by a plane.

The first convolutional layer C1 filters a 56×56×3 input image (AGE image) with a stride of 2 pixels by 64 kernels of size 5×5×3. The stride is the distance between the receptive field centers of adjacent neurons in the kernel map. The stride is set to 1 pixel in all convolutional layers.
In the first convolutional layer C1, input image data is scanned, for example, by raster scanning at predetermined size intervals. A plurality of feature amounts included in the input image are extracted by performing weight matrix convolution and pooling processing on the scanned data. In the first convolutional layer C1, a relatively simple single feature quantity such as a linear feature quantity extending in the horizontal direction or a linear feature quantity extending in an oblique direction is extracted. Also, a plurality of feature maps corresponding to the plurality of features included in the input image are generated.

The input of the second convolutional layer C2 is the batch normalized and max pooled output of the first convolutional layer C1. The second convolutional layer C2 filters the input with 128 kernels of size 3x3x64.
In the first convolutional layer C2, the input data input from the first convolutional layer C1 is scanned for each predetermined size by raster scanning, for example. A plurality of feature amounts included in the input image are extracted by performing weight matrix convolution and pooling processing on the scanned data. In the second convolutional layer C1, a higher-dimensional composite feature quantity is extracted by integrating the plurality of feature quantities extracted in the first convolutional layer C1 while considering the spatial positional relationship and the like. . Also, a plurality of feature maps corresponding to the plurality of features included in the input image are generated.

The third convolutional layer C3 has 128 kernels of size 3×3×64 and is connected to the output of the second convolutional layer C2 (batch normalization and MAX pooling). The third convolutional layer C3 extracts a higher-dimensional composite feature quantity by integrating the plurality of feature quantities extracted in the second convolutional layer C2 while considering the spatial positional relationship. do. Also, a plurality of feature maps corresponding to the plurality of features included in the input image are generated.

The fourth convolutional layer C4 comprises 128 kernels of size 3x3x128.
The fifth convolutional layer C5 comprises 128 kernels of size 3x3x128.
In this manner, in the CNN shown in FIG. 23, various feature amounts included in the input image data D1 are extracted at high dimensions by repeating the processing by the convolution layers C1 to C5. Then, the results obtained by the processing of the convolution layers C1 to C5 are output to the fully connected layer A1 as intermediate operation result data.

In this way, in the CNN calculation, a feature map is generated by repeating product-sum calculation while scanning a plurality of filter kernels pixel by pixel, and non-linearly transforming the final product-sum result. A feature map is obtained by cumulatively adding the outputs of the convolution filters and finally performing nonlinear transformation processing.

In the CNN shown in FIG. 23, the convolutional layers C1-C5 extract local features of the image, and the pooling layers aggregate the local features. By reducing the image while maintaining the features of the input image, the amount of information in the image can be significantly compressed.

A fully connected layer A1 is placed after the pooling layer of the fifth convolutional layer C5.
Fully connected layers A1 to A2 connecting all nodes between adjacent layers each have 1024 neurons.
The fully connected layers A1-A2 combine a plurality of intermediate operation result data obtained from the fifth convolutional layer C5 and output final operation result data. Specifically, the fully-connected layers A1 and A2 combine a plurality of intermediate operation result data obtained from the fifth convolution layer C5, and further perform a sum-of-products operation on the combined result while varying the weight coefficients. As a result, the final calculation result data, that is, the image data obtained by recognizing the detection object included in the image data which is the input data, is output. At this time, a part with a large value of the calculation result of the sum-of-products calculation is recognized as part or all of the object to be detected.

The output layer OUTPUT is the final layer fully connected with the fully connected layer A2. The output layer OUTPUT includes the same number of final nodes as the preset number of classification classes. For the purpose of class classification of an input image, the same number of nodes as the number of classification classes are arranged in the output layer OUTPUT, and a "softmax function" is used as the activation function of the output layer OUTPUT.
As shown in FIG. 23, when the detection target image data (the front image of CAR) is input, the final processing result is the detection target objects "CAR", "TRUCK", "VAN", ..., included in the image data. BICYCLE” image recognition result is output.

[DNN]
FIG. 24 is a diagram illustrating an example of the structure of a deep neural network (DNN).
As shown in FIG. 24, a deep neural network (DNN) 1 has an input layer 11, an arbitrary number of hidden layers 12, and an output layer 13. consists of
The input layer 11 has a plurality of (eight here) input nodes (neurons). There are a plurality of hidden layers 12 (here, three layers (hidden layer 1, hidden layer 2, hidden layer 3)). In practice, the number n of hidden layers 12 reaches 20-100, for example. The output layer 13 has a number of output nodes (neurons) to be identified (here, 4). Note that the number of layers and the number of nodes (the number of neurons) are examples.
In the deep neural network 1, the nodes of the input layer 11 and the hidden layer 12 are all connected, and the nodes of the hidden layer 12 and the output layer 13 are all connected.

An arbitrary number of nodes (see circles in FIG. 24) exist in the input layer 11, the hidden layer 12, and the output layer 13. FIG. This node is a function that takes an input and outputs a value. The input layer 11 has a bias node for inputting a value independent of the input node. Configurations are built by stacking layers with multiple nodes. Propagation weights the received input, converts the received input to the next layer with an activation function, and outputs it. Activation functions include non-linear functions such as sigmoid function and tanh function (hyperbolic tangent function), and ReLU (Rectified Linear Unit function: normalized linear function). By increasing the number of nodes, it is possible to handle more variables and determine values/boundaries in consideration of many factors. By increasing the number of layers, a combination of straight boundaries and complex boundaries can be expressed. Learning calculates the error and adjusts the weights of each layer based on it. Learning is to solve an optimization problem that minimizes the error, and it is common to use the error backpropagation method to solve the optimization problem. As for the error, it is common to use the sum of squares error. A regularization term is added to the error to improve the generalization ability. The error backpropagation method propagates the error from the output layer 13 and adjusts the weight of each layer.

By developing the configuration of the deep neural network 1 in FIG. 24 two-dimensionally, a CNN suitable for image processing can be constructed. Further, by inputting feedback into the deep neural network 1, it is possible to construct an RNN (Recurrent Neural Network) in which signals propagate in both directions.

24, the deep neural network 1 is composed of a circuit (hereinafter referred to as a neural network circuit) 2 that realizes a multi-layered neural network.
The present technology targets the neural network circuit 2 . The application locations and the number of applications of the neural network circuit 2 are not limited. For example, when the number of layers n of the hidden layer 12 is 20 to 30, it may be applied to any position in these layers, and any node may be used as an input/output node. Further, the deep neural network 1 is not limited to any neural network. However, the neural network circuit 2 is out of the scope because the node output of the input layer 11 or the output layer 13 requires a multi-bit output instead of a binarized output. However, even if a multiplier circuit remains in the circuit forming the node of the output layer 13, there is no problem in terms of area.
In addition, it is premised on evaluating what has been learned with respect to input data. Therefore, the weight wi is already obtained as a learning result.

<Neural network circuit>
FIG. 25 is a diagram illustrating an example of a configuration of a neural network circuit of a comparative example; FIG. 25 shows the organization of one neuron.
The neural network circuit 20 of the comparative example can be applied to the neural network circuit 2 forming the deep neural network 1 of FIG. In the notation of each figure below, a multi-bit value is indicated by a thick solid line arrow and a bundle, and a binary value is indicated by a thin thick solid line arrow.

As shown in FIG. 25, the neural network circuit 20 includes input nodes for inputting input values (discrimination data) X1 to Xn (multi-bit), an input section 21 for inputting weights W1 to Wn (multi-bit), and a bias W0. a bias W0 input unit 22 for inputting (multi-bit), a plurality of multiplication circuits 23 for receiving input values X1 to Xn and weights W1 to Wn, and multiplying the input values X1 to Xn by the weights W1 to Wn, and each multiplication It comprises a summation circuit 24 that sums the value and the bias W0, and an activation function circuit 25 that converts the summed signal Y with an activation function fact(Y).

The input values (discrimination data) X1 to Xn (multi-bit) are the input values of the neurons, and the output values of the previous-layer neurons in the second and subsequent layers.
The multiplication circuit 23 outputs the result of multiplying the output value of each previous layer neuron by the weights W1 to Wn (multi-bit) obtained by learning. The weights W1 to Wn are determined for each object to be detected using a generally known learning algorithm such as back propagation.
The summation circuit 24 cumulatively adds the multiplication results from the multiplication circuit 23 .
The activation function circuit 25 non-linearly transforms the cumulative addition result Y from the summation circuit 24 using a sigmoid function, a tanh function, or the like, and outputs the transformation result as a detection result Z. FIG.
In the above configuration, the neural network circuit 20 receives the input values X1 to Xn (multi-bit), multiplies them by the weights W1 to Wn, and then outputs the signal Y, which is the sum including the bias W0, to the activation function circuit 25. By passing it through, it realizes processing that imitates human neurons.

[Convolution operation circuit of comparative example]
Next, a convolution operation circuit of a comparative example will be described.
26 to 30 are diagrams showing the configuration of the convolution circuit 30 of the comparative example. The convolutional operation circuit 30 of the comparative example constitutes the convolutional operation units of the convolutional layers C1 to C5 in FIG. 23, for example. The convolution circuit 30 of the comparative example is provided in the neural network circuit 20 shown in FIG. The same components as those in FIG. 25 are denoted by the same reference numerals.
The convolution circuit 30 of the comparative example performs convolution of an input image (image data) and a weighting function when detecting an object (dilated convolution).

As shown in FIGS. 26 to 30, the convolution operation circuit 30 of the comparative example includes a buffer memory 31 that develops input values X00 to X44, a weight memory 32 that stores weights W1 to Wn (multi-bit), 3 A plurality of multiplication circuits 23 for multiplying the value X of a ×3 kernel 33 by the weights W1 to Wn read from the weight memory 32, a summation circuit 24a for summing the convolution multiplied values, and a bias for the summation result. It comprises an adder circuit 24b that adds W0 and an activation function circuit 25 that converts the cumulative addition result Y from the adder circuit 24b by an activation function fact(Y).
The summing circuit 24a and the adding circuit 24b constitute the summing circuit 24 of FIG. The activation function circuit 25 is a non-linear function such as a sigmoid function or a tanh function, ReLU.

As shown in FIGS. 26 to 30, the convolution circuit 30 of the comparative example inputs an image (each element of the matrix corresponds to one pixel of the image) to the Input Feature Map developed in the buffer memory 31. FIG. For this input image, the values of the kernel (in this example a 3×3 kernel 33 with K=3) are multiplied element by element by the values of the matrix, and the values are summed. By performing this operation on each element while sliding the kernel, the entire convolution operation is performed. A specific description will be given below.

First, as shown in FIG. 26, the convolution circuit 30 of the comparative example converts the 3×3 kernel 33 (see hatching in FIG. , X22. The multiplication circuit 23 multiplies the values (X00, X01, X02, X10, X11, X12, X20, X21, X22) of the 3×3 kernel 33 by the weights W1 to Wn read out from the weight memory 32 element by element. perform calculations. The summation circuit 24a sums the values (sums the convolution multiplied values), and the summation circuit 24b adds the bias W0 to the summation result. The activation function circuit 25 converts the cumulative addition result Y from the addition circuit 24b with the activation function fact(Y).

Next, as shown in FIG. 27, the convolution circuit 30 of the comparative example shifts the 3×3 kernel 33 by one from left to right. The multiplication circuit 23 multiplies the values (X01, X02, X03, X11, X12, X13, X21, X22, X23) of the 3×3 kernel 33 by the weights W1 to Wn read out from the weight memory 32 element by element. A summation circuit 24a sums those values. The adder circuit 24b adds the bias W0 to the total sum result, and the activation function circuit 25 converts the cumulative addition result Y from the adder circuit 24b with the activation function fact(Y).

Next, as shown in FIG. 28, the convolution circuit 30 of the comparative example shifts the 3×3 kernel 33 by one from left to right. In this case, the 3x3 kernel 33 is slid to the upper right end. A similar operation is performed on this input image. That is, the values of the 3×3 kernel 33 (X02, X03, X04, X12, X13, X14, X22, X23, X24) are multiplied element by element by the matrix values, and the values are totaled. The adder circuit 24b adds the bias W0 to the total sum result, and the activation function circuit 25 converts the cumulative addition result Y from the adder circuit 24b with the activation function fact(Y).

Next, as shown in FIG. 29, the convolution operation circuit 30 of the comparative example moves the 3×3 kernel 33 one position below the left end. For this input image, the values of the 3×3 kernel 33 (X11, X11, X12, X20, X21, X22, X30, X31, X32) are multiplied element by element by the matrix values, and the values are summed. .

Similarly, the 3×3 kernel 33 is shifted to the lower right, and the convolution operation circuit 30 of the comparative example performs similar operations on the input image. As shown in FIG. 30, the lower left and lower right are the ends for sliding the 3×3 kernel 33 . The convolution operation circuit 30 of the comparative example multiplies the values (X22, X23, X24, X32, X33, X34, X42, X43, X44) of the 3×3 kernel 33 and the values of the matrix element by element, and obtains the values sum up The adder circuit 24b adds the bias W0 to the total sum result, and the activation function circuit 25 converts the cumulative addition result Y from the adder circuit 24b with the activation function fact(Y).
After moving to the lower right, return the kernel to X00 (see FIG. 26) and perform the calculation again.

In this manner, the convolution operation circuit 30 of the comparative example multiplies the input image in the buffer memory 31 by the value of the 3×3 kernel 33 and the value of the matrix element by element, and totals the values. By performing this operation on each element while sliding the 3×3 kernel 33 from the upper left to the lower right, the entire convolution operation is performed.
Note that zero padding may or may not be performed.

In the CNN, the convolution operation of the observed image and the weight matrix as described above is executed about 10 million to 1 billion times. Therefore, the computer power, calculation time, and memory capacity for storing the weight matrix become enormous. Moreover, since a large amount of sum-of-products operations and their parameters are stored, it is difficult to implement in embedded devices.

(embodiment)
FIG. 1 is a diagram showing the configuration of a noise convolution arithmetic circuit 100 included in a neural network circuit according to an embodiment of the present invention. "Noise convolution" is what the inventor named based on his findings. The noise convolution circuit 100 is replaced with the convolution circuit 30 of the comparative example shown in FIGS.
As shown in FIG. 1, the noise convolution arithmetic circuit 100 receives a noise generation circuit 110 that generates noise _nc , and an input value X for performing a 1×1 convolution. An adder circuit 120 (arithmetic circuit, arithmetic means) that adds the noise _nc generated by the noise generator circuit 110 receives the weight W and multiplies the weight W by the noise convoluted value of the adder circuit 120 with added noise. and a multiplication circuit 130 (multiplication means).

The noise generation circuit 110 is composed of a random number generator that generates random numbers.
The noise n _c is the value output by the random number generator.
The random number generator can be any method (for example, XOR-Shift, LFSR (linear feedback shift register: linear feedback shift register), linear congruential generators, etc. if there is controllability (reproducibility) of random numbers. ) to generate random numbers.

In this embodiment, pseudo-random numbers are used as random numbers. Pseudo-random numbers can be easily created using a learning tool. However, it is also possible to use true random numbers as the random numbers. It is expected that learning a 1×1 convolution will be difficult if true random numbers are used. In other words, the distribution of weights with sufficient accuracy from the observation of the weight parameters after learning substantially matches the distribution by random number generation. At this time, non-matching weights are corrected by 1×1 convolution. Learning 1×1 convolution becomes difficult if true random numbers are used as random numbers.

The calculation for adding the noise _nc is addition by the adder circuit 120 . The reason why noise is superimposed by addition will be described. It is conceivable to add noise by multiplication, but multiplication makes learning impossible. Deep learning learning methods generally use error backpropagation. In the error backpropagation method, the differential value of the miscalculation is used for updating, but in the aspect of multiplying the noise, the differential value remains and the noise is updated. As a result, the assumption of the noise distribution collapses, and the learning does not progress no matter how long it takes. Therefore, the noise convolution arithmetic circuit 100 adds noise by addition. It should be noted that the inventor of the present invention was the first to find out that the operation of adding noise is performed by addition.

FIG. 2 is a block diagram of the noise convolution arithmetic circuit 100 included in the neural network circuit 50 according to the embodiment of the invention.
As shown in FIG. 2, the neural network circuit 50 includes a noise convolution computation circuit 100 that executes noise convolution computation, a CPU 60 that controls the entire device, an off-chip memory 70 that stores recognition data, and various parts. and a system bus 80 for connecting.

The noise convolution arithmetic circuit 100 includes a buffer memory 101 that develops input values X00 to X44, a noise generation circuit 110 that generates noise _nc , a value X of a 1×1 kernel 111, and a noise generation circuit 110 a weight memory 131 for storing the weight _W ; and the value X of the 1×1 kernel 111 added with the noise _nc is multiplied by the weight W read from the weight memory 131. A multiplication circuit 130, an addition circuit 140 that adds a bias W0 to the noise-convolved multiplication value, and an activation function circuit 150 that transforms the cumulative addition result Y from the addition circuit 140 with an activation function fact(Y). .
The activation function circuit 150 is a non-linear function such as a sigmoid function or a tanh function, ReLU.

The noise convolution arithmetic circuit 100 is implemented by an existing FPGA (field-programmable gate array) or the like by adding a CPU 60, an off-chip memory 70, and a system bus 80 to the outside.

The neural network circuit 50 stores recognition data in the off-chip memory 70 . The CPU 60 issues a DMA transfer command to a DMA (Direct Memory Access) controller (not shown), and sends recognition data from the off-chip memory 70 to the buffer memory 101 by DMA transfer. The CPU 60 activates the noise convolution arithmetic circuit 100, and the noise convolution arithmetic circuit 100 performs noise convolution. When the convolution processing is completed, the recognition data is transferred from the buffer memory 101 to the off-chip memory 70, and post-processing is performed by the CPU 60 (for example, object detection processing in general object recognition, region estimation processing in semantic region segmentation, and pose estimation processing). Then, perform skeleton estimation).

The effects of the noise convolution arithmetic circuit 100 configured as described above will be described below.
(Explanation of principle of the present invention)
First, the basic principle of the present invention will be explained.
The focal point of the present invention is the discovery that post-learning parameters (weights) of neural networks can have the same statistical properties as noise.

FIG. 3 is a schematic configuration diagram of a CNN for explaining the basic principle of the present invention, FIG. 3(a) is a module configuration diagram of the CNN, and FIGS. ) statistically. Blocks in FIG. 3(a) indicate convolutional layers (including pooling layers), and numerical values attached to the convolutional layers indicate sizes of the convolutional layers. Also, k is the kernel size, and s is the slide size for sliding the kernel.
Statistically analyzing the parameters (weights) after learning in the first convolutional layer C1 as indicated by symbol a in FIG. 3(a) results in a histogram shown in FIG. 3(b).
Statistical analysis of the parameters (weights) after learning in the second convolutional layer C2, as indicated by symbol b in FIG. 3(a), results in a histogram shown in FIG. 3(c).
Statistical analysis of the parameters (weights) after learning in the fifth convolutional layer C5, as indicated by symbol c in FIG. 3(a), results in a histogram shown in FIG. 3(d).

4A and 4B are diagrams showing histograms of noise for explaining the basic principle of the present invention, FIG. 4A is a diagram showing the noise statistically, and FIG. Histograms are systematically analyzed. The vertical axis of FIG. 4A is the amplitude (magnitude) of the generated noise, and the horizontal axis of FIG.
As shown in FIG. 4(a), noise is generated with a random amplitude (0.5 to -0.5) around no noise (noise 0).
A histogram shown in FIG. 4B is obtained by accumulating the noise shown in FIG. 4A in order of occurrence frequency over time.

The inventors have found that the histogram of noise shown in FIG. 4(b) approximately matches the histogram of weight parameters after training of the neural network (see FIGS. 3(b)-(d)). . In other words, we discovered that the parameters (weights) after training of the neural network have the same statistical properties as noise (except for the first layer of the neural network among the parameters (weights) after training).
Based on the above findings, the following findings were obtained regarding the conditions under which the parameters (weights) after training of the neural network have the same statistical properties as noise.

[Derivation of weight v of noise convolution layer]
First, the derivation of the weight v of the noise convolution layer will be described.
The weight v of the noise convolution layer is the weight itself of the existing convolution without noise. In FIG. 1, it means W (weight) in FIG. Equation (1) below indicates that once the sequence of noise is known, the weight can be derived from it. However, since it is practically difficult to intentionally generate random numbers, weights suitable for pseudo-random numbers (noise) that can be generated by hardware are learned and obtained as described in this embodiment.
The operation of the noise convolution layer can be represented by the following equation (1).

The weight v when the number N of noise filters and the number p of input channels are equal can be expressed by the following equation (2), and the weight v when the number N of noise filters is smaller than the number p of input channels is It can be expressed by the formula (3).

In other words, given the input x, the noise filter N, and the output y, the weight v can be derived.

[Equivalence between noise convolution layer and existing convolution layer]
Next, the equivalence between the noise convolutional layer and the existing convolutional layer will be described.
<Assumption>
・Statistical analysis is possible.
・The input data is
E(x _i )=0, E(x _i ² )=σ ²
however,
x _i : each pixel in the input tensor to be convolved E(x _i ): expected value σ ² : when the variance is satisfied

<Convolution operation output>
The output y from the convolution operation is given by Equation (4).

<noise n _c >
Expected value: E(n _c )=E(Σ _Nc ε _i W _i ′)=0, and
Variance: E(n _c ² )=E(Σ _Nc ε _i W _i ') ² = 2σ ² δ'
A noise n _c is used.

When the above noise (noise) n _c is used, the output y' by the noise convolution operation and the output y' by the existing convolution operation are statistically equivalent as shown in Equation (5) (details will be described later. ).
In equation (5), the output y' of the convolution operation by the convolution operation circuit 30 of the comparative example and the output y' of the noise convolution operation by the noise convolution operation circuit 100 are statistically equivalent. is shown in a frame.

<Details of derivation of statistically equivalent formulas for noise convolutional layers and existing convolutional layers>
Next, derivation of intermediate formulas showing that the noise convolution layer and the existing convolution layer are statistically equivalent will be described.
We consider the relationship between the noise convolutional layer and the existing convolutional layer from the viewpoint of probability distribution. The convolution layer uses the center pixel x _c , neighboring pixels x _i , the correlation function, the difference ε=x _i −x _c between the central pixel ρ and the neighboring pixels, and the set Nc of neighboring pixels of the input to be convolved. Here, as preconditions, E(x _i )=0 and E(x _i ² )=σ ² are assumed.
First, E(ε _i ), E(ε _i ² ), and E(ε _i ε _j ) are obtained according to the following equations (6), (7), and (8) for future use in calculations.

Here, the convolution operation shown in Equation (4) can be transformed into Equation (9).

Here, regarding Σ _{iεNc ε iwi} ', E(Σ _{iεNc ε iwi} '), E(Σ _{iεNc ε iwi '} ) ² according to the following equations (10) and (11): demand.

From the above, it is considered that the noise convolutional layer is equivalent to the convolutional layer (existing convolutional layer) when the noise n _c given by the following equation (12) is used.

E(n _c )=E(Σ _i∈Nc ε _i w _i ′)=0
E(n _c ² )=E(Σ _i∈Nc ε _i w _i ′) ² =2σ ² δ′ (12)

As can be seen by comparing Eq. A feature of the present invention is that a large amount of memory accesses and sum-of-products operations are replaced with signals generated by a noise generating circuit that can be realized by a lightweight circuit. It should be noted that the mean and the variance of the calculation results are the same, and it does not mean that the calculation results are exactly the same.

[Operation of noise convolution arithmetic circuit 100]
Next, the operation of noise convolution arithmetic circuit 100 shown in FIGS. 1 and 2 will be described.
5 to 13 are diagrams showing the configuration of the noise convolution arithmetic circuit 100. FIG. The same components as those in FIGS. 1 and 2 are denoted by the same reference numerals. The operation of the noise convolution arithmetic circuit 100 will be described with reference to FIGS. 5 to 13. FIG.
The operation of the noise convolution circuit 100 shown in FIGS. 5-13 corresponds to the operation of the convolution circuit 30 of the comparative example shown in FIGS. 26-30.

First, as shown in FIG. 5, the noise convolution arithmetic circuit 100 moves the 1×1 kernel 111 (see hatching in FIG. 5) to X00 on the upper left. Adder circuit 120 adds noise _nc from noise generator circuit 110 to value X00 of 1×1 kernel 111, as indicated by symbol d in FIG. The multiplication circuit 130 performs a convolution operation of multiplying the value X00 of the 1×1 kernel 111 to which noise _nc is added by the weight W read from the weight memory 131 . Adder circuit 140 adds a bias W0 to the noise-convolved product. The activation function circuit 150 converts the cumulative addition result Y from the addition circuit 140 with the activation function fact(Y).

Next, as shown in FIG. 6, the noise convolution arithmetic circuit 100 shifts the 1×1 kernel 111 from left to right by one to X01. Similar noise convolution calculations are performed thereafter.

Similarly, as shown in FIGS. 7 to 9, the 1×1 kernel 111 is shifted to the right end, moved to X04, and similar noise convolution calculations are performed.
Further, as shown in FIG. 10, the 1×1 kernel 111 is slid to the lower left and right end and moved to X10. A similar noise convolution operation is performed. Further, as shown in FIG. 11, the 1×1 kernel 111 is slid to the right and moved to X11. A similar noise convolution operation is performed.

As shown in FIG. 12, the 1×1 kernel 111 is slid, and the same noise convolution operation is performed at the lower right X44.
After moving the 1×1 kernel 111 to the lower right, as shown in FIG. 13, the 1×1 kernel 111 is returned to X00 and calculation is performed again. FIG. 13 shows the position of the 1×1 kernel 111 after recalculation, returning to the position of the 1×1 kernel 111 in FIG.

Thus, the noise convolution operation circuit 100 performs a 1×1 convolution operation with noise added instead of 3×3 weights. Since the noise convolution arithmetic circuit 100 can substitute most convolutions for noise, 1×1 convolution is sufficient. In this case also, the convolution operation is performed from the upper left to the lower right.

[Comparison between the noise convolution circuit 100 and the convolution circuit 30 of the comparative example]
14A and 14B are diagrams showing a comparison between the noise convolution circuit 100 and the convolution circuit 30 of the comparative example. FIG. 14A is a schematic configuration diagram of the convolution circuit 30 of the comparative example, and FIG. 3B is a schematic configuration diagram of the noise convolution arithmetic circuit 100. FIG.
As shown in FIG. 14A, the convolution circuit 30 of the comparative example multiplies the value X (input X) of the 3×3 kernel 33 by the parameter (weight) W read from the weight memory 32. and a sum calculation circuit (Matrix MAC) 35 .
The convolution circuit 30 of the comparative example performs convolution (usually 3×3 convolution) of all parameters (weights).
As indicated by symbol e in FIG. 14(a), in CNN, the convolution operation is performed about 10 million to 1 billion times. Since the amount of computation and memory increases explosively, there is an urgent need to reduce the weight.

As shown in FIG. 14(b), the noise convolution arithmetic circuit 100 includes a noise generation circuit 110 that generates noise _nc , a value X of a 1×1 kernel 111, and the noise from the noise generation circuit 110 An addition circuit 120 for adding (noise) _nc , and a multiplication circuit 130 for multiplying the value X of the 1×1 kernel 111 to which the noise (noise) _nc is added by a parameter (weight) W are provided.
The noise convolution operation circuit 100 performs a 1×1 convolution operation. That is, the convolution operation circuit 30 of the comparative example performs 3×3 convolution of all parameters (weights), whereas the noise convolution operation circuit 100 performs only 1×1 convolution.

As indicated by symbol f in FIG. 14(b), when noise is added, it becomes equivalent to 1×1 convolution (see equation (5)).

In this manner, the noise convolution arithmetic circuit 100 performs 1×1 convolution by adding noise instead of 3×3 weights. As a result, the weight is reduced to 3×3/1 at maximum, and the computational complexity is also reduced to 3×3/1. As a result, the amount of calculation (approximately 90% of the amount of calculation) and the amount of memory can be reduced (details will be described later). In addition, it was confirmed that there was no decrease in recognition accuracy despite the reduction in the amount of calculation and memory (details will be described later).

[Comparison of effects between the noise convolution circuit 100 and the convolution circuit 30 of the comparative example]
FIG. 15 is a diagram showing a comparison of effects of the noise convolution arithmetic circuit 100 and the convolution arithmetic circuit 30 of the comparative example. FIG. 15(a) is a diagram for explaining an input image and convolution when noise is not added by the convolution operation circuit 30 of the comparative example, and FIG. 15(b) is a diagram for adding noise by the noise convolution operation circuit 100. FIG. 10 is a diagram for explaining an input image and 1×1 convolution in a case;
As indicated by symbol g in FIG. 15A, the convolution circuit 30 of the comparative example convolves weights close to noise using a large kernel such as 3×3.

As shown in FIG. 15B, the noise convolution operation circuit 100 directly adds noise from the noise generation circuit 110 (not shown) to the value X of the 1×1 kernel by the adder circuit 120 . As indicated by symbol h in FIG. 15(b), noise is added to the input image. The noise convolution arithmetic circuit 100 applies noise directly to the input image and corrects it by 1×1 convolution.

A condition where noise does not need to be added is when combining with existing convolution using a large kernel such as 3×3 instead of noise.

[Features of noise convolution arithmetic circuit 100]
Features of the noise convolution arithmetic circuit 100 will be described.
The precondition that the noise convolutional layer and the convolutional layer of the comparative example (existing convolutional layer) are equivalent (statistically equivalent) is, as described above, that the input data is
E(x _i )=0, E(x _i ² )=σ ²
however,
x _i : each pixel in the input tensor to be convolved E(x _i ): expected value σ ² : when the variance is satisfied.

input data is
E(x _i )=0, E(x _i ² )=σ ²
To satisfy , we ensured that the convolutional layer was after convolution was performed at least once in the CNN.

16A and 16B are diagrams for explaining the features of the noise convolution arithmetic circuit 100. FIG. 16A is a histogram showing the value of each pixel of the input image and its frequency distribution, and FIG. 10 is a histogram showing the value of each pixel in the output of the eye convolution layer and its frequency distribution;
As shown in FIG. 16A, in the input image, the distribution of the values of the pixels of the image is biased. Therefore, the preconditions for equivalent convolutional layers: E(x _i )=0, E(x _i ² )=σ ² are not satisfied, and noise convolution cannot be applied to the input image.

As shown in FIG. 16(b), the output of the first convolutional layer of the CNN is a "noise histogram". Therefore, since the above preconditions: E(x _i )=0, E(x _i ² )=σ ² are satisfied, noise convolution can be applied.
That is, one or more layers of the CNN require the convolutional layer (existing convolutional layer) of the comparative example.

[Comparison of noise convolution CNN and existing convolution CNN when applied to input image]
CNN when the noise convolution circuit 100 is applied to the input image (hereinafter referred to as noise convolution CNN when applied to the input image) and CNN using the convolution circuit 30 of the comparative example (hereinafter, existing convolution CNN) will be described.

FIG. 17 is a diagram for explaining the recognition accuracy of the result of learning the image recognition tasks CIFAR-10 and CIFAR-100 using the VGG16 model of the noise convolution CNN and the existing convolution CNN when applied to the input image. .
In order to confirm the effects of noise convolution CNN and existing convolution CNN when applied to input images, we implemented a VGG16 (16 hidden layers) benchmark NN and confirmed whether learning was successful. VGG16 is a popular and reproducible benchmark.

As shown in FIG. 17, the noise convolutional CNN when applied to the input image exhibits recognition accuracy significantly inferior to the existing convolutional CNN. That is, the noise convolutional CNN is significantly inferior to the existing convolutional CNN when applied to an input image that does not satisfy the above preconditions: E(x _i )=0, E(x _i ² )= ^σ2 . It was verified that the recognition accuracy was improved.

[Configuration of noise convolution CNN]
Based on the above verification results, when constructing a noise convolutional CNN, one or more layers of the CNN must be provided with the convolutional layer of the comparative example (existing convolutional layer).

FIG. 18 is a block diagram showing the configuration of the noise convolution CNN. Take the case of 13 convolution layers as an example. It is assumed that the number of fully connected layers combined with the convolutional layer is three. k in FIG. 18 is the number of convolutional layers (k in FIG. 18 is not the kernel).
A noise convolutional CNN consists of an existing convolutional layer and a noise convolutional layer. The existing convolutional layer is, for example, a 3x3 convolutional layer. A noise convolutional layer is a 1×1 convolutional layer with noise on it.
As shown in FIG. 18, the noise convolutional CNN is composed of k layers of 3×3 convolution layers, 13-k noise convolution layers, and three fully connected layers.
The value of k is chosen considering the trade-off between recognition accuracy and number of weights. Experimental results when the value of k is changed will be described below.

[Number of noise convolution CNN layers and recognition rate]
FIG. 19 is a table showing experimental results of the number of convolutional CNN layers k, recognition accuracy (%), ratio, and weight of all layers (MB). The weight of all layers (MB) corresponds to the amount of memory, and the larger the parameter (weight of all layers), the larger the amount of memory used.
FIG. 19 shows only noise convolution CNN (see FIG. 19 surrounded by solid line), “CNN 1 layer + noise convolution CNN 12 layer (see FIG. 19 surrounded by dashed line)”, “CNN k layer + noise convolution CNN (13-k) layer (see double-dot chain line in FIG. 19)”, “existing convolutional CNN only (see dashed line in FIG. 19)”, recognition accuracy (%), ratio, and weight of all layers (MB).

As shown in FIG. 19, the recognition accuracy (%) is 63.8 to 71.5% for "CNNk layer + noise convolutional CNN (13-k) layer" and 70% for "existing convolutional CNN only". It can be seen that 1% is high. On the other hand, the weights (MB) of all layers are 8.8 for "noise convolution CNN only", 8.8 for "CNN 1 layer + noise convolution CNN 12 layers", and "CNN k layer + noise convolution CNN (13-k ) layer” is small when the number of convolutional CNN layers k=2 to 7 is 15.0 or less. Considering both the recognition accuracy (%) and the weight of all layers (MB), the recognition accuracy is (%) is improved and all layer weights (MB) are small.

In particular, when the number of convolutional CNN layers k = 6 in the “CNNk layer + noise convolutional CNN (13-k) layer”, the recognition accuracy (%) is as high as 70.5%, and the weight of all layers (MB ) is as small as 12.9. Comparing the case of “CNN k layer + noise convolution CNN (13-k) layer” with the number of convolutional CNN layers k = 6 and “existing convolutional CNN only”, the recognition accuracy (%) is almost the same (70 .5% and 70.1%), while the total layer weight (MB) is significantly reduced (12.9 and 61.1). That is, when k=6, it was confirmed that the recognition accuracy was improved and the parameters (weights of all layers) were reduced by about 80%.

[Learning time and inference time of noise convolution CNN]
FIG. 20 is a table showing the learning time and inference time of the noise convolution CNN. For comparison, the learning time (s) and inference time ( s). A GPU was used, batchsize: 10, number of learning data: 50,000, and number of inference data: 10,000.

As shown in FIG. 20, in the configuration of the noise convolution CNN, if the configuration of “CNN k layer + noise convolution CNN (13-k) layer” (see FIG. 19) is adopted, the “noise convolution CNN” only Both the learning time (s) and the inference time (s) for each epoch were shortened compared to the case of the "existing convolutional CNN" alone. In particular, when comparing only “CNN k layer + noise convolution CNN (13-k) layer” and “existing convolution CNN”, the learning time (s) decreased from 70.5 to 35.5, and the inference time (s) decreased from 70.5 to 35.5. was shortened by about half from 4.50 to 2.81. Thus, it was verified that the learning time and the inference time were reduced by setting the configuration of the noise convolution CNN to "CNN k layer + noise convolution CNN (13-k) layer".

[Results of implementation of noise convolution CNN]
FIG. 21 shows the result of implementing the noise convolution operation circuit 100 of this embodiment on an FPGA (ZCU102 board) and comparing the convolution operation (one layer, 56×56 size 64 channels) with the existing convolution CNN. It is a figure shown by making into.
Comparison of FPGA implementation is the present embodiment (“CNN k layer + noise convolution CNN (13-k) layer”) and the existing convolution CNN of the comparative example. The amount of hardware in the FPGA implementation is expressed in flip-flop (FF) numbers, LUT numbers, 18Kb BRAM numbers, and digital signal processor (DSP) 48E numbers.

The terms in the table of FIG. 21 are as follows.
The number of LUTs is the LUT (Look-Up Table) consumption of the FPGA and means the area.
The number of FFs is the number of logic gates and means the area.
The BRAM 18Kb number is the consumption of the internal memory block of the FPGA and means the area.
The number of DSP Blocks is the amount consumed by the internal sum-of-products operation block of the FPGA, and means the area.
Latency [Cycles] is data transfer wait time/transfer speed when an external memory is attached.
The operating frequency [MHz] is the operating processing time (processing speed) of the arithmetic unit.
Of particular note in this table are the following:

<memory amount>
The noise convolution CNN of this embodiment was able to reduce the amount of memory (BRAM18K) from 1737 to 193, about 1/9 compared to the existing convolution CNN shown in the table.

<Area of calculation part>
The area of the arithmetic unit is represented by the number of LUTs, FFs, and DSP48Es in the table. The number of LUTs is reduced from 26075 to 7784, the number of FFs from 9446 to 6467, and the number of DSP48Es from 53 to 43. Reducing the number of LUTs, FFs, and DSP48Es contributes to a reduction in the area of the arithmetic unit and achieves speedup. That is, the area reduction of the arithmetic unit is directly linked to the chip area reduction, and since the external memory becomes unnecessary, the latency [Cycles] is reduced from 18897927 of the existing convolution CNN to 2853127 of the noise convolution CNN, which is about 1/6. 0.6, and the operating frequency [MHz] can be increased from 77.6 to 118.7 by about 1.5 times, and high speed can be achieved. In addition to the reduction in the amount of memory described above, the noise convolution CNN eliminates the need for an external memory, and thus has the effect of simplifying the memory controller. Since the chip area is proportional to the price, it can be expected that the price will be reduced by about two digits.

<performance equivalent>
The noise convolution CNN of the present embodiment adopts a "CNN k layer + noise convolution CNN (13-k) layer" configuration, as shown in the table of the number of noise convolution CNN layers and the recognition rate in FIG. , it can achieve the same recognition rate as the existing convolutional CNN.

According to this embodiment, compared to the existing convolutional CNN, the weights are reduced to 3×3/1 at maximum, and the computational complexity is also reduced to 3×3/1, as shown in FIG. It was found that a CNN with almost the same recognition accuracy can be constructed. It is expected to be put to practical use as an edge-embedded device hardware method for ADAS (Advanced Driver Assistance System) camera image recognition using deep learning. ADAS, in particular, requires high reliability and low heat generation when mounted on a vehicle. Since the noise convolution arithmetic circuit 100 according to this embodiment does not require an external memory, it does not require a cooling fan or cooling fins for cooling the memory. It is suitable to be mounted on an ADAS camera.

[Example of implementation]
FIG. 22 is a diagram illustrating an implementation example of the noise convolution CNN according to the embodiment of the present invention.
<STEP1>
First, a computer 201 having a GPU (Graphics Processing Unit) uses Chainer (registered trademark), which is framework software for existing deep neural networks, to process a given data set (ImageNet this time, data for image recognition tasks). learned above. Learning is performed on the GPU. The computer 201 has a CPU (Central Processing Unit) such as an ARM processor, memory, storage means (storage section) such as a hard disk, and an I/O port including a network interface. In this computer, the CPU 201 executes a program (execution program for noise convolution calculation) read into the memory to operate a control section (control means) composed of each processing section described later.

<STEP2>
Next, an automatic generation tool was used to automatically generate a C++ code equivalent to the noise convolution arithmetic circuit 100 of this embodiment, and a C++ code 202 was obtained.

<STEP3>
Next, an FPGA vendor's high-level synthesis tool (Xilinx SDSoC) (registered trademark) was used to generate HDL (hardware description language) 203 for FPGA (field-programmable gate array) synthesis. For example, in a high-level synthesis tool (Xilinx SDSoC), a logic circuit to be implemented is described using a hardware description language (Verilog HDL/VHDL) and synthesized into a bitstream using a provided CAD tool. Then, when this bitstream is sent to the FPGA, the circuit is realized in the FPGA.

<STEP4>
Next, using the conventional FPGA synthesis tool Vivado (registered trademark), the image recognition task was verified by implementing it on the FPGA (FPGA synthesis 204).

<STEP5>
After verification, the substrate 205 was completed. The noise convolution arithmetic circuit 100 is implemented as hardware on the substrate 205 .

As described above, in the neural network circuit device 50 according to the present embodiment, the intermediate layer includes the noise convolution arithmetic circuit 100 that performs noise convolution arithmetic. a noise generator circuit 110 that generates _nc ; a summing circuit 120 that receives an input value X for performing a 1×1 convolution and adds the noise _nc generated by the noise generator circuit 110 to the input value X; a multiplier circuit 130 which receives the weight W and multiplies the weight W by the noise convoluted value of the adder circuit 120 carrying noise.

The noise (noise) n _c is
Expected value: E(n _c )=E(Σ _Nc ε _i W _i ′)=0, and
Variance: E(n _c ² )=E(Σ _Nc ε _i W _i ') ² = 2σ ² δ'
When using a noise n _c such that
The output y′ of the convolution operation by the convolution operation circuit 30 of the comparative example and the output y′ of the noise convolution operation by the noise convolution operation circuit 100 are expressed by the above equation (5) and are statistically equivalent. be.

Further, in the neural network processing method according to the present embodiment, the intermediate layer has a noise convolution operation step for executing a noise convolution operation. , a calculation step of adding noise to the input value X, and a multiplication step of receiving a weight W and multiplying the noise convolution calculation value of the calculation circuit by the weight W are executed.

As a result, some of the enormous number of convolutional calculations required in CNN are substituted by adding weighted random numbers (statistical behavior is noise). As a result, it is possible to shorten the convolution calculation time and reduce the memory by substituting the random number generator for the weight matrix.

In this way, the noise convolution arithmetic circuit 100 performs 1×1 convolution by adding noise instead of 3×3 weights (see FIGS. 14 and 15). As a result, the weight can be reduced to 3×3/1 at maximum, and the computational complexity can also be reduced to 3×3/1/3. As a result, the amount of calculation and memory can be reduced. Moreover, even though the amount of calculation and the amount of memory are reduced, there is no decrease in recognition accuracy (see the table in FIG. 19). Since the amount of calculation and memory can be reduced, it can be applied to embedded devices.

In this embodiment, both the number of convolution operations and the amount of parameters (the amount of memory for storing the weight matrix) can be reduced. Preliminary experiments have achieved recognition accuracy equivalent to that of convolution operations. Moreover, the noise convolution arithmetic circuit 100 has succeeded in reducing the arithmetic time by about 40% compared to the convolution arithmetic circuit 30 of the comparative example. It is characterized by applying noise, which is an obstacle to circuits, to the convolution operation.

In this embodiment, the configuration and effects of the noise convolution arithmetic circuit 100 are clarified through statistical analysis. As described above, the present invention holds from theoretical analysis only when the input statistically unbiased assumption of the noise convolution layer is satisfied (see FIG. 16). Therefore, in the neural network circuit device 50 according to the present embodiment, at least the first layer in the intermediate layer is composed of normal convolutional layers (existing convolutional layers). That is, the neural network circuit device 50 according to the present embodiment requires a combination of a normal convolutional layer and a noise convolutional layer.

From this point of view, the neural network circuit device 50 of the present embodiment includes the convolution operation circuit 30 of the comparative example in which the intermediate layer has n layers and the k layers perform convolution operations, and (n−k) a noise convolution circuit 100 for performing noise convolution in the layer.

Further, in the neural network processing method according to the present embodiment, the intermediate layer has n layers (n is any natural number), and k layers (k is a natural number that satisfies nk) execute convolution operations. and a noise convolution operation step for executing noise convolution operation in the (nk) layer, wherein the noise convolution operation step includes an input value X to be convoluted and noise , a calculation step of adding noise to the input value X, and a multiplication step of receiving a weight W and multiplying the noise convolution calculation value of the calculation circuit by the weight W are executed.

As a result, as shown in the table of FIG. 19, in the 13-layer CNN (see FIG. 18), the k layer is the existing CNN ((convolution operation circuit 30 of the comparative example shown in FIGS. 26 to 20), and the rest is noise. When executed by the convolution operation circuit 100 (see FIGS. 2 and 5 to 13), the weight memory (circuit scale) is reduced without reducing the recognition accuracy compared to the existing CNN with k=6 or more. In particular, when k=6, the computation time is reduced to about 1/2, as shown in the table of FIG.
As described above, it has been confirmed that it is possible to reduce the circuit and calculation time without degrading the recognition accuracy.

The present invention is not limited to the above-described embodiments, and includes other modifications and applications without departing from the gist of the present invention described in the claims.
For example, a LUT (Look-Up Table) may be used instead of a logic gate as a multiplication circuit. A LUT is a basic building block of an FPGA, has a high affinity for FPGA synthesis, and is easy to implement with an FPGA.

Moreover, the convolutional CNN layer number k shown in FIGS. 18 and 19, the method of combining the existing CNN and the noise convolutional CNN, the order of combination, and each parameter associated with the combination may be arbitrary.

Moreover, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to those having all the described configurations. Further, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. . In addition, the embodiment can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

Further, among the processes described in the above embodiments, all or part of the processes described as being automatically performed can be manually performed, or the processes described as being performed manually can be performed manually. All or part of this can also be done automatically by known methods. In addition, information including processing procedures, control procedures, specific names, and various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified.
Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

Further, each of the above configurations, functions, processing units, processing means, and the like may be realized by hardware, for example, by designing them in an integrated circuit. Further, each configuration, function, etc. described above may be realized by software for a processor to interpret and execute a program for realizing each function. Information such as programs, tables, files, etc. that realize each function can be stored in recording devices such as memory, hard disks, SSD (Solid State Drives), IC (Integrated Circuit) cards, SD (Secure Digital) cards, optical discs, etc. It can be held on a recording medium.
In the above embodiments, the device is called a neural network circuit device, but this is for the convenience of explanation, and the names may be deep neural network circuit, neural network device, perceptron, or the like. Moreover, although the method and program are named "neural network processing method", they may also be called "neural network operation method", "neural network program", or the like.

1 deep neural network 2 neural network circuit 11 input layer 12 hidden layer (intermediate layer)
13 Output layer 30 Convolutional operation circuit of comparative example (convolutional operation circuit means)
50 neural network circuit 60 CPU
70 off-chip memory 80 system bus 100 noise convolution arithmetic circuit (noise convolution arithmetic circuit means)
110 noise generation circuit 120 addition circuit (arithmetic circuit, arithmetic means)
130 Multiplication circuit (multiplication means)
131 weight memory 140 adder circuit 150 activation function circuit n _c noise (noise)
X input value W weight

Claims (13)

A neural network circuit device comprising at least an input layer, one or more intermediate layers, and an output layer,
The intermediate layer includes a convolution circuit that performs a convolution operation;
a noise convolution operation circuit that performs noise convolution operation ,
The noise convolution arithmetic circuit,
an input value X to be convolved, an arithmetic circuit that receives noise and adds the noise to the input value X;
a multiplier circuit that receives a weight W and multiplies the weight W by a noise convolution calculation value of the arithmetic circuit ;
The noise is
Expected value: E(n _c )=E(Σ _Nc ε _i W _i ′)=0, and
Variance: E(n _c ² )=E(Σ _Nc ε _i W _i ') ² = 2σ ² δ'
When using a noise n _c such that
Output y′ of the convolution operation by the convolution operation circuit;
The output y' by the noise convolution operation by the noise convolution operation circuit is represented by the following formula and is statistically equivalent

A neural network circuit device characterized by: A neural network circuit device comprising at least an input layer, one or more intermediate layers, and an output layer,
The intermediate layer has n (n is any natural number) layers,
A convolution circuit that performs a convolution operation in k layers (k is a natural number that satisfies nk);
a noise convolution circuit that performs noise convolution computation in the (nk) layer;
The noise convolution arithmetic circuit,
an input value X to be convolved, an arithmetic circuit that receives noise and adds the noise to the input value X;
a multiplier circuit that receives a weight W and multiplies the weight W by a noise convolution calculation value of the arithmetic circuit ;
The noise is
Expected value: E(n _c )=E(Σ _Nc ε _i W _i ′)=0, and
Variance: E(n _c ² )=E(Σ _Nc ε _i W _i ') ² = 2σ ² δ'
When using a noise n _c such that
Output y′ of the convolution operation by the convolution operation circuit;
The output y' by the noise convolution operation by the noise convolution operation circuit is represented by the following formula and is statistically equivalent

A neural network circuit device characterized by: The noise convolution arithmetic circuit,
3. The noise according to claim 1 or claim 2 , wherein the noise that makes the mean and variance of the calculation result by the convolution operation and the mean and variance of the calculation result by the noise convolution operation match, respectively. neural network circuit device. The weights used in the noise convolution circuit are derived according to the following equation based on the noise

3. The neural network circuit device according to claim 1, wherein: The arithmetic circuit is
3. The neural network circuit device according to claim 1, wherein the adder circuit adds the noise to the input value X. The noise convolution arithmetic circuit,
5. The neural network circuit device according to claim 1 , further comprising a noise generation circuit that generates the noise. 7. The neural network circuit device according to claim 6 , wherein the noise is a pseudo-random number generated by a random number generator including XOR-Shift, LFSR (linear feedback shift register), and linear congruential method. The noise convolution arithmetic circuit,
3. The neural network circuit device according to claim 1, wherein a 1*1 convolution operation using a 1*1 kernel is executed. The noise convolution arithmetic circuit,
characterized by comprising a summation circuit that sums each convoluted operation value and a bias W0, and an activation function circuit that converts the summed signal Y by an activation function f(u). 3. The neural network circuit device according to claim 1 or claim 2. A neural network processing method including at least an input layer, one or more intermediate layers, and an output layer,
The intermediate layer includes a convolution operation step for performing a convolution operation;
a noise convolution operation step of performing a noise convolution operation;
In the noise convolution operation step,
an input value X to be convoluted, a computing step of receiving noise and multiplying the input value X with the noise;
a multiplication step of receiving a weight W and multiplying the weight W by the noise convoluted value of the operation step ;
The noise is
Expected value: E(n _c )=E(Σ _Nc ε _i W _i ′)=0, and
Variance: E(n _c ² )=E(Σ _Nc ε _i W _i ') ² = 2σ ² δ'
When using a noise n _c such that
Output y' of the convolution operation by the convolution operation step;
The output y' by the noise convolution operation in the noise convolution operation step is represented by the following formula and is statistically equivalent

A neural network processing method characterized by: A neural network processing method including at least an input layer, one or more intermediate layers, and an output layer,
The intermediate layer has n (n is any natural number) layers,
A convolution operation step of performing a convolution operation in k (k is a natural number satisfying nk) layers;
a noise convolution operation step of performing a noise convolution operation in the (nk) layer;
In the noise convolution operation step,
an input value X to be convoluted, a computing step of receiving noise and multiplying the input value X with the noise;
a multiplication step of receiving a weight W and multiplying the weight W by the noise convoluted value of the operation step ;
The noise is
Expected value: E(n _c )=E(Σ _Nc ε _i W _i ′)=0, and
Variance: E(n _c ² )=E(Σ _Nc ε _i W _i ') ² = 2σ ² δ'
When using a noise n _c such that
Output y' of the convolution operation by the convolution operation step;
The output y' by the noise convolution operation in the noise convolution operation step is represented by the following formula and is statistically equivalent

A neural network processing method characterized by: A computer as a neural network circuit device including at least an input layer, one or more intermediate layers, and an output layer,
the intermediate layer includes convolution circuit means for performing a convolution operation;
noise convolution circuit means for performing noise convolution computation;
The noise convolution arithmetic circuit means is
Arithmetic circuit means for receiving an input value X to be convoluted and noise and multiplying the input value X with the noise;
Multiplication circuit means for receiving the weight W and multiplying the weight W by the noise convoluted value of the arithmetic circuit means;
function as
The noise is
Expected value: E(n _c )=E(Σ _Nc ε _i W _i ′)=0, and
Variance: E(n _c ² )=E(Σ _Nc ε _i W _i ') ² = 2σ ² δ'
When using a noise n _c such that
Output y' of the convolution operation by the convolution operation step;
The output y' by the noise convolution operation in the noise convolution operation step is represented by the following formula and is statistically equivalent

Neural network execution program. A computer as a neural network circuit device including at least an input layer, one or more intermediate layers, and an output layer,
The intermediate layer has n (n is any natural number) layers,
convolution circuit means for executing a convolution operation in k layers (k is a natural number satisfying nk);
noise convolution circuit means for performing noise convolution computation in the (nk) layer;
The noise convolution arithmetic circuit means is
Arithmetic circuit means for receiving an input value X to be convoluted and noise and multiplying the input value X with the noise;
Multiplication circuit means for receiving the weight W and multiplying the weight W by the noise convoluted value of the arithmetic circuit means;
function as
The noise is
Expected value: E(n _c )=E(Σ _Nc ε _i W _i ′)=0, and
Variance: E(n _c ² )=E(Σ _Nc ε _i W _i ') ² = 2σ ² δ'
When using a noise n _c such that
Output y' of the convolution operation by the convolution operation step;
The output y' by the noise convolution operation in the noise convolution operation step is represented by the following formula and is statistically equivalent

Neural network execution program.

Images