JPH05128284A - Neuro-processor - Google Patents

Abstract

PURPOSE:To prevent the overflow of parameters and functions by constituting a processor so that the calculation of a computational equation transformed by using a normalized parameter and a function may be performed. CONSTITUTION:Each parameter and each function are normalized to sizes to be fitted for dynamic range, respectively. The calculation of a computational equation transformed by using the normalized parameter can be performed by inputting the output of a sum of products operator 1 in a factor multiplier 2 and by inputting the output of the factor multiplier 2 in a neuron characteristic function circuit 3. Thus, the occurrence of the overflow of parameters and functions can be prevented, the word lengths alloted to parameters and functions, respectively can be fully utilized and a calculation with high accuracy becomes possible.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a neuroprocessor for performing neural network calculations with high accuracy.

[0002]

BACKGROUND OF THE INVENTION Neural networks model and imitate the function of nerve cells in the human brain,
It is intended to realize a new computer which is good at recognition, association, optimization problem, speech synthesis, etc., which are not good at the conventional so-called Neumann type computer.

There are various types of neural networks, such as a hierarchical structure in which neurons are arranged in layers and an interconnected structure in which all neurons are connected to each other. Among them, the hierarchical network can be easily learned by a learning algorithm called a backpropagation algorithm and is considered to be widely applicable to control, character recognition, image recognition, image processing, etc. ..

FIG. 3 shows an example of a hierarchical network. In FIG. 3, 101 is a neuron arranged in each layer, 102 is a connection between neurons called synapse, 103 is an input layer, 104 is an intermediate layer, and 105 is an output layer. Although FIG. 3 shows an example of a three-layer network, a network having a hierarchical structure of four or more layers can be formed by using a plurality of intermediate layers. Hereinafter, the description will be limited to the network of three layers, but the same applies to the network of four layers or more. Also, FIG.
Then, the numbers of neurons in each layer are illustrated as 3, 2 and 3 respectively, but the same applies when the number of neurons in each layer is increased or decreased. The input signal to the neural network is given to each neuron of the input layer 103. Then, the signal propagates in the order of the input layer, the intermediate layer, and the output layer, and the signal of the neuron in the output layer 105 becomes the output of the network. Such input layer, middle layer,
Normal propagation that propagates in the order of the output layer is called forward propagation.

FIG. 4 is a diagram showing the functions of neurons. In FIG. 4, 101, 107, 108, and 109 are neurons, 102 is a synapse, and 106 is a characteristic function f of the neuron. The neuron receives the output of the neuron in the previous layer via the synapse. Each synapse has a value called a connection weight, and the result obtained by multiplying the output value of the neuron in the previous layer by the value of the connection weight is given to the next neuron. The weight of synaptic connection is different for each synapse. For example, the synapse between the i-th neuron 108 of the l-th layer and the j-th neuron 101 of the l + 1-th layer in FIG. 4 has a connection weight w _lji , and the i-th neuron 1 of the l-th layer 1
The result w of multiplying the output O _li of 08 by the value of the weight of the connection
_lji × O _li is given to the jth neuron 101 of the (l + 1) th layer. Then, the neuron 101 adds all the inputs given by all the synapses connected to the neuron 101, causes the characteristic function 106 of the neuron to act on the addition result, and outputs the function value as the output O _{l + 1, j} of the neuron 101. To do. This can be expressed by the following equation.

[0006]

[Equation 1]

The characteristic function f of a neuron is usually

[0008]

[Equation 2]

The function represented by Therefore, 3
Forward propagation for a layer network is calculated by the following procedure. First of all, for all neurons in the middle layer, the output is

[0010]

[Equation 3]

Calculate according to Next, using the result, for all the neurons in the output layer, its output is

[0012]

[Equation 4]

Calculate according to: Next, a learning method using the backpropagation algorithm of the three-layer hierarchical network of FIG. 3 will be described. The backpropagation algorithm prepares a pair of an input and an ideal output corresponding to the input, and modifies synaptic coupling weights so that the difference between the actual output and the ideal output for the input is reduced. This ideal output is usually called a teacher signal. Hereinafter, a specific calculation method for the three-layer network will be described step by step. 1. Calculate the actual output by forward propagation. 2. A value (hereinafter simply referred to as delta) corresponding to the error of each neuron in the output layer is calculated according to the following equation.

[0014]

[Equation 5]

Δ _lj is the delta for the j-th neuron in the l-th layer, t _j is the teacher signal for the j-th neuron in the output layer, and g is

[0016]

[Equation 6]

The characteristic function f of the neuron as represented by
Is the differential coefficient of. Characteristics of neurons

Since the function f is a monotone non-decreasing function, the differential coefficient of the characteristic function can be expressed as a function of the function value of the characteristic function, as shown in (Equation 6). 3. The correction amount of the weight of the synapse connection between the intermediate layer and the output layer is calculated according to the following equation, and the weight is corrected.

[0019]

[Equation 7]

Η is a correction coefficient. 4. The delta for the hidden layer neurons is calculated according to the following equation:

[0021]

[Equation 8]

5. The correction amount of the synaptic connection weight between the intermediate layer and the input layer is calculated according to the following equation, and the weight is corrected.

[0023]

[Equation 9]

In the neuroprocessor, it is necessary to perform the above calculation formula calculations. Conventionally, a neuroprocessor capable of performing these calculations has been developed mainly by a fixed-point method that requires a small circuit scale.

[0025]

In fixed point arithmetic,
For each variable and each function, the position of the decimal point is fixed in advance with respect to the word length (bit number) set for each variable and fixed. Therefore, the dynamic range of each variable and each function becomes narrow. That is, a variable or function overflows, or the word length assigned to each variable or function cannot be fully utilized. Therefore, highly accurate calculation cannot be performed.

The present invention solves the above problems, and an object of the present invention is to provide a neuroprocessor capable of highly accurate calculation.

[0027]

In order to solve the above problems, the neuroprocessor of the present invention is configured to execute the calculation of the calculation formula converted by standardization of each variable and each function. For example, in the forward propagation calculation of the neural network, the output of the product-sum calculator is input to the coefficient multiplier, and the output of the coefficient multiplier is input to the neuron characteristic function circuit. Also, when calculating the delta of neurons in the middle layer of a hierarchical neural network learned by the back propagation algorithm, the value obtained by multiplying the normal calculation result by a coefficient by a coefficient multiplier is used. To the delta. As described above, by providing the coefficient multiplier, it is possible to execute the calculation of the calculation formula converted by the standardization of each variable and each function.

[0028]

With the above configuration, each variable and each function can be standardized to have a size that fits each dynamic range, and the calculation formula converted by the standardization can be executed. Therefore, variables and functions can be prevented from overflowing, and the word length assigned to each variable and function can be fully utilized, so that forward propagation of the neural network and backpropagation algorithm can be performed. The learning calculation can be performed with high accuracy.

[0029]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a data flow diagram for the forward propagation calculation of the neuroprocessor of one embodiment of the present invention. 1 in FIG. 1 is a product-sum calculator, 2 is a coefficient multiplier, 3
Is

The uron characteristic function circuit, 4 is a multiplier, and 5 is an adder. FIG. 1 shows a configuration in which one coefficient multiplier 2 is added to the configuration capable of calculating the normal forward propagation represented by (Equation 1).

FIG. 2 is a data flow diagram in the case of calculating the delta of the middle layer of the neuroprocessor according to the embodiment of the present invention. 2, 6 is a delta calculation circuit, 7 is a coefficient multiplier, 8 is a multiplier, 9 is an adder, 10 is a neuron characteristic function differential coefficient circuit, and 11 is a power.

It is a calculator. In FIG. 2, the delta calculation circuit 6
Is configured so that a normal delta represented by (Equation 8) can be calculated.

In this embodiment, the following standardization is performed. First, it is assumed that O, δ, and g are optimal and need not be standardized. Then, as in the following equation, the variables w and I are standardized using the standardization coefficients a and b.

[0035]

[Equation 10]

The symbol ˜ above the character representing a variable represents a standardized variable. Use (Equation 10)

Then, the forward propagation calculation formula (Equation 1) is converted as follows.

[0038]

[Equation 11]

Further, among the calculation formulas for learning by the backpropagation algorithm,

), (Equation 8) and (Equation 9) are converted as follows.

[0041]

[Equation 12]

[0042]

[Equation 13]

[0043]

[Equation 14]

Here,

[0045]

[Equation 15]

[0046]

[Equation 16]

The coefficient η and the function f are converted according to the standardization of the variables.

FIG. 1 is a diagram showing a data flow when calculating (Equation 11). As is clear from (Equation 11),
By normalizing the variables, a /
The number of operations for multiplying b increases. It is the coefficient multiplier 2 in FIG. 1 that performs this calculation. In this way, the output of the product-sum calculator 1 is input to the coefficient multiplier 2, and the output of the coefficient multiplier 2 is input to the neuron characteristic function circuit 3, so that conversion is performed using the standardized variables.

The calculation of the above calculation formula (Equation 11) can be performed. The neuron characteristic function circuit 3 in FIG. 1 may be an arithmetic circuit that actually calculates the characteristic function, a memory that stores a table of functions, or another configuration. However

The function generated by the neuron characteristic function circuit 3 is not f in (Equation 2) but the function represented by (Equation 16).

FIG. 2 is a diagram showing the flow of data when calculating (Equation 13). As is clear from (Equation 13),
By standardizing the variables, the number of operations for multiplying a increases. It is the coefficient multiplier 7 in FIG. 2 that performs this calculation. In this way, the output of the delta calculation circuit 6 which is the normal delta calculation result is input to the coefficient multiplier 7 and its output is

By using delta, it becomes possible to calculate the calculation formula (Equation 13) converted using the standardized variables. The neuron characteristic function differential coefficient circuit 10 in FIG. 2 may be an arithmetic circuit that actually calculates the differential coefficient of the characteristic function, a memory that stores a table of functions, or another configuration. It may be one.

The coefficient multiplier of this embodiment may be a normal multiplier or a barrel shifter. When the barrel shifter is used, it is necessary to select the normalization coefficient so that the coefficient becomes a power of 2.

Further, the present embodiment is an example in which the variables w and I are standardized as represented by (Equation 13). When other variables and functions are standardized, it is necessary to change the place where the coefficient multiplier is inserted, but the handling is the same.

As can be seen from the above description, the present processor is configured to standardize the variables and functions of the calculation formulas and perform the calculation of the calculation formulas converted using the standardized variables and functions. .. As a result, variables and functions can be prevented from overflowing, and the word length assigned to each variable and function can be fully utilized, enabling high-precision calculation. ..

[0056]

As is apparent from the above embodiments, according to the present invention, it is possible to standardize variables and functions of calculation formulas and perform calculation of calculation formulas that are converted using the standardized variables and functions. It has a simple structure. As a result, variables and functions can be prevented from overflowing, and the word length assigned to each variable and function can be fully utilized, so that forward propagation of the neural network,
It is possible to provide a neuroprocessor capable of highly accurately performing learning calculation by the back propagation algorithm.

[Brief description of drawings]

FIG. 1 is a data flow diagram for the calculation of forward propagation of a neuroprocessor according to an embodiment of the present invention.

FIG. 2 is a data flow diagram for the calculation of the middle layer delta of the neuroprocessor of one embodiment of the present invention.

FIG. 3 Diagram of hierarchical neural network

FIG. 4 is a diagram showing the function of neurons.

[Explanation of symbols]

 1 product-sum operator 2,7 coefficient multiplier 3 neuron characteristic function circuit 4,8,11 multiplier 5,9 adder 6 delta calculation circuit 10 neuron characteristic function differential coefficient circuit 101 neuron 102 synapse 103 input layer 104 intermediate layer 105 Output layer 106 Neuron characteristic function 107-109 neurons

Claims (2)

[Claims] 1. An arithmetic circuit for performing fixed-point arithmetic is provided,
A neuroprocessor characterized in that the arithmetic circuit is configured to execute a calculation of a calculation formula converted by standardization of each variable and each function. 2. A hierarchical neural network having a coefficient multiplier and learning by a backpropagation algorithm, having forward propagation and learning calculation functions, and a value corresponding to an error of a neuron in an intermediate layer. When calculating, input the usual calculation result to the coefficient multiplier,
A neuroprocessor characterized in that an output of the coefficient multiplier is set to a value corresponding to an error of a neuron in an intermediate layer.