JPH10187648A - Neural unit operation system and its device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a neural unit device which shortens a processing time accompanying an increase in the scale of a network by performing a repeating process for N-time multiplication for each neural unit. SOLUTION: A product Wji Xi calculated by the neural unit is -L to M (L, M >=0). When the sum of products is calculated by the neural unit, L+Wji Xji is added at each time, where -LN is its initial value (N: repetition frequency of a product sum). Here, it is notable that the numeral (L+Wji Xji which is added at each time is always not negative. Judging from the shape of a sigmoid function, the value is saturated on the positive side and negative side of an input (m). If the value (=-LN+Σi <k> (L+Wji Xi )) of the halfway sum of products up to (k term) before a final term (Nth term) is reached is in the saturation area of the sigmoid function as L+Wi Xji (i=1...N) is added, subsequent multiplication and cumulative sum calculation are omitted.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a neural unit operation method for performing a product-sum operation and a sigmoid function conversion, which are basic operations of a neural network, and an apparatus therefor.

[0002]

2. Description of the Related Art Recently, with the development of semiconductor technology, attempts to realize a neural network by a digital LSI have become active. FIG. 3 shows the basic configuration of the neural network. Referring to FIG. 3, a plurality of neural units are connected in a network, and each neural unit receives a signal from a plurality of other neural units and supplies an output to another neural unit. Each neural unit has a "synaptic weight value" that indicates the degree of coupling between the neural unit and a neural unit that supplies a signal to the neural unit.

For example, in FIG. 3, a j-th neural unit 301 receives outputs from A-th, B-th, and C-th neural units 302, 303, and 304. Jth
The neural unit 301 includes synapse load values W _jA , W _jB , and W _{jC indicating the} strength of connection between these three neural units ₃₀₂ , ₃₀₃ , ₃₀₄ .

The output signals of the A-th, B-th, and C-th neural units 302, 303, and 304 are respectively
Let X _A , X _B , X _C. Then, the signal X _j from the j-th neural unit is calculated and output as follows.

First, using the signals X _A , X _B , X _C and the synapse load values W _jA , W _jB , W _jC , the product sum m of the following equation (1) is obtained.
Is calculated.

_M = W _jA X _A + W _jB X _B + W _jC X _C (1)

The value of m is subjected to a non-linear conversion process using the following sigmoid function, and _Xj is output. That is, m and the output _Xj are related by the sigmoid function f as in the following equation (2).

X _j = f (m) (2)

The sigmoid function f is a non-linear function having a shape as shown in FIG. 2. When the input m is near zero, its value increases sharply with m. The value is saturated.

In the neural network, the neural unit, which is a component of the neural network, performs the above-described product-sum operation and sigmoid function conversion process, and outputs the result to the next-stage neural unit. After signal transmission is performed by the number of stages, a signal is output to the outside.

In such a neural network, basic operations to be processed by each neural unit are a product-sum operation and a sigmoid function conversion as in the above equations (1) and (2). Therefore, the neural unit can be realized by a digital circuit and a circuit configuration as shown in FIG.

FIG. 4 is a diagram showing a basic configuration of a conventional neural unit using a digital circuit. The operation of the conventional digital neural unit will be described below with reference to FIG.

In FIG. 4, the neural unit is a j-th unit.
Let it be a neural unit (see 301 in FIG. 3). It is assumed that N neural units (first to N-th neural units) supply signals X _{1 to} X _N to this neural unit, respectively. Further, the synapse load values between the first to Nth neural units and the jth neural unit are W _j1 , W _j2 ,
W _j3 ,..., W _jN . Note that the signals X _{1 to} X _N and the synapse load values W _{j1 to} W _jN are each represented by a binary number having an appropriate bit length.

As shown in FIG. 4, the j-th neural unit includes a multiplier 402, an adder 403, and
It comprises a synapse load value memory 401 for storing synapse load values (W _j1 , W _j2 , W _j3 ,..., W _jN ).

Output signals X _{1 to} X _N from the N neural units are sequentially supplied to an input signal line 400, and in synchronism _therewith , W _j1 , W _j2 , .., W _jN are sequentially read.

Thus, in the first cycle, the input signal line 4
X ₁ is supplied to 00, the synapse load value W _j1 is read from the synapse load value memory 401, and the product of the two is calculated by the multiplier 402.

Subsequently, in the second cycle, similarly, X ₂
And W _j2 are calculated. The products W _ji X _i obtained for each cycle are accumulated by the adder 403.

In this way, as shown in the following equation (3), the value of the sum of products m is calculated over N cycles.

[0019]

(Equation 1)

The value of m is calculated by the sigmoid function converter 40
4 is input. The sigmoid function converter 404 is a signal converter having sigmoid function input / output characteristics. Thus, the output signal _Xj is output from the sigmoid function converter 404.

Various variations are conceivable for the configuration and operation of the neural unit using the digital circuit described above. However, all of these will not be described here. Both variations are basically the same in that they are composed of a product-sum operation unit and a sigmoid function converter.

The construction of a neural unit using such a digital circuit is described in, for example, a paper (Yasunaga et al., "A Self-Learning Digital Neural Network").
Network Using Wafer Scale LSI ", I Triple E Journal of Solid State Circuits, 1993 Feb. vo
l28, no.2, pp106-114 (M. Yasunaga et.al “AS
elf-Learning Digitalneural Network Using Wafer
-Scale LSI ”, IEEE Journal of Solid-State Ci
rcuits, Feb. 1993, vol. 28, no. 2, pp. 106-114).

[0023]

In the conventional digital neural network as described above, it is necessary to execute a repetition of N multiplications for each neural unit. For this reason, as the number of connected neural units increases with the increase in the scale of the network, N also increases, and the processing time becomes extremely long.
There is a problem that.

Accordingly, the present invention has been made in view of the above circumstances, and an object of the present invention is to provide a neural unit operation method capable of realizing high-speed processing by reducing the number of product-sum repetitions. It is to provide the device.

[0025]

In order to achieve the above object, a neural unit operation system according to the present invention performs a predetermined number of multiplications of an input signal as array data and a synapse load value as array data. In the neural unit operation method of summing the result, converting the sum value to a sigmoid function, and outputting the result, a non-zero value is used as an initial value, and only a positive value or a negative value is summed. When the sum reaches a value specified in advance, the sum ends.

Further, the neural unit device of the present invention
A basic configuration unit of a neural network, a digital multiplier and a digital adder perform a predetermined number of seat wheel calculations of an array-type input signal and an array-type data of a synapse load value, and then use the result as a sigmoid. In a neural unit device that converts and outputs a function, a product-sum operation unit that accumulates only positive values or only negative values with a non-zero value as an initial value, and an output of the product-sum operation unit is a sigmoid function And a detection circuit for detecting whether or not the current value falls within the saturation range.

Further, the neural unit arithmetic unit of the present invention sets -LN as an initial value, and L + (L + (N), where N is the number of product-sum repetitions and L is the minimum value among values that can be output from the digital multiplier. A detector that accumulates (synapse load value) × (input signal value) and detects whether or not the accumulated value falls within the saturation range of the sigmoid function.

Further, in the neural unit device of the present invention, when the number of repetitions of the sum of products is N and the maximum value among the values that can be output from the digital multiplier is M, MN is set to an initial value, and -M + (synapse (A load value) × (input signal value), and a detector for detecting whether or not the accumulated value is within a saturation range of the sigmoid function.

The neural unit device according to the present invention comprises a synapse load value memory, a multiplier for calculating a product of an output of the synapse load value memory and an input signal, and an output value, a constant value, and an adder of the multiplier. A CSA that receives an output value and outputs a carry and an intermediate sum, a selector that receives a carry output of the CSA and an initial value of accumulation, an output of the selector and an intermediate sum of the CSA, , A sigmoid function converter receiving the output of the adder as an input, and a saturation determiner receiving the output of the adder as an input.

[0030]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments and examples of the present invention will be described below in order. First, the principle of the present invention will be described below.

The value of the product W _ji X _i calculated in the j-th neural unit (see FIG. 3) is generally in the range of the following equation (4).

[0032] _{-L ≦ W ji X i ≦ M} (L, M ≧ 0) ... (4)

The values of L and M are determined by the bit length when expressing W _Ji and X _i . For example, W _ji and X _i are both 4-bit numerical parts (negative numbers represent absolute values)
And a binary number with a 1-bit sign.

[0034] Then, since it is _{-15 ≦ W ji ≦ 15 -15 ≦} W i ≦ 15, for the product becomes _{-225 ≦ W ji X i ≦ 225} . That is, L = 225 and M = 225.

The value of the thus, L, M are determined by the bit length in representing the W and _i X _i.

In the present invention, the initial value is -LN in the product-sum operation performed by the neural unit. Here, N is the number of times the product-sum is repeated. And unlike conventional, going by adding every time the L + W _ji X _i. That is, m is calculated by the following equation (5).

[0037]

(Equation 2)

Here, the numerical value added each time (L + _WjiX
Note that _i ) is always a positive or zero number (non-negative value).

Now, paying attention to the shape of the sigmoid function (FIG. 2), the value is saturated on the positive and negative sides of the input m. Therefore, assuming that the initial value is -LN, L +
In the process of adding W _ji X _i (i = 1, 2,..., N), the sum value up to the k-th term in the middle (the following equation (Eq. 6), for example, if it falls within the saturation region a in FIG. 2, the subsequent multiplication and accumulation calculation can be omitted.

[0040]

(Equation 3)

Since only positive or zero values are added after the (k + 1) -th term, the final result is always in the saturation region a. Then, in the saturation region, the output value of the sigmoid function hardly changes. That is, even if the calculation is stopped when the product-sum operation result enters the saturation region, a value that is almost the same as when the calculation is repeated to the end is obtained.

Therefore, if the product sum is repeatedly executed for the calculation of the above equation (3), and if the product sum of all the N terms is not performed, and if the product enters the saturation region a during the iteration, By terminating the calculation, the number of product-sum repetitions can be reduced.

Of course, the final product-sum result is not always in the saturation region. In that case, N repeated calculations are required. However, averaging over the entire network would reduce the number of product-sum iterations,
As a result, even if N increases, a neural network device with a shorter processing time than the conventional configuration can be realized.

The neural unit operation method according to the present invention based on the above-described principle, in its preferred embodiment,
With a non-zero value as an initial value, only the positive value or only the negative value is accumulated, and when the accumulated value reaches a predetermined value, the accumulation is terminated.

In a preferred embodiment of the neural unit apparatus according to the present invention, a digital multiplier (102 in FIG. 1) and a digital adder (103 in FIG. 1) which constitute basic constituent units of the neural network.
With, by performing a predetermined number of times of multiplication of the input signal which is array type data and the synapse load value which is array type data,
In a neural unit device that accumulates the result, converts the accumulated value into a sigmoid function, and outputs the result, a non-zero value is used as an initial value, and only a positive value or only a negative value is accumulated. A detection circuit (109 in FIG. 1) for detecting whether or not the output of sigmoid function falls within the saturation range of the sigmoid function
Is provided.

More specifically, in the neural unit apparatus of the present invention, in a preferred embodiment, the number of product-sum repetitions is N, and a digital multiplier (102 in FIG. 1)
When the minimum value among the values that can be output is L, -LN
Is used as an initial value, L + (synapse load value W _ji ) × (input signal value X _i ) is accumulated, and a detection circuit (109 in FIG. 1)
, It is detected whether or not this sum value has entered the saturation range of the sigmoid function. Alternatively, when the number of product-sum repetitions is N and the maximum value among the values that can be output from the digital multiplier is M, MN is set as an initial value, and −M + (synapse weight value W _ji ) × (input signal value X _i ) are accumulated, and a detection circuit (109 in FIG. 1) detects whether or not the accumulated value falls within the saturation range of the sigmoid function.

[0047]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing an embodiment of the present invention;

FIG. 1 is a diagram showing the configuration of one embodiment of the present invention. In one embodiment of the present invention, the neural unit shown in FIG. 1 is referred to as a j-th neural unit (see FIG. 3), and the neural unit includes first to N-th neural units in the same manner as described in the above-mentioned prior art. , Are supplied with signals X _{1 to} X _N , respectively. Also the first
The synapse load values between the Nth neural unit and the jth neural unit are W _j1 , W _j2 ,
W _j3 ,..., W _jN . It is _assumed that the signals X _{1 to} X _N and the synapse load values W _{j1 to} W _jN are each represented by a binary number having an appropriate bit length. W _{j1 to} W _jN are stored in the synapse load value memory 101.

The output signals X _{1 to} X _N from the N neural units are sequentially supplied to the input signal line 100, and in synchronism _therewith , the synapse load value memory 101 outputs W _j1 , W _j2 , .., W _jN are sequentially read.

That is, in the first cycle, the input signal line 1
X ₁ is supplied to 00, W _j1 is read from the synapse load value memory 101, and the product of the two is calculated by the multiplier 102.

Subsequently, in the second cycle, similarly, X ₂ and W
The product of _j2 is calculated.

The adder 103 calculates the sum. In the conventional configuration, the initial value of the sum calculation is zero, whereas in the present embodiment, the initial value is -LN. . Here, L is as previously described as the principles of the present invention, the minimum value of W _ji X _i. Setting the initial value can be easily realized by the selector 106. That is, only at the very beginning of the cumulative sum calculation, the selector 106
LN is selected and input to the adder 103, and thereafter,
W _j1 X ₁ and W _j2 X ₂ may be sequentially accumulated.

The CSA 105 is a so-called Carrier Save Adder;
A carry-save adder), which inputs three numbers, adds them, and outputs two numbers, that is, a carry and an intermediate sum. Referring to FIG. 1, the CSA 105 includes an adder 1
The output value of 05, the output value of the multiplier 102 (W _ji X _i), and as input L, and the outputs of the carry 107 and intermediate sum 108. Both are added by the adder 103. L + W _ji X _i in every cycle in this way is going to be added.
As described above, the output of the adder 103 in the k-th cycle has the following value, as already described as the principle of the present invention (see the above equation (6)).

[0054]

(Equation 4)

The output of the adder 103 becomes the input of the sigmoid function converter 104. Then, for each cycle, the saturation determiner 109 sets the output value of the adder 103 to the saturation region a.
(See FIG. 2).

If the saturation has entered the saturation region a, the saturation determiner 109 sends a calculation end signal 140. When the calculation end signal 140 is transmitted, the neural unit ends the process even if the number of multiplications has not reached N.

Thus, even if the calculation is not performed up to N times, the value of the output X _j is already a sufficiently accurate value (that is, the value when the calculation is performed N times).
The principle of the present invention is as described above.

In the above description, -LN is used as the initial value of the summation.
The initial value of the sum may start from MN.

[0059] Here, M is as defined in the above formula (4), the maximum value of W _ji X _i. Then, the value to continue to Ruiwa may be set to -M + W _ji X _i.

That is, m is calculated by the following equation (7).

[0061]

(Equation 5)

In this case, starting from the maximum positive value MN, only a negative or zero number is added each time. Therefore, the intermediate result up to the k-th term is the saturation region b in FIG.
If entered, the calculation ends. What may be done is that only negative or zero values are added after the (k + 1) th term, so the final result is always in the saturation region b, and even if the calculation is stopped halfway, the calculation is repeated to the end. This is because a value that is almost the same as that of the time is obtained.

In this method, in the embodiment shown in FIG. 1, the initial value 120 is set to MN, and the input 130 of the CSA is
M can be realized.

[0064]

As described above, according to the present invention,
Since the number of product-sum repetitions required N times in the conventional neural unit can be reduced, a high-speed neural unit can be realized.

Further, according to the present invention, when a large-scale neural network is constructed by increasing the number of neural units, it is possible to suppress an increase in processing time.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of an embodiment of a neural unit according to the present invention.

FIG. 2 is a diagram showing a shape of a sigmoid function.

FIG. 3 is a diagram showing a basic configuration of a neural network.

FIG. 4 is a diagram showing an example of a conventional configuration of a neural unit using a digital circuit.

[Explanation of symbols]

100, 400 Input signal line 101, 401 Synapse load value memory 102, 402 Multiplier 103, 403 Adder 104, 404 Sigmoid function converter 105 CSA 106 Selector 107 Carry 108 Intermediate sum 109 Saturation detector 110, 410 Output signal X _j 120 initial value (-LN) 130 L 140 calculation end signal 301 j-th neural unit 302-A neural unit 303 first B neural unit 304 first C neural unit 305 neural unit 310 signals X _j 311 signal X _A 312 signal X _B 313 Signal X _C

Claims (5)

[Claims] A multiplication of an input signal, which is array type data, and a synapse load value, which is array type data, a predetermined number of times;
The result is summed, the sum value is converted to a sigmoid function and output.In the neural unit operation method, only the positive value or only the negative value is summed with the non-zero value as an initial value, A neural unit operation method, wherein the accumulation is terminated when the accumulated value reaches a predetermined value. 2. A digital multiplier and a digital adder, which are basic constituent units of a neural network, perform a predetermined number of multiplications of an input signal which is array type data and a synapse load value which is array type data, In a neural unit that accumulates the result, converts the sum to a sigmoid function, and outputs the result, a non-zero value is used as an initial value, and only the positive value or the negative value is summed. And a detection circuit for detecting whether or not the output of the product-sum operation unit is within the saturation range of the sigmoid function. 3. The neural unit device according to claim 2, wherein N is the number of repetitions of the sum of products, and L is the minimum value among the values that can be output from the digital multiplier. , L + (synapse weight value) × (input signal value), and wherein the detection circuit detects whether or not the sum value has entered a saturation range of a sigmoid function. Unit equipment. 4. The neural unit device according to claim 2, wherein the number of repetitions of the sum of products is N, and the maximum value among the values that can be output from the digital multiplier is M.
When MN is set as an initial value, −M + (synapse load value) × (input signal value) is accumulated, and in the detection circuit, whether the accumulated value is within the saturation range of the sigmoid function A neural unit device for detecting whether or not the neural unit is in a position. 5. A storage unit storing a synapse load value, a multiplier for calculating a product of a synapse load value output from the storage unit and an input signal, an output value of the multiplier, a constant value, and an adder CSA (carry / carry) that takes the output value of
Save adder), and a selector that receives a carry output of the CSA and an initial value of accumulation, and the adder receives an output of the selector and an intermediate sum that is an output of the CSA. And a sigmoid function converter that receives the output of the adder as an input, and a saturation that determines whether or not the output is within the saturation range of the sigmoid function with the output of the adder as an input. A neural unit device comprising: a determiner;