JP2003263624A - Learning arithmetic circuit for neural network device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To execute the learning of an analog neural network device as quickly as possible. <P>SOLUTION: This arithmetic circuit is proposed to calculate the optimal load value directly from the change of an output at the time of finely changing a load at the time of executing learning by a back propagation method in an analog neural network device. This arithmetic circuit is able to obtain the proper load as the output of an analog circuit directly after a transient response time by using the similarity of gate voltage/drain current characteristics and proper load arithmetic operation in the non-saturated area of an MOS transistor. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

TECHNICAL FIELD The present invention relates to a semiconductor device,
The present invention relates to a learning arithmetic circuit of a neural network device that executes neural network information processing.

[0002]

2. Description of the Related Art As a device capable of high-speed arithmetic processing such as pattern recognition and image processing, research has been conducted on a computer and a neural network device that operate like a living brain. Especially, in neural network information processing (hereinafter referred to as NN), the number of integrations and summations explosively increases as the number of inputs increases. Therefore, it is a dedicated non-Neumann method that processes these in parallel by analog operations. There is an approach to speed up the calculation by realizing the device.

Such a neural network device and its learning method are disclosed in, for example, Japanese Patent Laid-Open No. 3-2268.
The "analog neural network learning device" described in Japanese Patent Publication No. 84 is cited.

This conventional example discloses a general-purpose method for executing learning by a general back propagation method (hereinafter referred to as BP method) as an NN learning method by an analog neural network device.

FIG. 10 shows an element structure of a neural network device of a prior art example, and FIG. 11 shows an operation procedure in an actual learning operation. In the conventional example, the load is actually slightly changed, the optimum load in the BP method is calculated from the output change at that time, and the learning is executed by repeating this. Specifically, the learning is converged by repeating the calculation of the error in the evaluation function measuring device 213 of FIG. 10 and the calculation of an appropriate correction load in the correction amount calculating device 215 based on the error.

Furthermore, the greatest advantage of using this learning method is that the BP method can be applied without knowing the output of the intermediate layer,
In addition, even if the threshold function is unknown (or cannot be described as a function), the BP method can be applied depending on the output change, so that it is possible to flexibly cope with variations in threshold elements.

[0007]

However, in the above-mentioned conventional neural network apparatus, the evaluation function measuring apparatus 213 and the correction amount calculating apparatus 215 of FIG. 10 are not specifically disclosed. The calculation of the error amount at the time of learning and the correction load calculation need to be performed every time the load changes. Since it is empirically necessary to change the load a few hundred times to a few thousand times for all loads, even if analog / parallelization can be used to speed up the element, the learning time is calculated by error calculation. Had a problem that it could not be shortened too much.

[0008]

In order to solve the above-mentioned problems, the inventors of the present invention, when performing learning by a back propagation method in an analog neural network device, when the load is slightly changed, We propose a circuit that calculates the optimum load value directly from the output change. This arithmetic circuit can directly obtain an appropriate load as the output of the analog circuit after the transient response time by utilizing the gate voltage-drain current characteristic in the non-saturation region of the MOS transistor and the analog of the modified load calculation. .

By applying this circuit, the learning of the analog neural network device can be executed at extremely high speed.

[0010]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a diagram for explaining the configuration of the neural network system of the embodiment. In FIG. 1, reference numeral 1 is a neural network device. 3 is a neuro element. Reference numeral 5 is a memory that stores input / output pattern data. In this embodiment, the input data is stored as digital data as a vector in which the characteristic amount is represented by 0 to 1. An input data holding unit 7 stores an input pattern selected from the input / output pattern data 5. 9 is D
A / A converter, which inputs the digital information of the input data holding unit 7 to the neuro element as an analog output. 1
1 is a D / A converter. An A / D converter 13 converts the analog output of the neuro element 3 into a digital signal. Reference numeral 15 denotes a control unit, which performs input / output pattern selection, and also performs calculations such as changing the synapse load and calculating an output error to determine a new load. 17
Is a cell selection unit, and a predetermined synapse on the neuro element is selected by the control unit 15 to be in a writable state. When a predetermined synapse becomes writable, a write weight value is output from the control unit and transmitted to the neuro element through the D / A converter 11 as an analog value. Reference numeral 19 denotes a memory, which includes a RAM operating portion that holds variables necessary for calculation, a non-volatile RAM portion that stores a weight matrix when learning is completed, and the like. Reference numeral 21 denotes an output, and when the learning is completed by the control unit, the output of the neuro element 3 is output. Reference numeral 23 denotes a learning calculation circuit, which calculates a modified weight value by comparing the output of the neuro element 3 and the teacher signal from the input / output pattern data 5. The output from the learning arithmetic circuit 23 is converted into a digital signal by the A / D converter 13 and sent to the control unit 15.

The neural network system according to the present embodiment is a neural network operation that performs integration, summation,
In addition to the threshold value processing, another feature is that it has a learning calculation circuit that executes the calculation of the correction load amount that is frequently calculated by analog calculation. By providing the dedicated arithmetic circuit in this way, the learning time can be further shortened.

As described above, in this embodiment, the learning arithmetic circuit 2
The learning calculation circuit 23 will be described below because it has the characteristics of 3. As for the neuro element 3, a learning operation circuit described below can be applied as long as it operates to perform analog operation of neural network operation.

First, a BP generally used in the learning method in the NN.
The calculation procedure of the method will be described.

The output value when a certain input value is given to the neuro element 3 is O _k , and the target value is T _k (teacher value). Here, k represents the number of the output vector, and when the number of output terminals is n, k is a natural number of 1 or more and n or less.

At this time, the error E _p is defined by the following equation (1).

[0017]

[Equation 1]

The subscript p represents an input / output pattern number. When there are m patterns of input / output correlation data to be learned, p is a natural number of 1 or more and m or less.

[0019] The change amount of the load of the intermediate layer [Delta] V _kj, in the load amount of change in the output layer and [Delta] W _ji BP method, load correction amount from the E _p, the following equation (2)

[0020]

[Equation 2]

And equation (3)

[0022]

[Equation 3]

Is calculated by Note that j is the neuron number of the middle layer, and i is the input number. Further, α is a learning coefficient, and is set to a numerical value that enables efficient learning. In this embodiment, for example, α = 0.8.

Using the equations (2) and (3), the corrected load value is given by the equation (4).

[0025]

[Equation 4]

And equation (5)

[0027]

[Equation 5]

Is given by

From equations (1) to (5), the learning operation by the BP method can be executed by acquiring and calculating the change in the error when the load is slightly changed.

The correction load can be calculated by the above procedure.

Needless to say, such an arithmetic operation can be performed by forming an analog arithmetic circuit with a summing device, a subtractor, a multiplier, etc. using an operational amplifier or the like. However, if these are simply arranged, there is a problem that a very large number of elements are required and the cost of the device increases.

In the neural network system of this embodiment, the learning operation circuit can be made smaller, and the neural network system can be provided at a lower cost.

FIG. 2 shows an explanatory diagram of the operating principle of the learning operation circuit of the neural network system of this embodiment.

In FIG. 2, reference numeral 25 is a field effect transistor, which is an N-type MOS transistor having Vt of almost 0 [V] in this embodiment. Hereinafter, the field effect transistor is simply referred to as MOS. 27 is a current measuring circuit, one of which is connected to the source electrode of the MOS (25) and the other of which is connected to the other electrode, for example, is grounded. A voltage corresponding to the difference between the teacher signal and the output voltage obtained in the following description is input to the gate electrode of the MOS (25). Similarly, a voltage corresponding to the variation of the output voltage obtained in the following description is input to the drain of the MOS (25).

The present inventors have found that the learning operation based on the above-mentioned very simple circuit configuration can execute the learning operation shown in the equations (1) to (5).

Hereinafter, the reason why the learning operation can be executed by the configuration of FIG. 2 will be described by a concrete operation.

In order to obtain the corrected load value, equation (2),
It is necessary to execute the operation according to (3). The expressions (2) and (3) mean that the load change value is obtained from the change amount of the error when the load value is slightly changed.

Specifically, assuming that the output error when the load is W ₀ is E ₀ and the output error when the load is changed by ΔW to be W ₁ (= W ₀ + ΔW) is E ₁ , the change amount of the error is ΔE (=
From E ₁ −E ₀ ), the corrected load value is calculated by the equation (6).

[0039]

[Equation 6]

And equation (7)

[0041]

[Equation 7]

Is given by

A new load can be calculated by substituting the calculation result of equation (6) or (7) into equation (4) or equation (5).

Here, when the output at the load W ₀ is O _k0 , the output at the load W ₁ is O _k1 , and the target teacher signal is T _k , the change amount ΔE of the error is defined by the expression (E). Given in 8).

[0045]

[Equation 8]

When the relation of O _k1 = O _k0 + ΔO is input into the equation (8) and rearranged, the equation (9) is obtained.

[0047]

[Equation 9]

The present inventors have found that the equation (9) exhibits characteristics very similar to the drain current characteristics in the non-saturated region of the MOS transistor.

Here, the MOS drain current is expressed by the equation (10).
It is represented by.

[0050]

[Equation 10]

Here, β is defined by the equation (11).

[0052]

[Equation 11]

Here, ε ₀ is the dielectric constant of vacuum (= 8.85).
4 · 10 ⁻¹² [F / m]), ε _ox is the relative dielectric constant of the gate insulating film, and μ is the mobility of carriers.

Further, assuming that the threshold voltage of the MOS is so small as to be negligible (up to 0 [V]), the equation (10) can be transformed into the following equation (12).

[0055]

[Equation 12]

Focusing on the similarity between the equations (12) and (9), the voltage calculated by V _G = T _k −O _k0 (Equation 14) V _D = ΔO (Equation 15) It can be understood that, if each is applied to the MOS of FIG. 2, the drain current of the MOS outputs a value proportional to the error change amount.

Although FIG. 2 shows the case where the neural network device has one output, when there are plural outputs, the MOSs may be connected in parallel and the total current I _all may be obtained. In this case, the equation (13) is similarly established.

[0058]

[Equation 13]

As a result, the total error change amount at a plurality of outputs is ΔE = I _all · 2 / β (Equation 16), and the change in error amount can be easily measured on the circuit.

By using such a principle, the learning operation of the analog neural network device can be easily executed.

Equation (10) holds in the range of V _D <V _G -V _t , but since the output variation ΔO is sufficiently smaller than the teacher signal difference (T _k -O _k0 ), In the voltage application method of the invention, this formula holds in most cases. Although _{V D> V G -V t V} D of the saturation region is load correction value is slightly smaller operations than the original value even if they are applied, although the convergence number of times becomes somewhat large, the convergence calculation itself Is capable of performing a learning operation without affecting.

FIG. 3 is a diagram schematically showing the configuration of the learning arithmetic circuit of this embodiment. In FIG. 3, 31 is an output variation calculation unit, and 33 is an error calculation unit.

FIG. 4 shows an example of the configuration of the output variation calculation unit 31 shown in FIG. In FIG. 4, 41 is a switch, NN
Switch the circuit that transmits the output of. Reference numeral 43 denotes an output voltage holding circuit before load change, which receives the output after the load is finely changed by switching the switch 41 and holds the output. Reference numeral 45 denotes a load change output voltage holding circuit, which receives the output after the load is finely changed by switching the switch 41 and holds the output. Reference numeral 47 is a teacher signal holding circuit. A subtraction circuit 1 outputs a difference between outputs of the output voltage holding circuit 43 and the output voltage holding circuit 45 after the load change and before the load change as an output A. Further, 49 is a subtraction circuit 2, which outputs the difference between the output of the teacher signal holding circuit 47 and the output of the pre-load change output voltage holding circuit 13 as an output B. Regarding the voltage holding, it is possible to hold the input voltage and keep outputting it by using, for example, a peak hold circuit of an analog circuit.

FIG. 5 shows an example of the configuration of the error calculator 33 of FIG. In FIG. 5, 51a to 51k are error calculation circuits. Reference numeral 53 is a modified load calculation circuit. The output A and the output B calculated by the output variation calculation unit 31 of FIG. 4 are input to the error calculation circuits 51a to 51k respectively for the outputs of the neuro elements. Further, the error calculation circuits 51a to 51k respectively output currents corresponding to the error change amounts, which are input to the correction load calculation circuit 53, and finally output the correction load value as, for example, a voltage. Thus equation (9)
It is possible to simply calculate the sum of squared error variations according to the above by introducing into the same wiring a current corresponding to the error variation for a plurality of outputs from the neuro element.

FIG. 6 shows an example of the configuration of one error calculation circuit 51 shown in FIG.

In FIG. 6, reference numeral 61 denotes an input terminal 1 to which the output variation before and after the minute load change of the neural network device is input as the output A of the output variation calculation unit 31.
63 is the input terminal 2 and the output B of the output variation calculation unit 31
The difference between the teacher values of the outputs of the neural network device is input as. An amplifier circuit 65 amplifies the voltage of the output A. As an example of such an amplifier circuit, if a non-inverting amplifier using an operational amplifier as shown in FIG. 7 is used,
The output is V _O = V _in · (1 + R ₂ / R) due to the resistances R ₁ and R ₂ .
₁ ) is amplified and output. 66a and 66b are input inversion circuits, which invert the positive and negative of the input signal voltage and output it. As an example of such an input inverting circuit, an inverting amplifier using an operational amplifier as shown in FIG. 8 can be used. Here, if the resistors R ₁ and R _{2 have} the same value, the output V _O can be an output having the same absolute value as the input and the opposite sign. 67a and b are drain selection switches, and in this embodiment, 67a is an NMOS and 67b is a PMOS. The drains of the NMOS 67a and the PMOS 67b are connected to the input terminal 1 (61), and the gates are connected to the output of the amplifier circuit 65. In this embodiment, R ₁ and R _{2 in} FIG.
By setting the value of R _{2 to} be about 10 times larger than R _{1 and} outputting a sufficiently large voltage, the NMOS 67
The voltage drop due to the threshold loss when either a or the PMOS 67b is turned on is suppressed.
Reference numerals 69a to 69d are error calculation elements, which are MOS transistors described in FIG. Error calculation elements 69a, 6
The gate of 9c is directly connected to the input terminal 2 (63), and the gates of the error operation elements 69b and 69d are the input inverting circuit 66.
The output B is input by inverting the positive / negative by a and b. The sources of the error calculation elements 69a to 69d are connected to the common wiring, and the current is input to the correction load calculation circuit 53 of FIG.

In the description of FIG. 2, the basic operation of performing the error calculation by the MOS transistor has been described, but the circuit configuration of FIG. 6 has a circuit configuration for obtaining a more practical operation. That is, the value of ΔO or (T _k −O _k0 ) in the calculation of the equation (9) is not always a positive value, and therefore, FIG.
With the configuration of one transistor as described above, there is a case where an appropriate error calculation cannot be performed in some cases.

The operation of the error calculation circuit 51 of FIG. 6 will be described below. (A) When the input A and the input B are both positive voltages, the voltage of the output A amplified by the amplifier circuit 65 is NMOS.
67a and the gate of PMOS 67b are applied to the NMO
Only S67a is turned on. The meaning of amplifying the voltage of the output A is to suppress the Vt loss in the MOS transistor by inputting a voltage having a sufficiently large absolute value to the gate.

As a result, the voltage of the output A is applied to the drains of the error calculation elements 69a and 69b.

On the other hand, since the output B is positive, only the error calculation element 69a functions as a MOS transistor and outputs a drain current corresponding to an error. No current is output from the other error calculation elements 69b, c, d. (B) When the input A has a positive voltage and the input B has a negative voltage, only the NMOS 67a is turned on, and the error calculation element 6
The voltage of the output A is applied to the drains of 9a and 69b.

On the other hand, since the output B is negative, the error equivalent current is output only from the error calculation element 69a. (C) When the input A has a negative voltage and the input B has a positive voltage, only the PMOS 67b is turned on, and the error calculation element 6
The voltage (negative voltage) of the output A is applied to the drains of 9c and 69d.

On the other hand, since the output B is positive, the error equivalent current (negative) is output only from the error calculation element 69d. (D) When the input A and the input B are both negative voltages, only the PMOS 67b is turned on, and the error calculation element 6
The voltage (negative voltage) of the output A is applied to the drains of 9c and 69d.

On the other hand, since the output B is negative, the error equivalent current (negative) is output only from the error calculation element 69c.

By the above operation, any one of the error operation elements 69a to 69d performs the function operation of the error operation regardless of the polarities of the output A and the output B, and is stable regardless of the signal polarity. Error calculation is possible.

FIG. 9 shows an example of the circuit configuration of the modified load calculation circuit 53 of FIG.

FIG. 9 shows a current-voltage conversion circuit using an operational amplifier, which has the function of converting a current into a voltage by providing the wiring as shown in FIG. The error equivalent current output from the circuit of FIG. 8 is input to the negative terminal of the operational amplifier. On the other hand, if the positive terminal of the operational amplifier is grounded, the output voltage V _{O of the} operational amplifier can be calculated by the following equation using the resistor R shown in the figure.

V _O = −I _in · R (Equation 14) On the other hand, if Equation (13) is transformed, ΔE = 2 · I _all / β (Equation 15) Equation (15) and Equation (4) ), (5), (6), and (7), the load correction amount ΔW _modif is expressed by the following equation.

ΔW _modif = −α · 2 · I _all / β / ΔW = 2 · α · V _O / R / β / Δ W (Equation 16) The load correction amount is _calculated from V _{O according} to Equation (16). Obtainable.

[0079] By setting the value and _{R = 2 · α · V O} / β / ΔW Conversely, ΔW _modif = V _O, and the output V _O can be used as the load correction amount as it is, the more the amount of computation It is possible to reduce.

As described above, the learning operation circuit of the neural network device of this embodiment can perform a complicated operation called an error operation equivalently with a very small circuit scale by utilizing the characteristics of MOS, and output the digital information. It is possible to improve the learning speed in each stage in the case where the arithmetic processing is repeated after the conversion.

Needless to say, this embodiment is not dependent on the shape of the neuro element, and can be applied as a learning circuit for all analog neuro elements whose output changes when the load is changed.

Further, the greatest feature of this embodiment is that the error calculation of the neuro element is executed by applying the characteristic in the non-saturation region of the MOS transistor as described with reference to FIG. Although the circuit for calculating the voltage applied to the gate and the drain of the MOS transistor has been described in this embodiment with some examples, the configuration and operation of these peripheral circuits are not necessarily limited to this configuration and operation. Without limiting in any way the essence of the invention.

[0083]

The analog neural network device of the present invention has a circuit for calculating an optimum load value directly from a change in output when a load is slightly changed when performing learning by the back propagation method. It is equipped with. This arithmetic circuit can directly obtain the appropriate load as the output of the analog circuit after the transient response time by utilizing the analog of the gate voltage-drain current characteristic in the non-saturation region of the MOS transistor and the appropriate load calculation. .

By applying this circuit, the learning of the analog neural network device can be executed at extremely high speed.

[Brief description of drawings]

FIG. 1 is an explanatory diagram of a configuration of a neural network device according to an embodiment of the present invention.

FIG. 2 is an explanatory diagram of an operation principle of a learning operation circuit of the neural network device of the same embodiment.

FIG. 3 is a schematic configuration diagram of a learning arithmetic circuit of the same embodiment.

FIG. 4 is a diagram showing a configuration example of an output variation calculation unit according to the same embodiment.

FIG. 5 is a diagram showing a configuration example of an error calculation unit according to the same embodiment.

FIG. 6 is a diagram showing a configuration example of an error calculation circuit according to the same embodiment.

FIG. 7 is a diagram showing an example of an amplifier circuit of the same embodiment.

FIG. 8 is a diagram showing an example of an input inverting circuit of the same embodiment.

FIG. 9 is a diagram showing an example of a modified load calculation circuit configuration of the same embodiment.

FIG. 10 is a block diagram of a neural network device of a prior art example.

FIG. 11 is an explanatory diagram of a learning operation procedure of the prior art example.

[Explanation of symbols]

1 Neural network device 3 neuro elements 5 memory 7 Input data storage 9 D / A converter 11 D / A converter 13 A / D converter 15 Control unit 17 Cell selection section 19 memory 21 output 23 Learning arithmetic circuit 25 Field effect transistor 27 Current measurement circuit 31 Output variation calculator 33 Error calculator 41 switch 43 Output voltage holding circuit before load change 45 Output voltage holding circuit after load change 47 Teacher signal holding circuit 48 Subtraction circuit 1 49 Subtraction circuit 2 51a to 51k error calculation circuit 53 Corrected load calculation circuit 61 Input terminal 1 63 Input terminal 2 65 amplifier circuit 66a, b input inversion circuit 67a, b Drain selection switch 69a-69d Error calculation element

Claims (7)

[Claims] 1. A neural network before and after a minute change in load.
An output change amount, which is a voltage proportional to an output change of the network device, is applied between the drain and source of the field effect transistor, and a teacher signal difference, which is a difference between the teacher signal before the weight change and the output signal, is applied to the field effect transistor. A learning operation circuit of a neural network device, wherein an error change value is calculated by a drain current flowing when applied to the gate electrode of the. 2. The learning arithmetic circuit of the neural network apparatus according to claim 1, wherein the load is changed to a value proportional to a value obtained by dividing the error change amount by the minute change amount of the load. 3. A learning arithmetic circuit of a neural network device for performing learning by repeating the operation of executing the load changing operation according to claim 2 for all the loads. 4. A learning arithmetic circuit of a neural network device for executing the arithmetic operation according to claim 1 by different field effect transistors according to positive / negative combinations (4 types) of an output change amount and a teacher signal difference. 5. A learning operation circuit for a neural network device, wherein the neural network device has a plurality of outputs, and the learning operation circuit according to claim 1 is provided for each of the plurality of outputs. 6. A pre-load change output voltage holding circuit for holding an output before load change, a post-load change output voltage holding circuit for holding an output after load change, and a teacher signal holding circuit for holding a teacher signal value. A first subtraction circuit for calculating the difference between the output of the output voltage holding circuit after the weight change and the output of the output voltage holding circuit before the weight change, the output of the teacher signal holding circuit and the output voltage hold circuit before the weight change And a second subtraction circuit for calculating the difference between the outputs of the first subtraction circuit, the output of the first subtraction circuit is applied between the drain and the source of the field effect transistor, and the output of the second subtraction circuit is subjected to the field effect. The learning operation circuit of the neural network device according to claim 1, wherein the learning operation circuit is applied to the gate of the type transistor. 7. An output of an amplifier circuit for amplifying an output change amount is a first N-type field effect transistor (NMOS).
Is connected to the gate of a first P-type field effect transistor (PMOS), and the output change amount is equal to the first NMO.
S and the drain of the second PMOS are input, and the source of the first NMOS is the second NMOS and the third NMOS.
Of the second PMOS and the drain of the third PMOS are connected to the source of the first PMOS, the second NMOS, the third NMOS, the second PMOS and the second PMOS. The source of the third PMOS is connected (common source), and the teacher signal difference is the second NMOS.
And a gate of the second PMOS, and an output of an input inverting circuit for inverting the teacher signal difference is connected to the gates of the third NMOS and the third PMOS. A learning operation circuit of a neural network device, which calculates an error change value of a neuro element based on a current value flowing through a wire connected to the.