JPH04216160A - Neural network circuit - Google Patents

Abstract

PURPOSE:To obtain a neural network circuit which constitutes an analog LSI incorporating a circuit for calculating a back propagation method and has such a circuit configuration that the deterioration of learning ability caused by a difference in arithmetic operation resulting from the analog LSI hardly occurs. CONSTITUTION:This neural network circuit is constituted of a neuron circuit 10 and synapse circuits 15a and 15b. The neuron circuit 10 is constituted of nonlinear processing means 12a and 12b which generate transfer functions, a derived function generating means 13, and gain factor setting means 11a and 11b. The synapse circuits 15a and 15b are respectively constituted of coupling strength outputting means 17a and 17a which store, hold, and output synapse coupling strengths for multiplying data processing signals inputted from the circuit 10 and error signals and coupling strength modifying means 16a and 16b which modify the coupling strengths.

Description

[Detailed description of the invention]

0001

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a neural network circuit, and more particularly to a neural network circuit incorporating a backpropagation learning function.

[0002] A neural network circuit is a circuit modeled after a biological neural network. It is expected that by using neural networks, it will be possible to easily realize pattern recognition processes such as character recognition and speech recognition, approximate solutions to optimization problems, and robot control, which were difficult with conventional Neumann computers. There is.

0003

2. Description of the Related Art FIG. 11 shows a block diagram of a unit circuit of the simplest conventional neural network circuit. new
The configuration of the ral network is a combination of a large number of unit circuits in the form of a network. This unit circuit is called a neuron circuit, and one neuron circuit receives signals from multiple input terminals 101, 102, 103, 104, and 105, and assigns a weight (synaptic connection strength) to each input signal. (called ) Wij, add all the results, and perform nonlinear processing on the addition results to determine the output. Therefore, it is characterized by its ability to adapt to processes for which the algorithm is not known in advance.

FIG. 12 shows a conceptual diagram of a three-layer network structure. The network structure is a layered network (also called perceptron type or lamel heart type), with new layers within the layer.
There is no coupling between neurons, neurons between layers are coupled to each other, and signals are transmitted from the input layer 91 through the intermediate layer 93 to the output layer 95 in one direction. The learning model is made to learn by giving a desired output result as a teaching signal to the input data at the same time, so that it can finally output the desired result even in the absence of the teaching signal. This is a so-called supervised learning model, and among these, the error backpropagation learning method (hereinafter referred to as BP method) has a particularly wide range of applications. This method is realized by sequentially changing the connection strength from the output layer to the input layer so that the output value approaches the desired output value with respect to the input value. It is called the BP method because it is performed in the opposite direction to the motion. The specific learning algorithm can be found in the literature; “PARALLEL DISTRIB
UTED PROCESSING” (Parallel Distributed Processing) DE Rum
Elhart, J. L. McClelland,
and the PDP Research Group
p (D.E. Ramelhart, J.L. McCalland and PDP Research Group), MIT Press,
1986, Volume 1, Chapter 8 (P-318~), etc., but only the results will be briefly explained here.

If we define the layer closer to the output layer 95 as the upper layer and the layer closer to the input layer 91 as the lower layer, the following input-output relationship holds true for a certain upper layer neuron i and a lower layer neuron j immediately below.

[0006]

[Math 1]

[0007] oi = f(u i)
...(2)
Here, u i and oi are the internal state and output value of neuron i, respectively, and w ij is the synaptic connection strength. The transfer function f is an S-shaped function whose output value is finite and increases monotonically, and is a nonlinear function called a sigmoid function. Frequently used functions include the logistic function;

[
0008

[Math 2]

There is [0009]. Here, λ is a coefficient representing the gain of the neuron (referred to as gain factor here), and f
(ui) corresponds to the slope of the curve where the value of the sigmoid function changes most drastically.

Next, the learning algorithm is as follows.

[0011]

[Math 3]

δi=ζif′(ui)
...(5)

00
13]

[Math 4]

Δwij=−εδi oj
...(7) Here, E is an evaluation function representing the difference (error) between the output signal and the teacher signal, where oi is the output of the normal output layer neuron and ti is the teacher signal.

[0015]

[Math 5]

[0016] Therefore, in the output layer neuron i, equation (4) becomes ζi = oi −ti
…
(9), and the amount of modification Δwij of the synaptic connection strength with intermediate layer neuron j is given by equation (7). Here, ε is a parameter that determines the amount of correction.

[0017] Below, equation (6), equation (5) and equation (7) are again expressed.
The synaptic strength correction amount is calculated in the lower layers using the following steps: When these calculations are repeated many times for each set of input data and teacher data, the value of the evaluation function E converges to zero. This means that learning has been carried out.

[0018]

[Problem to be solved by the invention] However, until now, the neural network circuit described above has been mainly executed by computer simulation, but there was a problem that learning calculations took time. .

[0019] Therefore, attempts have been made to perform parallel computation using analog circuits, increase integration through LSI, and apply them to various applications. However, since the algorithm of the above-mentioned BP learning method is complicated, there have been no reports of its implementation in LSI.

Furthermore, one of the reasons why it is difficult to calculate the BP method using an analog circuit is that it is difficult to calculate the derivative f' of the transfer function f. Here, if the transfer function f is a logistic function as expressed by the above equation 3, then
The derivative f' is f'(ui)=λf(ui)[1-f(ui
)] ...(10). Formula (10
λf(ui)[1-f(ui)] on the right side of ) is when the transfer function is realized by a differential amplifier, and can be easily realized in terms of a circuit by the product of the inverting and non-inverting outputs of the differential amplifier. However, in reality, the transfer function created by electronic circuits such as differential amplifiers is not an accurate logistic function, so
The calculation result of equation (10) deviates from the correct derivative. Thereby, the right side of equation (10) is called an approximation function. Furthermore, in actual analog circuits, variations in the characteristics of analog elements making up the circuit may cause distortion in the function shape. In particular, calculation errors near the value 0 have a large effect.

Next, the results obtained by computer simulation will be explained. 13 and 14 show simulation results using approximate derivatives. In both figures, (
In A), a is the logistic function expressed by equation (3), and b
~d is a piecewise linear function that typically occurs in differential amplifiers. Using these sigmoid functions, equation (10)
13 using the approximate derivative obtained from the right-hand side λf(ui)[1-f(ui)], and FIG. 14 using the approximate derivative obtained by cutting the tail of the approximate derivative, the input layer - intermediate layer - 3-layer network in the output layer (number of neurons is 2-2-
This shows the percentage of correct learning achieved when exclusive OR learning was performed in step 1). As shown in FIG. 13B, it can be seen that for the function d that is closest to the piecewise linear function, as the learning parameter ε increases beyond 0.04, the learning cannot be performed satisfactorily. Furthermore, in FIG. 14 when the tail of the approximate derivative is cut, it can be seen that the learning ability deteriorates in any transfer function. For example, in the function b shown in FIG. 14(B), the learning parameter ε is 0.
When it reaches 1, the learning ability ratio drops to 0. Furthermore, the ratio of the function d is 0 when the learning parameter is 0.13.

The reason for this is that in learning that outputs a saturation value of 0.1, such as exclusive OR, it is necessary to train neurons in the middle layer to operate in the saturation region. This is because if the tail is cut, the value of the derivative becomes 0 before the operation begins, and learning will not proceed.

Deterioration of the learning ability caused by the inability of analog circuits to generate accurate derivatives is also a problem in LSI implementation of neural networks.

[0024] The present invention has been made in view of the above points.
It is an object of the present invention to provide a neural network circuit having a circuit configuration in which an analog LSI incorporating a circuit for calculating the BP method can be constructed, and a calculation error occurring in the analog LSI does not affect learning ability.

[0025]

[Means for Solving the Problems] FIG. 1 shows a diagram of the basic configuration of the present invention. In a layered neural network circuit in which there is no connection between neurons within a layer and neurons between layers are mutually connected, the neural network circuit has a synapse with the neuron circuit 10. The neuron circuit 10 has circuits 15a and 15b, and the neuron circuit 10 includes nonlinear processing means 12a . 12b, derivative generating means 13 for generating a derivative, and gain factor setting means 11a for independently setting a coefficient representing the gain of the neuron.
11b, and an error signal output means 14 which multiplies the error signal propagated from the upper neuron group 19 by a value generated by the derivative generation means and outputs the multiplied value signal in the opposite direction, and synapse circuits 15a and 15b. is a connection strength modifying means 16a that holds the synaptic connection strength, corrects the signal supplied from the neuron circuit 10 by multiplying it by the connection strength, and outputs the result.
, 16b, and coupling strength output means 17a, 17b for multiplying the error signal by the coupling strength and executing the error signal output means 14.
becomes more

[0026]

[Operation] In the transfer function and derivative generation means in the neuron circuit, the gain factor of the derivative is changed to the transfer function gain using the nonlinear processing means and the derivative generation means that can independently set the gain factors of both. This can be made smaller than the factor, thereby preventing deterioration of learning ability due to calculation errors in analog circuits. In addition, by combining a neuron circuit and a synapse circuit that process data signals going from the lower layer of the neuron circuit to the upper layer, and also have an error signal output means that processes error signals going from the upper layer to the lower layer, BP suitable for LSI
A neural network circuit with a built-in learning function can be created.

[0027]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 2 shows the results of a simulation in which the gain factor of the approximate derivative with its tail cut is fixed and the gain factor of the transfer function is varied. This method avoids deterioration of learning ability due to calculation errors that occur in analog circuits. Comparing this figure with Figures 13 and 14, when the gain factor of the transfer function was smaller than the gain factor of the derivative, no learning was possible regardless of the learning parameter ε, but the gain of the transfer function was the gain of the derivative. If it is larger than the factor, learning was successful as long as the learning parameter ε was small. This means that the shape of the derivative may be somewhat distorted because the transfer function is not a logistic function, or even if the calculation precision near the tail is poor, the transfer function may have a gain factor larger than the gain factor of the derivative. If used, deterioration of learning ability can be avoided. A method for avoiding deterioration of learning ability will be described below.

FIG. 3 shows a circuit diagram of a differential amplifier. In the figure, voltage Vb121 and voltage Vb222 are bias voltages for performing appropriate analog operations. Further, the input wirings 25 and 26 and the output wirings 23 and 24 are a pair of wirings, and are of a differential type that is resistant to noise and fluctuations in the reference level.

FIG. 4 shows a circuit diagram of a multiplication circuit. In this figure, as in FIG. 3, the voltage Vb131 and the voltage Vb232 are bias voltages. In addition, input wiring 33, 34, 35
, 36 and the output wirings 37 and 38 form a pair of wirings. Both the differential amplifier in FIG. 3 and the multiplication circuit in FIG. 4 have voltage input and current output.

FIG. 5 shows an outline of an embodiment of a neuron circuit which is a main part of the present invention. The neuron circuit 40 is composed of multipliers 41, 44, 46, 47 and differential amplifiers 45, 48. Further, there is a derivative generating section 42 that generates a derivative f', which is composed of two multipliers 42 and 46 and a differential amplifier 45, and a transfer function f, which is composed of a multiplier 47 and a differential amplifier 48. There is a transfer function generation section 43. In the figure, for simplicity, a pair of wires in the differential circuit is shown as a single line; however, as an exception, the neuron circuit 40
Only the output of the differential amplifier 45 of the derivative generator 42 which generates the derivative f' of the transfer function at the center of is shown as a pair.

In the figure, the output value ui of neuron i determined by the above equation (1) is input to multipliers 44 and 47. When inputted to the multiplier 47 of the transfer function generation section 43, oi=f(ui) is output. On the other hand, when input to the multiplier 44 of the derivative generation component 42, the above formula (10
) is executed, and the derivative f' is input to the multiplier 41 together with the error signal ζi to calculate the above equation (5),
An error signal δi is output.

FIG. 6 shows an outline of an embodiment of a synapse circuit which is another essential part of the present invention. The synaptic circuit 50 includes three multipliers 51, 52, and 54, and a synaptic connection strength adjustment circuit 53 that stores and modifies the synaptic connection strength.

In the figure, a synaptic connection strength adjustment circuit 53 holds an analog voltage proportional to the synaptic connection strength and has a function of constantly outputting it and a function of increasing or decreasing the synaptic connection strength in proportion to an external learning signal. This is an EEPROM
(Electrically erasable/pr
ogramable read-only memory
This can be realized by combining a control circuit with a non-volatile memory such as y). A reference document is "Analog Operation of Floating Gate MOS" by Fujita, Amemiya, and Iwata, Proceedings of the 1990 IEICE Autumn National Conference, page 5-224. Furthermore, when the memory is not non-volatile and only temporary storage is required, or when a refresh circuit is provided, a method of storing charge in a MOS capacitor is also used.

In the figure, the output value oj from the lower neuron circuit j and the output value from the synaptic connection strength adjustment circuit 53 are input to a multiplier 51 as input signals. The multiplier 51 multiplies the two input values and outputs the calculation result wijoj to the next stage. Furthermore, the multiplier 52 receives as input signals oj, which is the output value from the lower neuron circuit j, and δi, which is the error signal from the upper neuron i, and these two values are multiplied. The calculation result is output to the synaptic connection strength adjustment circuit 53. The synaptic connection strength adjustment circuit 53 stores the synaptic connection strength based on the input value, performs correction by executing equation (7) based on the input value, and outputs the value to the multipliers 51 and 54. Further, an error signal δi is also input to the multiplier 54. As a result, the multiplier 54 becomes 2
The calculation result wijδi is output, and the term related to wij in equation (6) is calculated.

FIG. 7 shows a block diagram of a circuit according to an embodiment of the present invention. This figure explains the flow of signals from input signals being supplied to input terminals 61 and 62 to being output from output terminals 63 and 64.

Signals ui from an external circuit inputted to input terminals 61 and 62 are supplied to neuron circuits Ni1 and Ni2 as input layers, respectively. Neuron circuit Ni
1 and Ni2 output the oi determined by equation (2) to the synaptic circuits S1, S3 and S2, S4, respectively. Synaptic circuits S1, S3, S2, S4
converts the error signal wijδi to the preceding neuron circuit Ni
1, Ni2 and neuron circuit Ni1,N
The signal oj from i2 is multiplied by the coupling strength wij
Neuron circuit Nh1,N with ijoi as the middle layer
Output to h2.

The neuron circuits Nh1 and Nh2 are expressed by the formula (
The neuron circuit Nh1, Nh is added to ζi obtained in 6).
2 is multiplied by the derivative f' found in 2, executes equation (5) for output in the opposite direction, and outputs the calculation result δi to the previous stage synapse circuits S1, S2, S3, S4 and the neuron output. oi is input to the next-stage synapse circuits S5, S6, S7, and S8, respectively. The synapse circuits S5, S6, S7, and S8 output the value of wijδi to the neuron circuits Nh1 and Nh2 in the previous stage, and the calculation obtained by multiplying the signal oj from the neuron circuits Nh1 and Nh2 by the coupling strength. Result wijoj
is output to neuron circuits No. 1 and No. 2 as output layers.

[0038] Neuron circuits No. 1 and No. 2 are expressed by formula (9)
By multiplying the calculation result ζi by the derivative f′ found inside the neuron circuit, the calculation result δi of equation (5) is obtained.
It outputs the neuron output oi to the lower layer. This output is also input to differential amplifiers 65 and 66, respectively. The differential amplifiers 65 and 66 each receive a teacher signal t1.
(desired value) is input, formula (9) is performed, and the calculation result ζi is returned to neuron circuits No. 1 and No. 2.

wijoi and wijoj calculated in each of the above synaptic circuits S1 to S8 are current amounts. This value is added by an external resistor to obtain the result of equation (6), which is input to the next neuron circuit. The functions of the neuron circuit and synaptic circuit are as described above, but the formula (
The function of calculating the sum in 1) and (6) is obtained by the sum of currents according to Kirchhoff's law, and is detected by the voltage generated across the resistor shown in FIG. Further, each resistor is fixed to a reference potential for analog operation.

FIG. 8 shows a detailed circuit diagram of a neuron circuit according to an embodiment of the present invention. The configuration in this figure is the same as that in FIG. 5, so the same components are given the same reference numerals. Terminals 71-7
The voltage input to terminal 7 is a reference voltage VNN that provides a neutral point, and the voltage input to terminal 82 is power supply voltage VDD. Voltage VSF input to terminals 78, 79, 80, 81,
VSF' is a multiplication voltage for setting the gain factor of the transfer function and the derivative of the transfer function, respectively. In the case of this embodiment, by setting VSF>VSF', the gain factor of the transfer function can be made larger than the gain factor of the derivative. In the figure, a current mirror circuit 82 modifies the waveform of the derivative of the generated transfer function.

Next, the operation in the figure will be explained. First of all,
The multiplier 47 of the neuron circuit 40 is supplied with the multiplication voltage VSF from terminals 80 and 81, and also receives the output signal ui from the lower neuron group. neuron input u
i is supplied to the multiplier 47 and also to the multiplier 44 .

The multiplier 47 multiplies the input signals and outputs the multiplied signals to the differential amplifier 48. At this time, terminals 75 and 76
The reference voltage VNN is supplied from the reference voltage VNN. This allows multiplier 4
7 and the differential amplifier 48 to generate a transfer function f, and formulate the equation (2
), the synaptic circuit 5 is used as the neuron output oi.
A circuit 43 that outputs 0 is configured.

Next, the output signal ui from the lower neuron group is input to the multiplier 44, and a multiplication voltage VSF' which determines the gain factor of the derivative is supplied from terminals 78 and 79. The multiplier 44 multiplies the input signals and outputs the result to the differential amplifier 48. At this time, the reference voltage VNN is supplied from the terminals 71 and 72. The differential amplifier 48 is an inverting
A non-inverted output is output to multiplier 46. The multiplier 46 executes equation (10) from the input value and converts the calculation result into a derivative f'
shall be. These multipliers 44, 46 and differential amplifier 45 constitute a derivative generating section 42.

However, in the circuit shown in FIG. 8, since the reference voltage input to the multiplier 46 is set to ground, an offset occurs in the differential output. For this reason, only one output is used to generate a signal that sets the reference voltage VNN to 0 level via the current mirror circuit 83 and is input to x+ of the multiplier 41. At this time, the other input terminal x- of the multiplier 41 has VN
Add N.

The multiplier 41 receives an error signal ζi expressed by equation (4) or (6) propagated from the upper layer neuron group. The multiplier 41 applies a derivative f to this error signal ζi.
', and calculate the calculation result δi in the opposite direction.
Output.

FIG. 10 shows a detailed circuit diagram of a synapse circuit according to an embodiment of the present invention. Since this figure has the same configuration as FIG. 6, the same reference numerals are given. First, the multiplier 51 of the synapse circuit 50 is supplied with the output signal oi of the neuron circuit 40 and the synapse strength wij. The multiplier 51 executes equation (1) for multiplying oi by the synaptic strength wij and outputs the result.

Next, the output signal oi of the neuron circuit 40
is also supplied to multiplier 52. Further, the multiplier 52 receives the signal δi input from the neuron circuit, executes equation (7), and calculates the correction amount Δwij of the synaptic connection strength.
The signal is output to the synaptic connection strength adjustment circuit 53 as a signal.

The synaptic connection strength adjustment circuit 53 generates a learning signal wij for changing the synaptic connection strength, and supplies it to the multiplier 54 and the multiplier 51 described above. The multiplier 54 uses δi supplied from the neuron circuit 40 and the synaptic connection strength wi of the signal supplied from the synaptic connection strength adjustment circuit 53.
Multiply by j and send to the previous stage.

FIG. 9 shows the results of a simulation of a neuron circuit according to an embodiment of the present invention. This figure shows the input/output relationship when a circuit simulation is performed by setting appropriate parameters for the above-mentioned neuron circuit. The multiplication voltage VSF that determines the gain of the transfer function is set to 0.
5V, and the multiplication voltage VSF' which determines the gain of the derivative f' of the transfer function f is set to 0.3V. The value of the error signal ζi input to the neuron circuit is 0.
2V, 0.3V, and 0.4V, and the output signal oi output from the neuron circuit in those cases is shown.

In this example, the circuit was constructed using only a differential amplifier and a multiplier to create a variable gain amplifier, but the circuit was constructed with the aim of reducing the number of elements and the area occupied by the circuit. This simplification is easily accomplished by those skilled in the art having ordinary circuit design skills.

[0051]

As described above, according to the present invention, by having a circuit in the neuron circuit that can independently set the gain factors of the transfer function and the derivative, the gain factor of the derivative can be set as the gain factor of the transfer function. This can prevent the learning ability from deteriorating due to calculation errors in analog circuits.

Furthermore, by combining a neuron circuit and a sinus circuit that simultaneously perform data signal processing and error signal processing, a neural network circuit with a built-in learning function of the BP method suitable for LSI implementation can be constructed. This allows for higher integration.

[Brief explanation of the drawing]

FIG. 1 is a diagram showing the principle configuration of the present invention.

FIG. 2 is a diagram showing simulation results when the gain factor of the approximate derivative is fixed and the gain factor of the derivative is varied.

FIG. 3 is a circuit diagram of a differential amplifier.

FIG. 4 is a circuit diagram of a multiplication circuit.

FIG. 5 is a diagram showing an embodiment of a neuron circuit configuration, which is a main part of the present invention.

FIG. 6 is a diagram showing an example of a synapse circuit configuration, which is another essential part of the present invention.

FIG. 7 is a block diagram of a circuit according to an embodiment of the present invention.

FIG. 8 is a detailed circuit diagram of a neuron circuit according to an embodiment of the present invention.

FIG. 9 is a detailed circuit diagram of a synapse circuit according to an embodiment of the present invention.

FIG. 10 is a diagram showing the results of a simulation of a neuron circuit according to an embodiment of the present invention.

[Figure 11] The simplest example of a conventional neural network
FIG. 2 is a configuration diagram of a circuit.

FIG. 12 is a conceptual diagram of a three-layer network structure of a neural network.

FIG. 13 is a diagram showing simulation results when approximate derivatives are used.

FIG. 14 is a diagram showing simulation results using approximate derivatives.

[Explanation of symbols]

11a, 11b gain factor setting means 12a, 1
2b Nonlinear processing means 13 Derivative generation means 14 Error signal output means 10, 40 Neuron circuits 15, 50 Synapse circuits 16a, 16b Connection strength output means 17a, 17b Connection strength correction means 18 Lower layer neuron group 19 Upper layer neurons Ron group 41, 44, 46, 51, 52, 54 Multiplier 42
Derivative function generation section 43 Transfer function generation section 45, 48 Differential amplifier

Claims (1)

[Claims] 1. In a layered neural network circuit in which there is no connection between neurons within a layer and neurons between layers are mutually connected, the neural network
The circuit has a neuron circuit and a synapse circuit, and the neuron circuit includes a nonlinear processing means that performs nonlinear processing expressed by a transfer function on the signal supplied from the lower neuron group, and a nonlinear processing means that generates a derivative. gain factor setting means for independently setting a coefficient representing the gain of the neuron; and multiplying the error signal propagating from the upper neuron group by the value generated by the derivative generation means. and serves as an error signal output means for outputting a signal of the multiplied value in the opposite direction,
The synapse circuit holds the synaptic connection strength, and includes a connection strength correction means for multiplying and correcting the signal supplied from the neuron circuit by the connection strength, and outputting the signal; A neural network circuit comprising: coupling strength output means for executing error signal output means.