JPH06131482A - Analog neural network circuit - Google Patents

Abstract

PURPOSE:To provide the learning function incorporated type neural network circuit for evading incomplete learning due to the offset error of the circuit. CONSTITUTION:A process for learning a learning pattern vector includes a process wherein a specific learning rule is performed by using a tutor pattern vector as an error signal to calculate a 1st coupling intensity variation quantity, a process wherein a learning rule is executed by using a network output pattern vector as an error signal to calculate a 2nd coupling intensity variation quantity, and a process wherein coupling intensity is updated with a quantity proportional to the difference between the 1st and 2nd coupling intensity variation quantities; and those processes are repeated as to plural learning patterns to perform learning operation. Namely, a tutor signal and the output of a neuron are sent back to the network while switched by a switching terminal 100.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an analog neural network circuit, and more particularly to an analog neural network circuit having a so-called error correction type learning function for performing learning so as to reduce a difference between an actual output and a desired output. .

[0002]

2. Description of the Related Art A neural network circuit is a circuit that models a neural network of a living organism. In recent years, pattern recognition processing such as character recognition and voice recognition, which has been difficult with conventional Neumann computers, and approximation of optimization problems. It is expected to be effective for solution and robot control.

One of the important functions of neural networks is to automatically establish relationships between given data by learning. Therefore, it has an adaptive ability even for a process whose algorithm is unknown.

Although there are various kinds of learning methods, the error correction learning method is one that has been widely studied. This is a learning method that changes the coupling strength to correct the output of the neural network and the correct answer, the teacher signal, when they are different. Above all,
The “error back-propagation learning method” (also referred to as backpropagation method, hereinafter abbreviated as BP method) has high capability and various applications have been studied. This will be briefly described below.

The structure of the network is a layered network in which there is no connection between neurons in the layers, connections are made between neurons in layers, and signals are transmitted in one way from the input layer to the output layer via the intermediate layer. .

FIG. 4 shows a conceptual diagram of a three-layer network structure. For the BP method, refer to the document: "PARALLEL DISTRIBUTED P
ROCESSING ”(Parallel Distributed
Processing) DE Rumelhart, JLMcClelland, and t
he PDP Research Group (DE Ramelhart, JL McCarland and PDP Research Group), MIT Press, 1st 1986
Volume 8 (page 318-) and the like are described in detail, but only the results will be briefly described here.

When the layer closer to the output layer is defined as the upper layer and the layer closer to the input layer is defined as the lower layer, the following input / output relationship is established for a certain upper layer neuron i and a lower layer neuron j immediately below.

[Equation 1]Where u_iAnd o_iIs the internal state of each neuron i
State and output value, w _ijIs the synaptic bond strength.
The transfer function f is an S-shaped function whose output value is finite and monotonically increasing.
And a logistic function is often used;

[Equation 2] There is. Here, λ is a coefficient that represents the gain of the neuron.

Next, the learning algorithm is as follows. The evaluation function E for evaluating the progress of learning is expressed by the difference (error) between the output signal and the teacher signal, where the output of the normal output layer neuron is o _i and the teacher signal is t _i.

[Equation 3] And Learning is a process of changing the coupling strength w _ij so that the evaluation function becomes smaller. In normal BP learning, a method called steepest descent method is adopted, and the change amount of the bond strength is

[Equation 4] It is represented by. Here, ε is a learning parameter that determines the correction amount. Rewriting this with the differential chain rule,

[Equation 5] Becomes

Since the expression (6) in the output layer neuron i is ζ _i = t _i −o _i (10), the correction amount Δw _ij of the synaptic connection strength with the intermediate layer neuron j is expressed by the expression (9). Given in.

Thereafter, again using equations (8), (7) and (9), the synaptic strength correction amount is calculated toward the lower layer. In this way, the learning signal propagates in the direction opposite to the signal to be originally processed. Therefore, ζ is hereinafter referred to as a back propagation signal.

When the above calculation is repeated many times for each set of input data and teacher data, learning progresses so that the output error shown in equation (4) becomes smaller.

One method for performing BP learning on an analog LSI is described in the literature: "T. Morie, O. Fujita, and Y. Ame.
miya, “Analog VLSI Implementation of Adaptive Algo
rithms by an Extended Hebbian Synapse Circuit, ”I
EICE Trans. Electron., Vol. E75-C, pp. 303-311, 19
92 "or Japanese Patent Application No. 2-402928.

FIG. 5 shows a conventional BP learning built-in type analog neural network circuit configuration. Fig (A) shows the 2-2-2BP nets constituted by synapse blocks S _ij of the structure of BP neuron blocks N _i and drawing of the configuration of FIG. (B) (C). The BP neuron block N _{i in} FIG. 9B has multipliers M1 to M6.
And a differential amplifier A1 and a derivative function generation circuit 30. The amount obtained by the equation (1) is input to the multipliers M2 and M3. The signal input to the multiplier M3 is o _j = f (u
_i )) and output. On the other hand, the signal input to the multiplier M2 is passed through the derivative generating circuit 30 and the error signal δ
Output _i .

The synapse block S shown in FIG.
_ij is a multiplier M4 to M6 and a coupling strength control circuit (hereinafter W
(Abbreviated as PU) 31. In the synapse block, the output values o _j and W from the lower neuron circuit j
The output value w _ij from the PU 31 is input to the multiplier M6, and the multiplication result w _ij o _j is output to the upper layer neuron block. The error signal [delta] _i from the multiplier M5, which is the output value from the lower layer neuron j as input signals o _j and the upper layer neuron i is input, the two values are multiplied. As a result, Δw _ij becomes an input of WPU 31. W
The PU 31 stores the synaptic bond strength and corrects the bond strength based on the input bond strength correction signal,
The value is output to the multipliers M4 and M6. The multiplier M4 multiplies the error signal δ _i and Δw _ij and outputs w _ij δ _i .

FIG. 6 shows the configuration of the coupling strength control circuit (WPU) shown in FIG. 5 (C). In addition, FIG. 7 shows FIG.
The differential amplifier A1 and the multipliers M1 to M1 used in FIG.
The structure of M6 is shown. The differential amplifier in FIG. 7A is a differential amplifier that is resistant to noise and fluctuations in the reference level, and a bias voltage for performing an appropriate analog operation is applied.
The multiplier shown in FIG. 7B is a voltage input / current output differential four-quadrant multiplier to which an appropriate bias voltage is applied.

FIG. 8 shows the derivative function generating circuit 3 shown in FIG.
It is a circuit diagram of 0. The derivative generating circuit 30 obtains a derivative f ′ of the transfer function f used in the equation (7). this is,
From equation (3)

[Equation 6] Therefore, the square operation circuit using the multiplier can be calculated with a slight modification.

The operating principle of the coupling strength control circuit WPU (hereinafter referred to as WPU circuit) used in FIG. 5C will be described with reference to FIG.

In the voltage-pulse conversion circuit 10 of the WPU circuit, the coupling strength correction signal Δw _ij which is an analog signal inputted from the input terminals 13 and 14 is compared with the comparison signal inputted from the comparison voltage terminal 15 having a triangular waveform. By converting into a plurality of pulse signals having a pulse width proportional to the input signal voltage, the charge injection / extraction to / from the charge storage element such as a floating gate element or a capacitor, which is a storage element, is performed based on this. To do. Memory circuit 11
Then, the non-linearity (hysteresis) of the writing characteristics of the memory element
To alleviate the problem, a feedback circuit is added so that a constant write bias is always applied to the storage element. The coupling strength w _ij is read and output as a voltage proportional to this charge amount. The reason for writing with a pulse signal is that writing accuracy (linearity) is improved as compared with the case of performing writing with an analog voltage. Regarding the initialization of the coupling strength, the coupling strength can be made almost zero by inverting and coupling the input and output of the WPU circuit.

[0019]

Generally, when BP learning is performed by an analog circuit, there is a problem that it is difficult to obtain sufficient calculation accuracy. In the case of a digital circuit, the calculation is a bit-precision fixed-point operation that is determined by the circuit, so it is easy to simulate the effect on the learning ability, but in the case of an analog circuit, various factors must be taken into consideration, so much Not considered. Although the result of our study is described in the above-mentioned document, it is the time-constant offset error added in the learning process that has an extremely large effect on the BP learning. This is done by setting equations (8) and (9) as follows:

[Equation 7]

In the example of FIG. 5, this corresponds to the offset generated in the multipliers M4 and M5. Although both influence learning to the same extent, the simulation result of the rate of successful learning of the multiplier M5 is shown.

FIG. 9 is a simulation result showing the effect of the offset error in the multiplier M5 on the learning success rate in the conventional BP learning circuit. This figure shows the learning success rate when an arbitrary amount of offset whose upper limit is defined by the maximum offset amount is randomly incorporated in the multiplier M5 in the conventional BP learning circuit. The bond strength is [-1,
1], the offset amount is standardized in this sense. This simulation result is 2-2
This is the result of learning the relation of exclusive OR with the three-layer network of -1, and the learning parameter in that case is ε = 0.0.
It is 1.

From the figure, it can be seen that the learning success rate is significantly reduced when there is an offset of about 0.5% or more compared with the neuron state or the saturation value of the coupling strength. The reason why the learning ability deteriorates due to the offset error is
In addition to the amount of change in the coupling strength according to the learning rule, a constant change due to the offset error is always added, so that learning proceeds in the wrong direction, and even if there is no error between the teacher signal and the network output, the change due to the offset Because I will proceed
It means that the correct convergence point is not reached.

Although the influence of the offset on the BP learning has been described above, this is a problem that generally occurs in the error correction learning method that minimizes the difference between the teacher signal and the network output. Such an offset occurs due to parameter variations in the LSI manufacturing process, and even if an accurately made differential circuit is used, there is an offset of about 10 mV and it cannot be completely removed.

Further, since the neural network LSI requires a high degree of integration, it is difficult to adopt a highly accurate circuit having a large number of elements, and a simplified circuit is often used. The amount increases. Therefore, if BP learning is performed by an analog circuit, the efficiency will be considerably reduced. Further, when the differential amplifier as shown in FIG. 7A is adopted, the number of elements is nearly doubled, so that not only the circuit area, the amount of wiring, and the number of necessary package pins are increased, but also a constant current source is used. There is a problem that high integration is difficult because the power consumption increases due to the use.

The present invention has been made in view of the above points,
It is an object of the present invention to solve the above problems and to provide an analog neural network circuit for avoiding incomplete learning due to the offset error of the circuit in the learning method of the neural circuit.

[0026]

SUMMARY OF THE INVENTION An analog neural network circuit of the present invention uses a teacher pattern vector and an input pattern in order to learn a plurality of learning pattern vectors consisting of a predetermined input pattern vector and a teacher pattern vector corresponding thereto. Learning a learning pattern vector in an analog neural network circuit that incorporates a circuit that executes a predetermined learning rule that updates the coupling strength in the direction that minimizes the error signal defined by the difference between the vector and the network output pattern vector The learning process for performing a predetermined learning rule using the teacher pattern vector as an error signal, calculating the first coupling strength change amount, and the learning rule using the network output pattern vector as an error signal, Calculating the amount of change in bond strength of And a process of updating the beauty bond strength in an amount proportional to the difference between the second coupling strength change amount includes circuitry for performing learning by repeating the series of learning process for a plurality of learning patterns.

Further, in the analog neural network circuit of the present invention, the learning executes a predetermined learning rule by using the teacher pattern vector as an error signal, calculates the first coupling strength change amount, and calculates the first coupling strength change amount. A process in which the coupling strength is updated by a proportional amount and a learning rule is executed by using the network output pattern vector as an error signal to calculate the second coupling strength change amount, and the sign of the second coupling strength change amount is changed. A circuit for performing the process of updating the bond strength by an amount proportional to.

[0028]

In the analog neural network circuit of the present invention, one learning is divided into two phases,
In each of them, a teacher signal and a network output signal are given as learning signals (counterpropagation signals). The bond strength is changed in proportion to the difference between the changes in the two phases. In this method, the offset error that is constant in each phase is canceled when the difference is obtained, and the offset error has no effect on learning. further,
If the learning parameter is made sufficiently small, the learning will be performed correctly even if the coupling strength is changed for each phase instead of changing the coupling strength according to the difference between the phases. In this case, since it is not necessary to store the coupling strength change amount in each phase, it is advantageous in hardware implementation. The above learning method can be considered as an application of Boltzmann machine learning in which “learning” and “anti-learning” are repeated to error correction learning.

In the learning sequence presented by the present invention, the learning signal in the unit to which the teacher signal is given is given by the linear combination of the teacher signal and the network output, and the change amount for each coupling strength is linearly combined with the learning signal. It is generally applicable to learning rules such as those given by.

According to the present invention, the error learning is independently learned for each of the teacher signal and the network output signal, and the learning is advanced by the difference between them. Therefore, the offset error is inevitably incorporated in the analog circuit and causes a remarkable deterioration of the learning ability. The effect of can be completely canceled. Therefore, a learning method that requires high accuracy, such as BP learning, can be correctly executed even in an analog LSI having low calculation accuracy.

[0031]

Embodiments of the present invention will be described below with reference to the BP learning described in the section of the prior art. In the learning method according to the present invention,
As equation (10), a phase for performing learning as ζ _Li = t _i (13) (following learning of Boltzmann machine is called a learning period, a subscript of L is added), and ζ _Ui = o _i (14) One set of learning phase (called anti-learning period, suffixed with U. At this time, the amount of change in the bond strength is the one with the sign reversed) is set as one learning process and repeated. Then, proceed with learning. This coincides with the original BP learning because the equations (7) to (9) are linear with respect to the backpropagation signal ζ. In each phase (9)
Change amounts Δw _Lij and Δw _Uij of the bond strength corresponding to the equation are calculated, and the bond strength is changed according to Δw _ij = ε (Δw _Lij −Δw _Uij ) (15).

Strictly speaking, it is necessary to calculate the amount of change in the bond strength in each phase as in the above equation, and then change the actual bond strength by an amount proportional to the difference between the two. If the learning parameter ε is sufficiently small, the bond strength is changed in each phase (that is, the bond strength is changed by εΔw _Lij in the learning period, and −ε in the anti-learning period).
Learning is correctly performed by changing the coupling strength by Δw _Uij ). Like this method, the method of sequentially calculating the bond strength change amount and immediately changing the bond strength is called "online learning", and it does not require an extra memory to store the change amount. It is advantageous for hardware implementation.

According to the above learning method, (11) to (1)
It can be seen that the influence of the offset error shown in the equation (2) is canceled out and is not reflected in the learning result.

Next, the outline of the circuit configuration of the analog circuit for executing the learning method of the present invention and the learning sequence will be described.

FIG. 1 shows a block diagram of an analog neural network circuit with built-in BP learning according to an embodiment of the present invention, and FIG. 2 shows a timing chart for explaining the learning sequence in FIG. This embodiment uses almost the same circuit block and circuit architecture as the conventional example shown in FIG. 2A shows an input signal input to the neuron block N ₁ , and FIG. 2B shows a teacher signal input terminal 103.
, A clock signal for switching the backpropagation signal is shown in FIG. 6C, and a polarity inversion clock φ2 applied to the polarity inversion terminal 101 of the WPU is shown in FIG.

The neural network circuit of FIG. 1 is shown in FIG.
The point different from the circuit of (1) is that the above-mentioned learning method is executed by synchronously switching the polarity inversion and the back propagation signal input section in the coupling strength changing circuit for each learning pattern.

That is, in the prior art of FIG. 5, the difference between the teacher signal and the output of the output neuron was calculated by the differential amplifier and sent back to the network as a backpropagation signal, but in the embodiment of FIG. 1, the teacher signal and the output. Each signal is sent back to the network while switching between the output of the neuron and the reverse error propagation signal switching terminal 100. The timing is the clock signal φ1 for switching the backward propagation signal in FIG.
However, while presenting one learning pattern, the teacher signal and the network output are switched so as to backpropagate for the same time. At the same time, the polarity reversal clock φ
2 is applied to the polarity reversal terminal 101 (FIG. 6) of the WPU.

In the coupling strength control circuit (WPU), the coupling strength correction signal Δw _ij inputted as an analog signal is converted into a pulse signal as described in the section of the prior art. It is possible to insert a summing circuit to invert the coupling strength correction signal. That is, FIG.
As is clear from the circuit diagram, the polarity reversal terminal 101 indicated by is either the coupling strength correction signal Δw _ij input to the WPU circuit of the synapse block as the High or Low is input to the terminal, or the sign The bond strength is changed in proportion to the amount of reversal. In the conventional BP learning, it is not necessary to invert the correction signal, so the polarity inversion terminal 101
Was always fixed at High.

FIG. 3 shows a simulation result showing the effect of the offset error in the multiplier M5 on the learning success rate when the learning method of the present invention is used. The conditions of the simulation shown in the same figure are the same as those of the multiplier M5 described in the section of the prior art (FIG. 9). As can be seen from the figure, the effect of the offset error is canceled.

[0040]

As described above, the BP described in the prior art is used.
In the case of applying the present invention to an analog neural network circuit with a built-in learning function, the changes in the circuit configuration are extremely small, and only a clock operation is added to the learning operation sequence, but it is unavoidable in the analog circuit. The effect of being incorporated and capable of completely eliminating the influence of the offset error, which causes a significant deterioration of the learning ability, is extremely large.

Since it is no longer necessary to consider the problem of offset, it becomes possible to operate the differential circuit in the subthreshold region where the operation variation is large, and it is possible to make a highly integrated analog LSI with low power consumption. Become.

Furthermore, it is not always necessary to employ a differential circuit having a large number of elements, and a more highly integrated analog neural network LSI can be manufactured. In addition, this circuit configuration can be effectively adopted even in an optical circuit such as a differential circuit which is difficult to take, and it becomes possible to develop a highly functional and highly integrated optical neural network having a built-in learning function.

[Brief description of drawings]

FIG. 1 is a block diagram of an analog neural network circuit with built-in BP learning according to an embodiment of the present invention.

FIG. 2 is a timing chart showing a learning sequence in FIG.

FIG. 3 is a diagram showing a simulation result showing an influence of an offset error in the multiplier M5 (FIG. 5 (C)) on the learning success rate when the learning method of the present invention is used.

FIG. 4 is a diagram showing a concept of a neural network (3
Layer network structure).

FIG. 5 is a diagram showing a configuration of a conventional analog neural network with built-in BP learning.

6 is a configuration diagram of the coupling strength circuit (WPU) of FIG. 5 (C).

FIG. 7 is a circuit configuration diagram of a differential amplifier and a multiplier used in FIGS. 5B and 5C.

FIG. 8 is a circuit diagram of the derivative generation circuit of FIG.

FIG. 9 is a diagram showing a simulation result showing an influence of an offset error in a multiplier M5 on a learning success rate in a conventional BP learning circuit.

[Explanation of symbols]

A1 Differential amplifier M1 to M6 Multiplier 10 Voltage pulse conversion circuit 11 Storage circuit 12 Differential amplifier 13,14 Input terminal 30 Derivative function generation circuit 31 Coupling strength control circuit (WPU) 100 Back propagation signal switching terminal 101 Polarity inversion terminal 103 Teacher signal input terminal u _i Internal state of neuron i Output value of o _i neuron i w _ij Synapse coupling strength f Transfer function λ Neuron gain coefficient t _i Teacher signal ε Learning parameter Δw _ij Change in coupling strength Quantity ζ backpropagation signal

Claims (2)

[Claims] 1. To learn a plurality of learning pattern vectors consisting of a predetermined input pattern vector and a teacher pattern vector corresponding to the vector, the teacher pattern vector and a network output pattern vector corresponding to the input pattern vector. In an analog neural network circuit that incorporates a circuit that executes a predetermined learning rule that updates the coupling strength in the direction that minimizes the error signal defined by the difference between the learning pattern vector and the learning pattern vector, A process of executing the predetermined learning rule by using a pattern vector as an error signal and calculating a first coupling strength change amount, and executing the learning rule by using the network output pattern vector as an error signal, and changing the second coupling strength change. The process of calculating the amount and the amount proportional to the difference between the first and second bond strength change amounts An analog neural network circuit including a circuit for performing learning by repeating the process for a plurality of learning patterns. 2. The learning executes the predetermined learning rule by using the teacher pattern vector as an error signal, calculates a first coupling strength change amount, and combines the first coupling strength change amount by an amount proportional to the first coupling strength change amount. The process of updating the strength, the learning rule is executed by using the network output pattern vector as an error signal, the second coupling strength change amount is calculated, and the second coupling strength change amount is proportional to the changed sign. 2. The analog neural network circuit according to claim 1, comprising the step of updating the bond strength by the amount.