JP3095798B2 - Analog back-propagation learning circuit and its driving method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an analog back-propagation learning circuit in which a neural network is implemented as hardware and a driving method thereof.

[0002]

2. Description of the Related Art In recent years, research on information processing using a neural network imitating the nervous system of a living body has been advanced. Various models are proposed, pattern recognition,
It has been confirmed that it is superior to conventional Neumann computers in fields such as optimization problems.

However, in software simulation on a general-purpose processor, there is a problem in that a large-scale network causes a sudden increase in the amount of calculation and high-speed processing. Therefore, research on hardware implementation using circuit technology centered on LSIs has been conducted. Initially, the hop field (Hopfield) with a fixed weighting factor
d) The model has been implemented as an LSI, but subsequently, an LSI having a function of making synaptic connections variable has been studied. However, research on hardware implementation of networks with learning functions is still inadequate.

[0004] The inventors of the present application are studying a sensing system for recognizing gas or odor by performing pattern recognition of a plurality of sensor output patterns using a neural network. Currently, this pattern recognition is performed by software simulation, but in order to create a small and portable system, it is necessary to develop dedicated hardware including a learning function.

[0005] By the way, neural networks include:
There are interconnected and layered networks. The former is being studied for optimization problems, and the latter for application to pattern recognition and control. The backpropagation learning method is known as a learning rule for a layered network having no feedback connection. The network is configured as shown in FIG. 14 by neurons modeled as shown in FIG. The neuron obtains an output value by applying a nonlinear function to an amount obtained by weighting and adding the output of the previous layer. The non-linear function may be a continuous and differentiable monotonically increasing function, but in general, a sigmoid function represented by Expression 1 is often used.

[0006]

(Equation 1)

The connection weight coefficient w _ij in FIG. 13 indicates a positive excitatory connection, a negative suppressive connection, and a zero value indicates no connection. When an input signal of the pattern p in the input layer of the network is given, the signal is transmitted in the order of input layer → intermediate layer → output layer to obtain an output. At this time, the i-th output of the output layer is _defined as y _{Pi and} the desired output (teacher signal) of the neuron at that time is _defined as t _Pi . The square error between the two values can be expressed by Equation 2.

[0008]

(Equation 2)

The back-propagation learning method is a learning method in which the weighting factor of the network is changed so as to minimize Equation 2, and the output approaches a desired output. At this time, the weight is changed in the order of the output layer, the intermediate layer, and the input layer, which is opposite to the input signal, and thus the name is given. The learning signal δ _Pi of the output layer is given by Expression 3.

[0010]

(Equation 3)

Here, f and f 'are output functions and their derivatives, and net _Pi is an internal state of the i-th neuron (weighted sum of outputs of the previous layer). This learning signal δ
_{When Pi} is used, the learning signal δ _Pj of the intermediate layer is expressed as in Expression 4.

[0012]

(Equation 4)

When the learning signals of Equations 3 and 4 are used, each weight coefficient w _ij is changed by the width Δw _{ij of} Equation 5.

[0014]

(Equation 5)

Here, ε is a constant determined empirically,
It has a large effect on the number of times of learning until convergence and the stability of the network. By repeating a series of learning, the output value y _Pi of the neuron in the output layer converges on the desired output value t _p1 . (1) In general, when the back propagation learning method is used, a method of obtaining weight coefficient change amounts for all patterns and changing the weight coefficients collectively is often adopted.

[0016]

The electronic circuit of the learning circuit includes an analog system, a digital system, a hybrid system of both, and an optical system. The inventors of the present application have been studying to adopt an analog system which can be realized by a simple circuit with a small number of wirings and to make a neural network using a back propagation learning rule into hardware. Almost at the same time, there was reported an example in which the backpropagation learning rule was implemented digitally.

However, in the above-described back propagation learning by the digital method, a large amount of hardware is required to introduce a large approximation to a circuit or to realize a sigmoid and its derivative by table conversion or the like. There's a problem. Therefore, here, the analog method is used in consideration of the trade-off between the hardware amount and the performance. In addition, a time-division multiplexing method in which input and transmission are performed in time series was adopted for hardware implementation.

As a result, not only the amount of wiring can be reduced, but also the number of multipliers required for the product-sum operation can be reduced. Further, the analog method requires a non-linear element having good linearity for storing synaptic connection weights between neurons, but it is difficult to realize the non-linear element using a semiconductor element. Therefore, in the present invention, the change and the holding of the weight are realized by using an electrochemical integrating element which is a kind of concentration cell having a small leak current and a good linearity.

Further, according to the above-described method of changing the weighting factors collectively, the required number of analog memories increases, so the present invention uses a sequential conversion method. In addition, this time, the network is examined by simulation, circuit constants are optimized, and an efficient method of driving an analog back propagation learning circuit is obtained.

As described above, the present invention provides an analog back-propagation learning circuit and a method for driving the analog back-propagation learning circuit, which have a simple circuit configuration and can greatly reduce the learning time by hardware that operates in parallel between neurons. The purpose is. It is another object of the present invention to improve a sigmoid function circuit, a weight coefficient changing circuit, and a multiplication circuit to improve the pattern recognition capability and reduce the number of times of learning until convergence.

[0021]

SUMMARY OF THE INVENTION In order to achieve the above object, the present invention provides an analog back propagation having a forward propagation mode transmitted from an input layer to an output layer and a learning mode transmitted backward from an output layer to an input layer. In the learning circuit, in the positive propagation mode, the output of the neuron in the previous layer is switched by an analog switch to obtain a time-series input, and a weight coefficient is read out in synchronization with the time-series input, and the product of the input and A means for obtaining an internal state of a neuron by adding a product for each input by an integrator; a means for obtaining an output value by applying a sigmoid function to the internal state of the neuron; and an output layer in the learning mode. A subtractor that obtains the error between the output of the neuron and the teacher signal in learning from the middle to the middle, and a value obtained by applying a sigmoid derivative to the internal state of the neuron at that time , A means for obtaining a first learning signal, a means for obtaining a change amount of a weighting coefficient by a product of the first learning signal and an output of a previous layer, and an input from an intermediate layer in the learning mode. In the learning to the layer, means for obtaining the product of the learning signal and the weighting coefficient in the next layer by a multiplier, calculating the sum of them by an adder, and obtaining an error signal; Means for obtaining a second learning signal by multiplying the data at that time by a value obtained by operating the sigmoid derivative on the internal state at each unit; and obtaining the second learning signal at each unit. Means for changing the weighting coefficient based on the signal in the same manner as the learning from the output layer to the intermediate layer.

Further, in the method for driving an analog back propagation learning circuit having a forward propagation mode transmitted from the input layer to the output layer and a learning mode backwardly propagated from the output layer to the input layer, in the forward propagation mode, Switching the output of the neuron to obtain a time-series input; reading a weight coefficient in synchronization with the time-series input to obtain a product with the input; and adding the product for each input to the internal state of the neuron Obtaining an output value by applying a sigmoid function to the internal state of the neuron;
In the learning from the output layer to the hidden layer in the learning mode, a step of obtaining an error between the output of the neuron and the teacher signal and a product of a value obtained by applying a sigmoid derivative to the internal state of the neuron at that time are obtained. In the step of obtaining a first learning signal, the step of obtaining a change amount of a weight coefficient by the product of the first learning signal and the output of the previous layer, and the learning from the intermediate layer to the input layer in the learning mode, Obtain the product of the learning signal and the weight coefficient in the layer, calculate the sum of them,
Obtaining an error signal, each unit in the hidden layer captures data at a predetermined timing, and each unit multiplies the data by a value obtained by operating the internal state at that time on a sigmoid derivative to obtain a second learning signal. And a step of changing the weight coefficient based on the second learning signal in the same manner as the learning from the output layer to the intermediate layer.

[0023]

According to the present invention, there is provided a method of driving an analog back propagation learning circuit having a forward propagation mode transmitted from an input layer to an output layer and a learning mode transmitted backward from an output layer to an input layer. In the propagation mode, the output of the neuron in the previous layer is switched to obtain a time-series input, (2) a weight coefficient is read out in synchronization with the time-series input, and a product with the input is obtained, (3) The product of each input is added to obtain the internal state of the neuron, (4) a sigmoid function is applied to the internal state of the neuron to obtain an output value, and (5) the output from the output layer to the hidden layer in the learning mode. In learning, the error between the output of the neuron and the teacher signal is obtained,
(6) The product of a value obtained by applying a sigmoid derivative to the internal state of the neuron at that time is obtained to obtain a first learning signal. (7) The first learning signal and the output of the previous layer are obtained. (8) In learning from the hidden layer to the input layer in the learning mode, the product of the learning signal and the weighting coefficient in the next layer is obtained, and the sum of them is calculated. (9) Each unit in the intermediate layer fetches data at a predetermined timing, and each unit multiplies the data by a value obtained by multiplying the internal state at that time by a value applied to the sigmoid derivative. (10) The weight coefficient is changed based on the second learning signal in the same manner as the learning from the output layer to the intermediate layer.

Therefore, it is possible to reduce the number of times of learning until convergence and to improve convergence performance in various patterns.

[0025]

Embodiments of the present invention will be described below in detail with reference to the drawings. FIG. 1 is a block diagram of an analog back propagation learning circuit showing an embodiment of the present invention. In the figure,
4 is a multiplier, 5 is a sigmoid derivative circuit, 6 is a writing circuit, 7, 10, and 11 are analog switches, 8 is an integration circuit, 9 is a sigmoid function circuit, and 12 is an electrochemical integration element.

Here, this is realized by transmitting the analog voltage in a time division multiplex system. The operation of the network can be divided into a forward propagation mode transmitted from the input layer to the output layer and a learning mode transmitted backward from the output layer to the input layer.
Hereinafter, the operation will be described with reference to FIG. (A) Positive Propagation Mode The input is a time-series input by switching the output y _Pj of the neuron in the previous layer with an analog switch. The weighting coefficient w _ij is read out in synchronization with this, and the multiplier 3 obtains the product with the input. The product for each input is added by an integrator to obtain the internal state net _pi of the neuron.

[0027]

(Equation 6)

An output function (sigmoid function) is applied to this to obtain an output value y _Pi .

[0029]

(Equation 7)

(B) Learning Mode As shown in FIG. 2A, learning from the output layer to the hidden layer is performed by:
An error between the output y _{Pi of the} neuron and the teacher signal t _pi is obtained by the subtractor 13. The product of the internal state net _pi at that time and a value obtained by applying a sigmoid derivative to the sigmoid derivative is calculated by a multiplier to obtain a first learning signal δ _p1 of Expression 3.

The change amount Δw of the weight coefficient is obtained by the product of the learning signal and the output (input signal) of the previous layer (see Equation 5).
ε is determined by adjusting the gain of the circuit. The actual weight is changed by passing a current corresponding to the weight coefficient change amount Δw to change the voltage of the electrochemical integration element. On the other hand, as shown in FIG. 2B, in the intermediate → input layer, the learning signal δ _pi and the connection weight coefficient w in the next layer (output layer)
The product of the _ij calculated with the multiplier, and calculate their sum e _j in adder 14, and the error signal.

[0032]

(Equation 8)

Since this error signal is a time series, each unit in the intermediate layer fetches data into the sample & hold circuit 15 at an appropriate timing. Each unit multiplies this data by a value obtained by operating the internal state at that time on the sigmoid derivative to obtain a second learning signal of Expression 4. The second
The weight coefficient is changed in the same manner as in the case of the intermediate layer-output layer from the learning signal of.

FIG. 3 shows a network configuration for performing the above-described normal propagation learning. This is a configuration diagram when the input, intermediate, and output layers are each 3 neurons. In FIG. 3, 20 is input 1
Analog switch for switching between 3 and 21, unit 1
3, 22 are analog switches for switching the outputs of the units 1 to 3, 23 is a unit connected to the analog switch 22, 1 to 3 and 24 are subtractors, 25 is an adder,
6 is a sample & hold circuit. (C) Operation Timing A control signal is required for switching between the normal propagation mode and the learning mode and for switching the weighting coefficient. In addition, it is necessary to reset the integrator of the product-sum operation, the timing of the sample and hold circuit, and the like. Control signal is counter and EPROM
It is generated by a timing sequencer using. FIG. 4 shows a timing chart.

First, an integrator reset signal is input (step (a)). At the same time, the input and teacher signal patterns are advanced by one. The weighting coefficients are sequentially changed in the intermediate layer positive propagation mode, a product-sum operation is performed, and the internal state and output are determined (steps (b) to (e)). In the positive propagation mode of the output layer, the output of the preceding layer and the weighting coefficient are synchronized to be read out, the product sum operation is performed, and the internal state and the output are determined [step (f) to
(R)].

In order to hold the output of the network, a hold signal of the sample & hold circuit is input [step (nu)]. This is the normal propagation mode, and when only determination is performed, the process returns here. Next, an error signal is calculated by sequentially multiplying the learning signal of the output → intermediate layer by a weighting coefficient, and the corresponding neuron of the previous layer is held by a synchronized signal [steps (l) to (f)].

Output → The learning signal of the intermediate layer is calculated at the same time when the internal state is determined. Similarly, a learning signal can be obtained from the middle to the input layer if the error signal is determined. The combination is changed by multiplying the learning signal by the input and writing the corresponding weight (steps (Y) to (S)). The learning of one input pattern is completed by the above 18 clocks. Currently, the clock frequency is 1
One learning time of one pattern is 18 ms, as KHz
I am going over it.

Next, the circuit configuration of each unit shown in the above block diagram will be described. (A) Integrating circuit An integrating circuit is used to add a time-series value obtained by multiplying an input by a weight coefficient. As this integration circuit, a general one using an operational amplifier 30 as shown in FIG. 5 was used. At this time,
The output Vo ₁ can be represented by Expression 9, and it can be seen that an addition sum proportional to the input can be obtained.

[0039]

(Equation 9)

The on / off type analog switch S ₁ is turned ON only during forward propagation, and when OFF, the integrating circuit serves to keep the internal state of the neuron. Resistance R ₂ of the feedback are connected in order to stabilize the voltage of the hold. The analog switches S ₂ connected in parallel with the capacitor C _1, the step of FIG. 4 (b)
And the output voltage is reset to 0V. Incidentally, the resistance R ₁ is 32 KΩ, the resistance R ₂ is 3 MΩ, C ₁ is 0.2 μF, and the operational amplifier 30 is TL072. (B) Sigmoid function circuit A sigmoid function is used as an output function. FIG. 6 shows an approximate function circuit using an operational amplifier 31 and diodes 32 and 33. A diode limiter 34 is provided on the input side to convert the voltage to about ± 0.5V. At this time, a saturation type nonlinear characteristic is obtained by the characteristics of the diodes 32 and 33. The input of the operational amplifier 31 has a negative voltage of about-
By shifting the gain to 1 to 0 V and adjusting the gain of the operational amplifier 31, a sigmoid approximation function having an output value of 0 to 1 V is obtained.
Incidentally, the resistance R ₄ is 17.8 KΩ, and the resistance R ₅ is 10 K
Ω, the resistance R ₆ is 270 KΩ, the resistance R ₇ is 8.87 KΩ,
Diodes 32 and 33 are 1S1588, operational amplifier 31
The TL072, -V _B is -15V.

FIG. 7 shows the input / output characteristics. In FIG.
The horizontal axis is the input voltage (mV), and the vertical axis is the output voltage (mV). If net _pi is inverted and input, it can be used in the same manner as a normal sigmoid function. The sigmoid derivative required for learning will be described later. (C) Electrochemical integrating element and writing circuit An electrochemical integrating element (Sanyo MD2B2) is used to hold the connection weight which is a problem in the analog circuit. This electrochemical integration element is composed of a silver ion conductive solid electrolyte Ag.
₁₂₁ I ₉₃ P ₁₄ O ₄₉ and silver ion / electron mixed conductor Ag
_{Using 415} Se ₁₈₅ P ₁₅ O _60, it has the property of integrating and storing the amount of electricity supplied in the form of a voltage, and the voltage changes in proportion to the amount of charge. After stopping the energization, the output voltage holding is also very stable (memory potential time change 0.3 mv / 48
hr). Also, since both positive and negative voltage values can be held, excitatory coupling and inhibitory coupling can be realized by one circuit. From the above points, this electrochemical integration element is suitable for an analog quantity storage element holding a weight coefficient.

When the electrochemical integration element is used as a weight coefficient holding element, a circuit for writing a charge amount proportional to the input voltage is required. FIG. 8 shows a voltage-controlled constant current source circuit for this purpose.
Shown in In FIG. 8, the analog switch 43 is for switching the weight coefficient, and the analog switch 44 is for reading (a
Terminal) and writing (b terminal). When the input voltage Vs ₂ enters at the time of writing, the inverting input side of the operational amplifier 41 also has the voltage Vs ₂ , and the current I =
Vs ₂ / R ₈ flows. Since no current flows into the operational amplifier 41, all the current flows through the electrochemical integration element 42.
Therefore, regardless of the internal voltage of the electrochemical integration element 42,
A current proportional to the input voltage Vs ₂ can flow. Incidentally, the resistor R ₈ is 390Ω, and the operational amplifier 41 is TL07.
2.

Although a circuit was used to connect the output of the multiplier to the analog switch via a resistor at the beginning, when the output of the multiplier becomes smaller, the amount of charge to be written is not proportional to the input voltage. did. (D) Multiplication circuit As shown in FIG. 9, a transconductance multiplier (Intersil ICL8013) was used as the multiplication circuit. The transconductance multiplier performs multiplication by controlling the gain of the differential amplifier with an external voltage. 9, the gain A _V of the differential amplifier 45, where _{A V = V 03 / V X} = R L / r e, r e is the emitter _{resistance, r e = 1 / g m} = KT / q _{· I E V 03 = V X} · a V = V X · q · I E · R L / (KT) Note that, K is Borumman constant, T is the absolute temperature.

Here, the current value of the constant current source is set to the external voltage V _y
In if to control, it is possible to take the product of the external voltage V _y and V _X. Therefore, as shown in FIG. 9, the emitter current _IE is controlled. In this circuit, V _x can be both positive and negative, but V _Y must be positive. This is extended to both positive and negative to improve linearity and feedthrough characteristics.

Therefore, multiplication in four quadrants can be performed as it is, utilizing the characteristics of the transconductance of the transistor. Input and output are voltage values, 10 of product of 2 inputs
Output a voltage value of 1 / (E) HC4051 with 8 contacts per circuit using analog switch C-MOS IC
And HC4053 having three contacts and three circuits. In each case, the ON resistance is as small as about 40Ω.

The adder and the subtracter are standard operational amplifier circuits, and the sample and hold circuit is a commercially available IC (LF3
98). Next, circuit constants of the analog back propagation learning circuit of the present invention will be described. When a network is implemented as hardware, various restrictions are imposed. For example,
Examples include saturation at the power supply voltage and limitation of input / output current. Also,
The more accurately the nonlinear function is approximated, the larger the circuit scale and the more complicated it becomes.

Therefore, circuit simulation was performed under various conditions to optimize the circuit constants. Furthermore, the sigmoid derivative was changed to a simplified form that was easy to realize, and the convergence state was simulated to examine the effect of the simplification. (A) Comparison between Simulation and Actual Measured Value Whether or not the simulation accurately represents the circuit operation was confirmed by comparing the value of the weight coefficient after learning with the output value. For simplicity, the comparison was performed using a two-layer network (input layer 3 and output layer 3 neurons) excluding the sigmoid derivative. Table 1 shows the results when ε = 0.56. The input is a simple case where three pattern vectors are orthogonal to each other. From Table 1, it can be seen that the weighting factor and the output value are almost the same, and it is considered that this simulation sufficiently reflects the circuit operation. Thereafter, the circuit was examined by an easy simulation of parameter change.

The points finally considered are output voltage saturation at the power supply voltage of each IC, output current limitation of the weight coefficient changing current circuit, ON resistance of the analog switch, input limitation for analog switch protection, and the like. As the sigmoid function, a value actually measured by a sigmoid function circuit is used as a line-line approximation.

[0049]

[Table 1]

(B) Two-Layered Network In a circuitized network, the value of ε at the time of learning is determined by the gain of each arithmetic circuit. In the above-described two-layer network, by changing the amplification factor of the weight coefficient,
And the change of convergence was examined. The input is three patterns, which are close patterns in which the angle between each pattern vector is only 10 degrees apart. The initial value of the weight coefficient is a random number in the range of ± 10 mV. Table 2 shows the results.
The time when the square error between the output and the teacher signal became 0.1 or less was regarded as convergence. When ε is 0.56 or less, the update of the weight coefficient is slow, and the number of times of learning until convergence increases. Conversely, if ε is too large, such as 56.6, it will vibrate,
Stable convergence cannot be obtained. In the end, ε worked well at 5.6. Further, by reducing the initial value (± 1 mV), the number of times of learning until convergence was reduced to １ ／ to の of Table 2. It was confirmed that the convergence was achieved with the same number of times of learning in an actual circuit.

[0051]

[Table 2]

(C) Examination of Sigmoid Derivatives It is known that in a two-layer network, a pattern that cannot be linearly separated, such as EXOR (exclusive OR), cannot be identified. In order to identify even such a pattern, the network has a three-layer structure, and a sigmoid derivative is added. However, since it is difficult to realize an accurate function of the sigmoid derivative, a study was performed considering a simplified function.

(A) Evaluation of sigmoid derivative Referring to the actual shape of the sigmoid derivative (FIG. 10A), the derivative is simplified and triangular approximation [FIG.
(B)], square approximation [FIG. 10 (c)], constant value [FIG.
(D)] was examined. A constant value means that there is no derivative. EX input in a three-layer network
Table 3 shows the results of examining the effect of each derivative on convergence when the OR pattern was used.

The number of neurons in the input, intermediate and output layers is 2, 3, and 2, respectively. ε is 0.1
The initial value of the weighting coefficient was changed ten times with random numbers in the range of ± 10 mV. The convergence judgment is the same as in the previous section. It does not converge at a constant value, indicating that an appropriate derivative is required. In the case of the square approximation, the number of convergence learnings shown in Table 3 was obtained when M = 2 V. Even if the value of M was changed, no further improvement was obtained. It can be seen that when the triangular approximation is performed, the number of times of learning up to convergence is not much different from that of the sigmoid derivative, and can be sufficiently substituted.

[0055]

[Table 3]

Next, Table 4 shows the results of a similar study on the influence of the spread of the tail in the triangular approximation. If the spread (L) is too small, the reduction of the error after learning progresses slowly,
The number of times until convergence increases. This is presumably because the derivative value decreases before the desired output is obtained, and the change of the weighting factor becomes slow. If (L) is too large, the number will increase depending on the initial value. In addition, as for the magnitude of (L), it was found that an error of about ± 0.5 V centered at 4 V does not affect convergence.

[0057]

[Table 4]

(B) Sigmoid Derivative Circuit Based on the evaluation described in the preceding section, the sigmoid derivative has a value in the range of ± 4 V using a circuit approximated by a triangle. FIG. 11 shows a broken line approximation circuit using an operational amplifier and a diode. Triangular approximation is realized by adding two broken lines. In the first operational amplifier 51, the voltage applied to the diode 54 is

[0059]

(Equation 10)

At this time, it becomes zero. At this point, the input resistance value changes. That is, when the amplification degree is smaller than V _s3 ,

[0061]

[Equation 11]

At the time of large V _s3

[0063]

(Equation 12)

, And becomes a polygonal line at the point of − (R ₁₀ / R ₁₁ ) V _B. The same applies to the second operational amplifier 52. When the input is − (R ₁₅ / R ₁₆ ) V _B , the input resistance is-
₁₉ / R ₁₈₎ changes the feedback resistance when V _B, respectively, to change the amplification degree. By properly selecting the resistance value, the input / output characteristics shown in FIG. Incidentally, 53, 55 and 56 are diodes.

FIG. 12 shows the characteristics of the actual circuit. FIG.
In the graph, the horizontal axis represents the input voltage (mV), and the vertical axis represents the output voltage (mV). (D) Comparison with Simulation in Three-Layer Configuration A network having a three-layer configuration and a simulation in which the above-described triangular approximation derivative was incorporated was compared with measured values. is 0.1 and the initial value is a random number in the range of ± 10 mV.

Each of the orthogonal pattern, the proximity pattern, and the EXOR pattern became an output pattern that almost coincided with the teacher signal with the same number of learnings as the simulation. However, the values of the weighting factors were completely different in the case of the proximity and EXOR patterns. It was confirmed by simulation that the square error was small even when the weight coefficient value at that time was used. In addition, the circuit was operated using the weighting coefficients after some learning by simulation. The square error at that time is 1.6
At 3, the weight coefficient was in the range of ± 30 mV. The weighting factors at the time of convergence were almost equal (Table 5). Therefore, it is considered that the actual circuit includes an error in the initial stage of learning, starts as an initial value different from the simulation, and converges to another solution.

[0067]

[Table 5]

As described above, the circuit was examined by simulation. As a result, when there is no sigmoid derivative in the network having the two-layer structure, even if the input pattern is close to the input pattern, the convergence can be achieved quickly enough by adjusting ε. I was sure to do. EXOR in a three-layer network
To learn a pattern that cannot be linearly separated like a pattern, a derivative is needed. Even in that case, it was confirmed that the trigonometric derivative can be substituted. This function can be realized by a broken line approximation circuit using an operational amplifier and a diode.

It should be noted that the present invention is not limited to the above-described embodiment, but various modifications are possible based on the spirit of the present invention, and these are not excluded from the scope of the present invention.

[0070]

As described above in detail, according to the present invention, the hardware of the back propagation learning method is realized by a method of transmitting an analog voltage in a time-division manner, thereby simplifying wiring and circuits. And the learning time can be greatly reduced. The retention of the weighting factor was performed by an electrochemical integration element, but the linearity was good and the retention characteristics were excellent.
The ability to hold both positive and negative voltages is suitable for handling variable weighting factors.

Further, according to the simulation, if there is no sigmoid derivative in the network having the two-layer structure, by adjusting ε, it is possible to converge a sufficiently fast number of times even for a pattern whose inputs are close to each other. In order to learn a pattern that cannot be linearly separated like an EXOR pattern in a three-layer network, a derivative is required. In this case, a derivative approximated by a triangle can be used instead.

Therefore, with the above configuration, it is possible to reduce the number of times of learning until convergence and to improve the convergence performance in various patterns.

[Brief description of the drawings]

FIG. 1 is a block diagram of an analog back propagation learning circuit showing an embodiment of the present invention.

FIG. 2 is an explanatory diagram of an error signal in an output → intermediate layer and an intermediate → input layer of the present invention.

FIG. 3 is a configuration diagram of a network for performing normal propagation and learning according to an embodiment of the present invention.

FIG. 4 is an operation timing chart of the analog back-propagation learning circuit according to the embodiment of the present invention.

FIG. 5 is a configuration diagram of an integration circuit of the analog backpropagation learning circuit according to the embodiment of the present invention.

FIG. 6 is a sigmoid function circuit diagram of an analog backpropagation learning circuit showing an embodiment of the present invention.

FIG. 7 is a sigmoid function input / output characteristic diagram of the analog back propagation learning circuit showing the embodiment of the present invention.

FIG. 8 is a circuit diagram of a weight coefficient changing circuit of the analog back propagation learning circuit according to the embodiment of the present invention.

FIG. 9 is a multiplier circuit diagram of the analog back propagation learning circuit showing the embodiment of the present invention.

FIG. 10 is a diagram schematically illustrating a sigmoid and an approximate derivative of the analog back propagation learning circuit according to the embodiment of the present invention.

FIG. 11 is a triangular approximation derivative circuit diagram of an analog back propagation learning circuit showing an embodiment of the present invention.

FIG. 12 is a triangular approximation derivative input / output characteristic diagram of the analog back propagation learning circuit showing the embodiment of the present invention.

FIG. 13 illustrates a modeled neuron.

FIG. 14 is a schematic configuration diagram of an analog back propagation learning circuit.

[Explanation of symbols]

1-4 Multiplier 5 Sigmoid derivative circuit 6 Write circuit 7, 10, 11, 20, 22, 43, 44 Analog switch 8 Integrator 9 Sigmoid function circuit 12, 42 Electrochemical integral element 13, 24 Subtractor 14, 25 Adder 15, 26 Sample & hold circuit 21, 23 Units 1-3 30, 31, 41 Operational amplifier 32, 33, 54, 55, 56 Diode 34 Diode limiter 45 Differential amplifier 51 First operational amplifier 52 Second operational amplifier 53 Third operational amplifier

────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-2-64788 (JP, A) JP-A-2-66688 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06G 7/60 G06F 15/18

Claims (6)

(57) [Claims] 1. An analog back propagation learning circuit having a forward propagation mode transmitted from an input layer to an output layer and a learning mode transmitted back from an output layer to an input layer, wherein: (a) in the forward propagation mode, Means for switching the output of the neuron with an analog switch to obtain a time-series input;
(B) a multiplier that reads out a weight coefficient in synchronization with the input of the time series and obtains a product of the input, and (c) means that obtains an internal state of the neuron by adding the product for each input by an integrator. d) means for obtaining an output value by applying a sigmoid function to the internal state of the neuron, and (e) subtraction for obtaining an error between the output of the neuron and the teacher signal in learning from the output layer to the intermediate layer in the learning mode. Means for obtaining, by a multiplier, a product of a value obtained by applying a sigmoid derivative to the internal state of the neuron at that time to obtain a first learning signal; and (g) obtaining the first learning signal. Means for obtaining the change amount of the weight coefficient by the product of the signal and the output of the previous layer; and (h) in learning from the intermediate layer to the input layer in the learning mode, the product of the learning signal and the connection weight coefficient in the next layer is calculated. Obtain with a multiplier and calculate their sum with an adder Means for obtaining an error signal,
(I) Each unit in the intermediate layer takes in data into the sample & hold circuit at a predetermined timing, and in each unit, the data is multiplied by a value obtained by operating the internal state at that time on a sigmoid derivative to perform second learning. Means for obtaining a signal;
(J) means for changing the weight coefficient based on the second learning signal in the same manner as the learning from the output layer to the intermediate layer. 2. The analog back-propagation learning circuit according to claim 1, wherein a diode limiter is connected to an input side of the sigmoid function circuit to obtain a saturated nonlinear characteristic. 3. The writing circuit of the weighting coefficient changing circuit has a voltage-controlled constant current source circuit configuration, and obtains a current proportional to the input voltage regardless of the internal voltage of the electrochemical integration element. 2. The analog back-propagation learning circuit according to claim 1, wherein the analog back-propagation learning circuit is caused to flow. 4. The analog back-propagation learning circuit according to claim 1, wherein the multi-quadrant multiplication is performed by using a transconductance characteristic of a transistor of the multiplier. 5. The analog back-propagation learning circuit according to claim 1, wherein, when learning a pattern that cannot be linearly separated, learning is performed using a trigonometrically approximated derivative. 6. A method for driving an analog back-propagation learning circuit having a forward propagation mode transmitted from an input layer to an output layer and a learning mode transmitted backward from an output layer to an input layer, wherein (a)
In the positive propagation mode, a step of switching the output of the neuron in the previous layer to obtain a time-series input, and (b) reading a weight coefficient in synchronization with the time-series input and obtaining a product of the input and the weight coefficient. (C) adding the product to each input to obtain an internal state of the neuron; (d) obtaining an output value by applying a sigmoid function to the internal state of the neuron; and (e) outputting in the learning mode. Obtaining the error between the output of the neuron and the teacher signal in learning from the layer to the intermediate layer; and (f) obtaining the product of the value obtained by applying a sigmoid derivative to the internal state of the neuron at that time. Obtaining one learning signal;
(G) obtaining a change amount of a weight coefficient by a product of the first learning signal and an output of a previous layer; and (h) learning in a next layer in learning from the intermediate layer to the input layer in the learning mode. Obtain the product of the signal and the coupling weight coefficient, calculate the sum of them,
Obtaining an error signal; and (i) each unit in the intermediate layer captures data at a predetermined timing, and in each unit, the data is multiplied by a value obtained by operating the internal state at that time on a sigmoid derivative to obtain a second signal. And (j) changing the weighting factor based on the second learning signal in the same manner as the learning from the output layer to the intermediate layer. .