GB2220771A

GB2220771A - Integrating circuit

Info

Publication number: GB2220771A
Application number: GB8816908A
Authority: GB
Inventors: Christopher Webber
Original assignee: British Aerospace PLC
Current assignee: BAE Systems PLC
Priority date: 1988-07-15
Filing date: 1988-07-15
Publication date: 1990-01-17
Also published as: GB8816908D0

Abstract

An integrating circuit comprises analogue integrating means 20 (Fig. 2) for receiving an input signal, digital counting means 36 and means for altering a value stored in the digital counting means according to the polarity of a signal received from the integrating means whenever the magnitude of the signal from the integrating means reaches a predetermined threshold, set by Schmitt triggers 32, 34. Thus the count valve represents an integrated value of the input signal (Fig. 4). A d/a converter 38 provides an analogue output. The effects of feedback capacitor leakage are thus much reduced. A logarithmic circuit may be provided by use of non-linearity in the d/a converter 38. The application is to neural networks. <IMAGE>

Description

INTEGRATING CIRCUIT The present invention relates to an integrating circuit and relates particularly, but not exclusively, to an integrating circuit for use in neural networks.

Algorithms which prescribe the manner in which the weights of links between the units of neural networks are updated during the learning commonly require that the weights be changed between iterations by an amount which depends only on the current values of parameters of the system and not on the values at earlier iterations.

If the increments at each pattern presentation are small compared with the scale of the weight magnitudes, so that the network can be considered to assume a steady response throughout the duration of each pattern presentation, the weights may be considered as varying continuously with time. Where the rate of the change of the weights is a function only of the current values of the parameters and not of their time derivatives, the link weights may therefore be computed as the time integral of some function of the values of the network parameters.

The time integration of analogue voltage levels is performed naturally by the known compact operational amplifier arrangement of Figure 1. In Figure 1, a basic time integrator is indicated generally at 10 and comprises an operational amplifier 12 having a capacitor C1 connected in a negative feedback loop. An input signal Vin is in connected to the inverting terminal via a resistor R and the non-inverting terminal is connected to earth. The output voltage Vout is determined from the input Vin by the relation

Input current charges or discharges the capacitor, which maintains an output voltage that changes whenever input current flows.

In Figure 1, the time integration of analogue signals is performed using an operational amplifier arrangement in which the feedback component is a capacitor. Such an arrangement relies on the storage of charge on this capacitor, which should ideally not leak away when the integration is terminated and it is desired to freeze the output of the integrator. This can in practice never be the case, since small leakage currents at the inputs of the operational amplifier and across the substance of the capacitor will always cause the stored charge to decay slowly towards zero. Purely analogue memory is volatile.

This problem is especially significant in the context of learning in analogue electronic implementations of neural networks, for which application this invention was initially conceived. The knowledge contained within a neural network is stored in terms of the strengths of the links between its many processing elements. If the network is to be capable of learning, the strengths of these links must be altered as the experience of the network increases with the acquisition of new information and stored at their optimal values when the learning has finished. Analogue electronic implementations of learning networks must store analogue voltage values which represent the strengths of the links.Learning algorithms prescribe that the values of these link strengths should change during learning in such a way that they are given at any time by the integral over time of some function of the parameters of the network. Thus, a circuit element is required which can perform an integration over time of an input waveform but which can store its output voltage indefinitely, without drift or decay, once the learning has finished. The decay of voltages representing the strengths of the links of a neural network would correspond to the forgetting of the knowledge acquired during learning.

It is necessary for the implementation of neural network learning algorithms that the integration be suspended for certain periods during the sequence of pattern presentations while the network assumes its steady-state response to each new pattern and that the link weights be frozen after the learning is deemed to be complete.

The suspension of integration and, to a certain degree, the freezing of the link weights may be achieved by the use of the known arrangement of Figure 2.

In Figure 2, an integrator is indicated generally at 20 and comprises an operational amplifier 22 comprising a capacitor C2 in a negative feedback loop and comprises hold and reset switches 24 and 26 respectively. The hold switch 24 comprise MOSFETS Q1 and Q2 and the reset switch 24 comprises MOSFETS Q3 and Q4.

The integrator 20 operates as follows: while the gates of the MOSFETs Q1 and Q2 of Figure 2 are held at +15V, their channels conduct with resistances Ron - l00L and the system integrates with time constant (R + 2Ron)C. When the gates of Q1 and Q2 are brought to -15V, their channel resistances rise to Roff - 101 , so isolating the capacitor from V. . The lOM # resistor pulls the drain in of Q2 to ground, at which potential are its source and substrate. No leakage current flows through the channel of Q2 since its drain, source and substrate are at the same potential. Minimal leakage current flows through the capacitor during this hold phase, therefore, so that the integration is suspended and the output voltage is held on the capacitor.The gates of Q3 and Q4 are held at -15V, except when it is desired that the output voltage be reset to zero, so that no leakage current flows in Q3. When the capacitor is to be reset, the gates of Q3 and Q4 are brought to +15V so that their channels conduct and the capacitor discharges with time constant 2R C.

on The output voltage cannot be held on the capacitor indefinitely after the learning, however: even when Q4 is switched off, it presents a finite resistance Roff - 1010D to discharge current.

Leakage across the capacitor itself will reduce the memory time even further. Consequently, even this low-leakage integrator suffers from the ultimate impermanence of purely analogue memory.

The transience of analogue memory is a severe problem for the analogue electronic implementation of neural networks. The problem of long-term storage of analogue levels has been addressed by a suggestion which extends the refresh principle of dynamic random-access memories from the two logic states of these devices to an arbitrarily large number of quantised voltage levels. In this known scheme an analogue voltage is sampled once by a capacitive sample-and-hold device and is thereafter maintained by refresh circuitry within a range of values which spans the value sampled initially. A number of such ranges are arranged contiguously to cover the entire range of possible sample levels.As the decay of the stored voltage approaches the lower limit of the window in which it is to be maintained, the sample-and-hold device is refreshed by switching the upper voltage limit of that window onto the storage capacitor. The stored voltage, having been returned to the top of the window, decays towards the bottom of the window to be refreshed again.

This analogue memory device maintains a voltage level indefinitely within a tolerance set by the complexity of the circuitry: the entire sample range may be divided into as many windows of resolution as are necessary, though a basic arrangement consisting of a voltage comparator and analogue switch together with logic gates must be duplicated for every window. The complexity of the circuitry scales linearly with the degree of resolution needed, ie.

exponentially with the number of bits required to define that degree of resolution needed, ie. exponentially with the number of bits required to define that degree of resolution. Provided that the resolution required is not too high, the refreshed analogue memory is reasonably economical in terms of complexity of circuitry though a comparator is needed for every window of resolution. In order to store a voltage to a tolerance of 8 bits, therefore, 255 comparators and sets of logic gates are needed. The fact that the complexity of the circuitry scales exponentially with the bit-accuracy may prove to be a significant limitation of the scheme.

The refreshed analogue memory is designed as a sample-and-hold device rather than as a time integrator. The integration would presumably have to be carried out either by a modified sample-and-hold stage or by explicit analogue summation: the sum of the current value of the link weight and the increment ~ # at each pattern presentation would be latched onto the sample-and-hold to constitute the new value of the link weight. The former alternative seems undesirable because the refresh operations would interfere with the integration phase. The integration would cease to be a simple, continuous operation extending throughout the duration of each pattern presentation and would need to be strictly coordinated with a refresh cycle.The alternative of explicit analogue summation presents a potential problem in that the analogue output of the summing device, which represents the sum of the current weight and the increment A4, will in general not have quite the same scale of representation (in terms of volts per unit weight) as its input: the equality of these scales of representation requires equality of resistance values to high tolerance within the summing amplifier.

Because the output of the summing amplifier is fed back to the sample-and-hold to be presented again at the input for the next pattern presentation, any inequality between the scales of representation of the weights at the input and output of the summing amplifier would be compounded at every iteration of the learning procedure. The link weights would grow or decay exponentially with time even if the value of ~ presented at the other input of the summing amplifier were always zero.

This type of analogue memory may suffer from a more subtle limitation in the context of neural network applications, which is related to the discrete quantisation of the voltage range. If the learning rule prescribes a change in the link weight which, at each of many iterations, is smaller than the window size, the representation of the weight will not be shifted to the neighbouring window at any of these iterations. The failure of a small prescribed mto shift the link weight to another window will cause the prolonged effect of small prescribed increments to be lost: the refreshed analogue memory does not allow very slow learning to have a cumulative effect over many iterations.This effect is made more severe as more refresh cycles are fitted into an iteration of the learning algorithm: if one integration phase is distributed over separate periods punctuated by refresh cycles, the value of ,@ prescribed by the algorithm for one pattern presentation must be larger in proportion to the number of refresh cycles per presentation in order that the link weight can be shifted between windows from one refresh cycle to the next.

Hence several limitations of refreshed analogue memory are foreseen in the context of their application to electronic implementations of neural networks. The refreshed analogue memory is primarily a voltage storage device and is not naturally adapted to perform the time integration which is fundamental to many neural network learning rules. The occurrence of refresh cycles interrupts the integration of the link weights and the coordination of refresh cycles complicates the interaction between the integrator and the other analogue processing elements. The quantisation of the voltage levels that can be stored may abolish the cumulative effect of slow learning. Perhaps the most significant limitation is that the complexity of the circuitry required for the implementation of refreshed analogue memory rises exponentially with the bit-accuracy of storage.

An aim of the present invention is to provide an integrating circuit which is an improvement on the known integrating circuits described above.

According to the present invention we provide an integrating circuit comprising analogue integrating means for receiving an input signal, digital counting means and means for altering a value stored in the digital counting means according to the polarity of a signal received from the integrating means whenever the magnitude of the signal from the integrating means reaches a predetermined threshold.

An integrating circuit according to the present invention provides an analogue integrator whose output voltage can be maintained indefinitely without decay.

In the embodiments to be described, the integrating circuit comprises a hysteresis device arranged to supply a chosen output logic level to the digital counting means when the magnitude of the signal from the integrating means reaches a predetermined threshold.

The hysteresis device may comprise a Schmitt trigger.

Preferably, the integrating circuit is arranged symmetrically with respect to the polarity of the input signal. In the embodiment to be described, the integrating circuit comprises two hysteresis devices, one for incrementing a value stored in the digital counting means and the other for decrementing a value stored in the digital counting means.

Optionally, the -output from the digital counting means may be combined with the output from the integrating means in digital to analogue converting means. This feature has the advantage of increasing short term accuracy.

The integrating circuit may have a logarithmically coded output.

This feature is particularly advantageous in the context of neural networks as will be explained. One possible way of implementing this feature would be to include digital to analogue converting means comprising a summing amplifier having a feedback element chosen to have an exponential voltage-current relationship.

Particular embodiments of the present invention will now be described, by way of example, with reference to the accompanying drawings in which: Figure 3 shows an integrating circuit according to the present invention; Figure 4 shows graphically the relationship between the digital output (N) of the counter and the output (Videal) of the hypothetical perfect integrator.

Referring to Figure 3, a drift-limited integrating circuit according to the present invention is indicated at 30. The integrating circuit 30 comprises the known integrator 20 of Figure2 with hold and reset switches. The output of the integrator 20 is connected to two Schmitt triggers 32 and 34 which are arranged in parallel. The output of the Schmitt trigger 32 and the inverted output of the Schmitt trigger 34 are supplied to an n-bit counter 36 which is connected to a digital to analogue converter 38 to provide an analogue output. The outputs from the Schmitt triggers 32 and 34 are also supplied to an AND gate 40 the output of which is connected to the reset switch of the integrator 20.

The output of the integrator 20 can take values of either polarity between limits orange and +V . If V. has negative range in polarity, the integrator output grows during the integration phase from some small value until it reaches Vrange, the upper threshold of the Schmitt trigger 32, at which point the upper threshold of this hysteresis device changes to some value slightly above ground (+ C say) and the output of the Schmitt trigger 32 swings to logic high.

This has the effect of resetting the integrator output to + g and simultaneously incrementing the binary counter 36. The Schmitt trigger output remains high until the integration capacitor has discharged as far as + E, the lower threshold of the trigger 32. In this way, a clean pulse is generated at the output of the Schmitt trigger 32 which increments the counter 36 by one only. If the input voltage V. is still negative, the integrator output will rise again from in + IE to + V range when the counter 36 will be incremented again.

The integrating circuit 30 is arranged symmetrically with respect to the polarity of the input voltage, so that if Vin is positive, the output of the integrator 20 will fall towards -Vrange, at which voltage is set the threshold of a second Schmitt trigger 34. The output of this second trigger 34 then swings high to reset the integrator 20 to - ( and decrement the counter 36 by one.

In this way, the counter 36 keeps track of the difference between the number of times the integrator output has reached and and the number of times it has reached orange This difference is equal to that multiple of V range through which the output of an ideal integrator (one with infinite output range and no leakage) would have moved if fed with an identical waveform Vin(t).

The digital output of the counter 36 (equal to N, say) therefore defines a sub-range of the range of output V idealof the hypothetical ideal integrator, which lies between (N-l) V and (N+l) V range range as shown in Figure 4. The output of the actual integrator 20 (which lies between -v and +V ) defines the value of V idealof the range range output of the ideal integrator within that sub-range. A voltage proportional to Videal (in fact scaled down by a factor 2 n, where n is the bit range of the counter) is obtained using a digital-to-analogue converter 38 such as a simple ladder of scaled current sources.This sort of converter is very economical in terms of circuitry: a single transistor at each bit of the counter's output switches current flowing through a resistor into the summing junction of one operational amplifier. The output of the actual integrator 20 may be added (scaling by 2 ) to the total analogue output V out of the arrangement to provide short-term accuracy substantially beyond the n bits defined by the counter 36, which may be, exploited during the learning phase. This addition is straightforward: the (buffered) output of the integrator is allowed to inject current into the summing junction of the DAC 38. In this way the output Vout of the integrating circuit 30 becomes a continuous representation of V.

The significance of the slow discharge of the integration capacitor through leakage is reduced in this arrangement below that n of the standard integrator by a factor 2 , where n is the bit-range of the counter 36. This is because the drift rate of the output is smaller than that of the standard integrator 20 of Figure 2 by a factor of 2n:

In this respect, therefore, the integrating circuit 30 behaves like a standard integrator with time constant 2nRleakageC. The drift-limited integrating circuit 30 therefore has a short-term component of memory which has duration 2n times longer than that of a standard integrator 20, during which period the resolution of the output is better than n bits by a factor equal to the fractional resolution of the standard analogue integrator.The rate of short-term drift can be reduced to an arbitrarily low level by extending the bit-range of the counter 36: the drift rate falls exponentially with increase of the bit range. More significantly, the arrangement has a long-term component of memory which has resolution limited to the bit-accuracy of the counter 36: if the integrator 20 is left to drift without input for a time much greater than RleakageC, the charge stored on the capacitor will decay to zero, so that the output V out will settle at a value in the middle of that sub-range of 2 nVideal which is determined by the state of the counter 36.The digital memory of the counter 36 maintains the output voltage at the middle of that sub-range indefinitely: current leakage will not cause a transition in the state of the counter 36 because leakage attracts the output voltage of the integrator 20 from the transition thresholds and towards ground, provided that the leakage current is significantly greater than the input offset current.

When the integrating circuit 30 is incorporated in a neural network, the sequences of events described above are asynchronous with the cycle of pattern presentations and proceed independently of it.

The drift-limited integrating circuit 30 lends itself to the coding of its output as a logarithmic representation, which may be particularly useful in the context of neural network applications. In many neural network algorithms, the weights of links are unbounded and may grow to large values. In these cases the precise magnitude of a weight is of much less significance than is the fact that it is large: the accuracy is more critical in the cases of weights close to zero. The unbounded range of link weights presents a problem for analogue representation: the voltages representing the link weights are bounded by the physical limitations of the devices supporting them. The greater the range of link weights that are to be represented by a limited voltage range, the less is the accuracy with which small values can be represented.A solution to this dilemma is provided by the use of a logarithmic representation of weights, whereby the voltage representing the numerical value of a link weight is proportional to the logarithm of that value. Such a representation emphasises small values of weights and represents them with greater accuracy than large values: the accuracy of representation as a fraction of the magnitude of the weight is independent of the magnitude. In this way, small values of weights could be represented accurately whilst the effect of saturation of the voltages representing the largest weights is minimized. The logarithmic representation is very convenient for analogue multiplication, since the operation of multiplication is equivalent to the summation of logarithmic representations .

The embodiment of Figure 3 is ideally suited to logarithmic representation: if the feedback element of the summing amplifier of the DAC 38 is chosen to have an exponential voltage - current relation (a diode junction of bipolar transistor), the voltage developed across it will be proportional to the logarithm of the current at the summing junction. The DAC 38 would then be constrained to produce output voltages of only one polarity, though negative weights could be represented using positive output voltage in conjunction with a sign bit taken from the counter. The counter would in this case be arranged to represent negative numbers in the sign magnitude representation in which a negative binary number is encoded as the conjunction of a binary representation of its absolute magnitude with sign bit.

An integrating circuit described above overcomes the problems foreseen for the refreshed analogue memory. Its operation is transparent to other circuitry and it may change its internal state at any time and asynchronously with other events of the cycle of pattern presentations: no externally-coordinated clocking or refresh cycle is necessary and the integration may proceed uninterrupted for the entire duration of each pattern presentation. The prolonged effect of very small weight increments /\;is cumulative, in the same fashion as in the case of the standard analogue integrator: a transition of the counter may be brought about as a consequence of the prolonged effect of small increments distributed over many pattern presentations.Another advantage of the arrangement is that the complexity of the circuitry required rises only linearly with the bit accuracy (ie. logarithmically with the degree of resolution) since the addition of one bit to the counter doubles the resolution of both long-term and short-term storage. In the short term, ie. over timescales less than a lifetime of charge storage on the capacitor, the resolution is substantially beyond the bit-accuracy of the counter.

The permanent storage medium is a digital encoding medium rather than a medium which is truly analogue, though the arrangement is much more compact than a conventional hybrid memory which employs a parallel analogue-to-digital converter and a set of bistable elements, the complexity of which rises exponentially with the bit accuracy and which performs no integration function.

The drift-limited integrating circuit described would perform the function of updating the weights of links in an analogue electronic neural network quite naturally and could be used to implement a wide variety of learning rules. The integrating circuit serves both as a processing element for the execution of learning rules and as a permanent voltage storage device.

Claims

1. An integrating circuit comprising analogue integrating means for receiving an input signal, digital counting means and means for altering a value stored in the digital counting means according to the polarity of a signal received from the integrating means whenever the magnitude of the signal from the integrating means reaches a predetermined threshold.

2. An integrating circuit according to claim 1 comprising a hysteresis device arranged to supply a chosen output logic level to the digital counting means when the magnitude of the signal from the integrating means reaches a predetermined threshold.

3. An integrating circuit according to claim 2 wherein the hysteresis device comprises a Schmitt trigger.

4. An integrating circuit according to any preceding claim which is arranged symmetrically with respect to the polarity of the input signal.

5. An integrating circuit according to any one claims 2 to 4 comprising two hysteresis devices, one for incrementing a value stored in the digital counting means and the other for decrementing a value stored in the digital counting means.

6. An integrating circuit according to any preceding claim wherein the output from the digital counting means is combined with the output from the integrating means in digital to analogue converting means.

7. An integrating circuit according to any preceding claim having a logarithmically coded output.

8. An integrating circuit according to claim 7 comprising digital to analogue converting means comprising a summing amplifier having a feedback element chosen to have an exponential voltage-current relationship.

9. An integrating circuit substantially as herein defined with reference to, and as illustrated in, Figures 3 and 4 of the accompanying drawings.