WO1993018474A1

WO1993018474A1 - Devices for use in neural processing

Info

Publication number: WO1993018474A1
Application number: PCT/GB1993/000509
Authority: WO
Inventors: Trevor Grant Clarkson
Original assignee: University College London; King's College London
Priority date: 1992-03-11
Filing date: 1993-03-11
Publication date: 1993-09-16

Abstract

A device for use in neural processing comprises a plurality of probabilistic RAMs (pRAMs), or other neural elements, having digital outputs connected in common to a digital pulse accumulator. A digital decay circuit is operable to cause the contents of the pulse accumulator to decay, and the output of the decay circuit is connected to the input of a threshold device. This generates an output signal when the signal at its input exceeds a predetermined level. Means may be provided for modelling other neural features, including a refractory period, pre-synaptic weights, and synaptic noise.

Description

DEVICES FOR USE IN NEURAL PROCESSING

This invention relates to artificial neuron-like devices (hereinafter referred to simply as "neurons") for use in neural processing.

One of the known ways of realising a neuron in practice is to use a random access memory (RAM) . The use of RAMs for this purpose dates back a considerable number of years. It has been suggested that if one were able to construct a RAM in which a given output, say a '1', was produced by a given storage location with a probability of between 0 and 1 (rather than with a probability of either 0 or 1 as in a conventional RAM) , such a RAM would have a potential for constructing neural networks which mimicked more closely than hitherto the behaviour of physiological networks. (See Gorse, D., and Taylor, J.G., 1988, Phys. Lett. A. 131, 326-332; Gorse, D. , and Taylor, J.G., 1989, Physica D, 34, 90-114). The term "pRAM", an abbreviation for "probabilistic RAM", is used there and herein for a RAM in which a given output is produced with a given probability between 0 and 1 when a particular storage location in the RAM is addressed, rather than with a probability of either 0 or 1 as in a conventional RAM. In PCT publications WO92/00572 and O92/00573 and in a paper entitled "Hardware realisable models of neural processing", published in Proceedings of the first IEE International Conference on Artificial Neural Networks 1989, pp 242-246 there is a description of how a pRAM may be constructed. There is described a device for use in a neural processing network, comprising a memory having a plurality of storage locations at each of which a number representing a probability is stored; means for selectively addressing each of the storage locations to cause the contents of the- location to be read to an input of a comparator; a noise generator for inputting to the comparator a random number representing noise; and means for causing to appear at an output of the comparator an output signal having a first or second value depending on the values of the numbers received from the addressed storage locations and the noise generator, the probability of the output signal having a given one of the first and second values being determined by the number at the addressed location.

It is an object of the present invention to provide a novel neuron in which the effect of time on the output thereof is incorporated in a form which at least approximates what is currently understood to be the case with biological neurons, whilst also incorporating stochastic properties.

According to the invention there is provided a neural device which comprises a plurality of elements each of which has at least one input and a digital output, the elements having their outputs connected to a digital pulse accumulating means and a digital decay circuit operable to cause the contents of the pulse accumulating means to decay, the output of the decay circuit being connected to the input of a thresholding device, the thresholding device generating an output signal at its output of a thresholding device, the thresholding device generating an output signal at its output when the signal on its input exceeds a predetermined level.

Preferably, the each of the said elements is a pRAM (as hereinbefore defined) . The pRAMs may be 1-pRAMs, that is to say they have a single address line and thus two addresses, corresponding respectively to a 0 and a 1 as the input address and two storage locations, one for each address.

In the accompanying drawings:

Figure 1 shows diagra matically an aspect of the way in which biological neurons act;

Figure 2 shows a first embodiment of the invention; Figure 3 consists of two graphs, the top graph being a plot of the voltage at the input of the thresholding device in Figure 1, against time, and the bottom graph showing the number of pRAMs which are firing, each pRAM producing a unit voltage pulse of unit time duration;

Figure 4 shows the use of a shift register for providing a refractory period;

Figure 5 shows a modification of Figure 2, providing a preferred implementation of post-synaptic weight;

Figure 6 shows an alternative to Figure 5, which allows for the degree of synaptic noise to be altered;

Figure 7 shows the architecture of a preferred embodiment of the present invention;

Figure 8 is a graph showing a typical pulse produced by a synapse;

Figure 9 shows an embodiment of the invention in which pulses of the type shown in Figure 8 are employed;

Figure 10 shows a preferred architecture for the embodiment of Figure 9; and

Figures 11 and 12 show two alternative ways of modelling a refractory period.

The temporal modelling of real neurons is normally based on the leaky integrator model by Hodgin & Huxley, (Hodgin A L and Huxley A F, J Physiol. (London) 117, 500, 1952).

A biological neuron operates in digital and analogue fashions. Nerve impulses correspond to a digital form of information transmitted between neurons, but the encoded information is an analogue value, i.e. an impulse density. Synapses can either be excitatory or inhibitory.

The way in which biological neurons use excitatory or inhibitory synapses in shown in Figure 1.- Here, the post-synaptic potential 3 of synapse 1 may be inhibited or reduced in amplitude by synapse 2 so that the' combined post-synaptic potential 4 is reduced. For excitatory action, the post-synaptic potential 3 of synapse 1 may be enhanced or increased in amplitude by synapse 2 so that the combined post-synaptic potential 4 is strengthened. In the context of a pRAM, excitatory action results in an increased firing probability for the pRAM, and inhibitory action results in a decreased firing probability.

The embodiment of the invention in Figure 2 will now be described. This embodiment comprises four 1-pRAMs 10, though there is no particular significance in the use of four, and more or less than four could be used. Each pRAM has two storage locations each of which holds a number representing a probability α. The two values are referred to below as α₀ and c_₁. The values of α₀ and a in each pRAM are independent of the values of α₀ and a_λ in each other pRAM. As is explained in the IEE paper and PCT publications O92/00572 and O92/00573, identified above, the probability of a 1 appearing at the output of the pRAM when a given storage location is addressed is equal to the value of α. For example, if = 0.7, the probability of a 1 appearing at the output of the pRAM concerned when that location is addressed is 0.7 and the probability of a 0 is 0.3.

Each pRAM has a single address input 11, and the four inputs receive trains of pulses. Each pulse train can be used to represent a real number n = [0,1], i.e. a number from 0 to 1, with the probability of there being a 1 as opposed to a 0 being equal to the number n.

The output 12 of each pRAM, which is a pulse representing a 0 or a 1, passes to a pulse accumulator, or counter, 19 whose value increases linearly with each pulse applied. The output 28 of the counter 19, which may, for example, be a 16 bit signal, is taken to a multiplier 20' where it is multiplied by a decay rate 21 at regular

• intervals. The decay rate is in the range 0 to 1 and is normally close to 1. Therefore a fraction of the counter output is written back into a LOAD DATA input 27 of the counter at the same regular intervals using a LOAD ENABLE input 25. This has the effect of producing an exponential decay in the counter output.

The thresholding device 22 generates a pulse representing a 1 at its output 23 each time the counter value 28 at its input goes from a value less than a predetermined threshold value to a value greater than the threshold value.

Suppose, by way of example, that the pulses 12 which represent l's are spikes of unit binary amplitude and duration. The number of pulses which the pRAMs can apply to the input 16 of the device 19 is therefore 0, 1, 2, 3 or 4, depending on how many pRAMs have a l at their output and how many have a 0. This accumulated pulse count decays over time as a result of the decay circuit comprising the counter 19 and the multiplier 20, but can be increased or caused to decay more slowly as further pulses arrive at the input 16. An example of this is shown in Figure 3 of the accompanying drawings.

After a pulse has been produced by the threshold circuit, the neuron enters a refractory period during which it will not fire again. This may be achieved by a standard counter circuit or by a shift register as shown in Figure 4 of the accompanying drawings. The threshold 22 operates by simply selecting the most significant bit of the counter output. A pulse representing this 1-bit signal passes from the threshold circuit through the gate 29 which is normally enabled by the signal 34. This pulse is applied to the input of a shift register 30 which is clocked at a rate determined by a refractory period clock 31. The number of stages in the shift register determines the refractory period. Whilst the pulse propagates through- the shift register a 1 appears at each of the outputs 32 in turn. The outputs 32 are applied to a comparator 33. The output of the comparator 33 is asserted if a 1 is present at any of the shift register outputs 32 since its input is non-zero. When the comparator output is asserted the gate 9 is disabled by the signal 34. In this way no further pulses may appear at the neuron output 24 for the duration of the refractory period, a duration which is determined by the length of the shift register. Only when the refractory period has expired and the outputs 32 of the shift register are all zero will the gate 29 be enabled and the neuron may fire when the counter output 28 next exceeds the threshold. The neuron may fire immediately if the counter value is already above the threshold.

The neuron described above has no provision for learning. However, this can be incorporated by arranging for the values of α₀ and a± in each pRAM to each pRAM to alter over time depending on the firing pattern of the complete neuron, as represented by the output 24, or by reinforcement training using signals representing success or failure. Other methods of training are mentioned below.

The value of the decay rate 21 is chosen to give a suitable decay rate, for example so that the decay by 63% of the initial value is of the order of- 10-100 times the clock rate of the pulses being applied to the inputs of the pRAMs. This gives an effect broadly equivalent to what is believed to occur at the leaky membrane in biological neurons. However, other decay rates may be used if more appropriate for the particular purpose in hand. The decay rate 21 will change under external control as the learning process proceeds. The values of ₀ and α_χ provide the equivalent of pre-synaptic weights in biological neurons.

It should be emphasised, however, that there is no suggestion that the neuron described herein operates in precisely the same way as a biological neuron, and indeed- the operation of the latter is far from completely

« understood. Post-synaptic effects may be obtained by counting the pulses 12 produced at the output of a pRAM more than once. The number of times that such pulses are presented to the counter input 16 is also dependent upon the learning process and the number chosen is a weighting factor for that synapse. The number of times a spike from a synapse is counted is controlled by a COUNT GATE signal 26 indicated in Figure 2, which is in turn controlled by the post-synaptic weight. If no provision is made for a COUNT GATE signal 26 the pulses from all pRAMs count equally towards the end result. The provision of a COUNT GATE signal makes it possible to give greater weight to the output of some pRAMs than others. This can be achieved with maximum stochasticity if a given pRAM pulse is counted once, and with less stochasticity if a given pRAM is caused to fire a given number of times.

A preferred implementation of the post-synaptic weight is shown in Figure 5 of the accompanying drawings. Here the counter 19 is replaced by a functionally equivalent adder and accumulator circuit. The advantage of the latter circuit is that signals representing many bits may be added simultaneously whereas the counter operates with signals representing single bits over a number of clock periods. Thus the adder-accumulator is preferred as it enables faster operation of the neuron.

In this circuit, an output pulse 12 is produced from a pRAM. This signal is used to operate a switch 43. If the pRAM fires, representing a 1, then the post-synaptic weight 41 is presented to the adder input 39. If the pRAM does not fire, representing a 0, then a signal representing zero 42 is applied to the adder input 39. Signals 38 from other identical synapses are also presented to the adder input 39 at different times.

The adder adds the post-synaptic signals 38 to- the current value of the post-synaptic potential 37 which is held in an accumulator 36. The result of the addition, the combined post-synaptic potential 40, is passed to the threshold circuit 22 and also to the multiplier 20 where it is multiplied by the decay rate 21 and the decayed value 35 is stored in the accumulator 36. This process repeats at regular intervals.

The above circuit operates at maximum stochasticity since either the full post-synaptic weight 41 is added to the post-synaptic potential ^"37 or the zero term 42 is so added. The addition is dependent upon the state of the pRAM output 12 which represents synaptic noise in biological neurons. An extension to the above circuit allows the degree of synaptic noise to be altered.

A preferred implementation of the variable synaptic noise circuit is shown in Figure 6 of the accompanying drawings. Here, the signal 38 generated as described above, may be added a variable number of times to the post-synaptic potential, this factor being 1, 2, 4, 8 or 16 times for example. When a factor of 16 is used, for example, the signal 38 is dependent upon 16 firings of the pRAM 10 which in turn operates the switch 43 for 16 time periods and therefore the signal 46 may be considered to be an average of those 16 time periods. That is to say, the pRAM is operated 16 times in succession with the same input 11 applied thereto. The greater the factor, the lower the synaptic noise. A factor of 16 is chosen by way of an example and is not to be taken as a limiting factor in this design.

In order to maintain the same characteristics of the neuron at each noise level, as set by a noise level input 44 described below, a barrel-shifter 49 is included to scale the input data 47 to the adder 48. A barrel-shifter acts as a simple divider but is restricted to division by powers of 2 which corresponds to an integral number of right-shifts of a binary register. For this ' reason, the factor by which a signal 38 is multiplied is

• also a power of 2, for example, 1, 2 , 4, 8 or 16. By way of example, if a factor of 8 is chosen, then data 38 is produced by the synaptic input over 8 time steps. In order to ensure that the output 40 of the adder 48 is independent of the noise factor used, the data 46 must be divided by a factor of 8 on each occasion. In this case l/8th of the signal 38 is added to the post-synaptic potential on each of the 8 occasions.

When the noise factor is greater than 1 the switch 50 must be operated in order to maintain the same decay dynamics for all noise levels. Its operation is as follows. A noise level is set by the input 44 mentioned above. This input carries a number equal to the noise factor. It directly determines the number of iterations that the input pRAM 10 is operated for each input 11, and correspondingly determines the division factor of the barrel-shifter 49 by programming the number of binary places to shift the data 46 using the input 45. On each iteration the pRAM 10 produces an output which may be a 0 or a 1, the signal 46 is divided by a factor 45 carried by the input, using the barrel-shifter 49. The resultant signal 47 is added to the previous post-synaptic potential 37 which is held in the accumulator 6. The adder output 40 is written back directly to the accumulator 6 where switch 50 is in the ACCUMULATE position. On the last iteration, the switch 50 is moved to the DECAY position where the decay characteristic of the leaky-integrator neuron is implemented by writing back into the accumulator 36 the adder output 40 multiplied by the decay rate 21. For the highest noise level, where the number of iterations is 1, the first iteration is also the last and the description above should be understood to imply that the switch 50 is in the DECAY position during this operation.

A preferred implementation of the neuron of the present invention is shown in Figure 7 of the accompanying- drawings. The pRAM inputs are serially processed and the pulses accumulated in a similar way to that described in' International Patent Publication W093/00653 and UK patent application No. 9226192.4. In this implementation the individual pRAMs are not physically distinguishable from each other, but rather are virtual pRAMs defined by an architecture in which some elements are common to the individual pRAMs and are assigned successively thereto.

Input 87 is an input which selects the noise level in the same way as input 46 in Figure 6. 88 is a clock divider which generates the timing of the processing operations. 89 is an output of 88, which carries the virtual pRAM number of the pRAM being processed. This number cycles from 0 to 255, for example, during one processing pass if 256 synapses, and thus 256 virtual pRAMs, are used.

90 is another output of clock divider 88, which controls the noise level of the neuron in accordance with the input 87 to the clock divider. Output 90 controls the virtual pRAM block and determines how many times each post- synaptic weight 92, i.e. each pRAM output, is added in the adder 48. Output 90 also controls the barrel shifter 49 to scale the pRAM output 92 for different noise levels. The operation of the barrel shifter 49, the accumulator 36 and the adder 48 is as described above.

The virtual pRAM block further comprises an address generator 91 which is used to select the desired pRAM weight, i.e. the desired memory contents of a pRAM, from a RAM block 96, using the address line 95. Also, the virtual pRAM block comprises a combination 93 of pseudorandom number generator and comparator, which together with the weight selected by the address lines 95 and carried by the data lines 94, form a virtual pRAM. The output of 93 determines whether the post-synaptic weight 92 is added to the combined post-synaptic potential.

Within the above mentioned RAM block 96 are held- not only the pRAM weights, but also output lists, which hold the current outputs of all the pRAMs and the values of' any external inputs to the system, and a connection table specifying from where the pRAMs receive their inputs. Further details of this can be found in the above identified WO93/00653 and UK Patent Application No. 9226192.4

The output of the neuron above is dependent upon the firing rates of its inputs and its own output activity through the modelling of the refractory period. An extension of this model allows the output of the neuron to be additionally dependent upon the timing relationship between pulses arriving on its inputs. In this way, the dynamic characteristic of the neuro-transmitter release can be modelled. Furthermore, the parameters concerned are trainable.

Figure 8 of the accompanying drawings shows a typical pulse produced by a synapse in response to a spike arriving at the time 81. The shape is described as a piecewise-linear approximation of an exponential waveform. On arrival of a pulse at time 81 there is a delay d₂ (82) before the full height of the pulse is reached at time 83. The height of the pulse at time 83 is equivalent to the post-synaptic weight in the models above which do not use input pulse shaping. From time 83 the pulse height corresponding to the post-synaptic weight is held for a time T (84) after which it is allowed to decay over a period d₂ (85) until it again reaches zero at time 86.

The period d₂ may include the leaky-integrator decay implemented at the synapse level, so that d₂ represents the combined synaptic decay and leaky-integrator decay in the one term. This would avoid the use of a further multiplier to provide the decay term. However, in the following description d₂ is taken to be the delay in the synaptic pulse only; the leaky-integrator delay is modelled separately using a multiplier-accumulator circuit. In- other words, over the period d₂ the graph is flatter than the desired end result, the additional decay being provided by the multiplier 20.

In Figure 9, an input pulse arriving at the input 11 generates an output pulse 12 with a probability determined by the pRAM 10 memory contents a_λ . If no pulses are present at the input then a pulse 12 is generated spontaneously with a probability determined by the pRAM 10 memory contents α₀- The pulse shape shown in Figure 8 is ^•generated by the circuit shown in Figure 9 in response to a pulse 12. This pulse 12 is connected to a TRIGGER input 59 of a control unit 58 and a sequence of events starts at a rate determined by a clock input 52.

The starting level 81 of the pulse is assumed to be zero before the pulse 12 arrives and the starting level is represented by the contents of the accumulator 71. When triggered, the control unit 58 actuates switch 62 in order to apply an incremental rise variable held in a register 60 to an adder unit 72 once per clock cycle. The adder unit is clocked by signal 73 derived from a control unit 58. The adder combines the signal 63 from the register 60 with the contents of an accumulator 70. The result of the addition 74 is applied to the accumulator input 71. In this way accumulator 70 increases in value with each clock cycle. This process continues until the output of the adder 74 equals or exceeds the desired post-synaptic weight value which is held in a hold level register 78. This occurs at time 83 on the timing diagram. A comparator 76 compares the output of the adder unit 74 with the contents of the register 78 and sends a signal to an input 53 of the control unit to terminate the incremental rise. This is achieved by the control unit outputting a signal 57 to the switch 62 which disconnects the adder 72 from the incremental rise register 60. At the same time a signal 56 is output by the control unit which presets the counter 66 to the value held in the register 64. The counter 66. counts down at a rate determined by the clock input 67. Whilst the counter is counting, the output of the adder' unit is maintained at a constant value.

When time T (84) expires, the counter 66 reaches zero and sends a signal 55 to the control unit which marks the end of the hold time T. At this point the control unit outputs a signal 57 to the switch 62 which connects an incremental decay register 61 to the input 63 of the adder unit 72. The incremental decay register 61 contains a ^■ number, normally a two's complement negative number, which is passed by the switch 62 to the adder unit input 63 and added to the contents of the accumulator 70 every time the adder 72 is clocked by the signal 73. The result of each addition 74 is written back into the accumulator 70. If a negative number is repeatedly added to the accumulator contents by the operation described, the accumulator contents decrease for each clock period. The incremental decay rate 61 and the post-synaptic weight, or hold level, held in the register 78, determine the decay time d₂ (85). The- contents of the accumulator 70 continue to decrease until the output of the adder 74 reaches zero. A zero-detect circuit 79 sends a signal 55 to the control unit 58 whenever the signal 74 reaches zero. This occurs at time 86 and marks the end of the synaptic pulse. At this time the accumulator 70 is cleared by the control unit using a CLEAR input 69 and the control unit is reset and awaits the next TRIGGER pulse 59 from the pRAM output 12.

The signal generated by the process above has the characteristic shape shown in Figure 8. Similar signals 80 are obtained from other synapses and these pulses are combined with the synaptic output 74 using an adder 97 to form the combined post-synaptic potential 40. The combined post-synaptic potential 40 is subject to an exponential decay caused by the multiplier 20 and the decay rate 21.

A preferred form of the aspect of the invention- just described, incorporating the same type of architecture as that used in Figure- 7, is shown in Figure 10 of the' accompanying drawings. The neurons are serially processed and the pulses accumulated in a similar way to that described in the above mentioned WO93/00653 and UK patent application No. 9226192.4, to which attention is directed for details additional to those given below.

Separate memory RAM 96 is used to hold synaptic weights and parameters, an inter-neuron connection table and the output lists showing the state of the outputs of all neurons in the network. Figure 10 defines a virtual leaky integrator device containing multiple leaky neurons and multiple pRAM synaptic inputs. Accordingly, the RAM 96 also contains a synapse assignment table determining which pRAMs are the synapses of each neuron. The number of pRAM synaptic inputs for each neuron may vary from neuron to neuron.

A clock and divider circuit 88 is used to generate the neuron number 89 of the neuron currently being processed. This information is used to generate part of an address 91 which is then used as a memory address pointer 95 into the RAM 96 where the weights and parameters for the currently-processed neuron are stored. The parameters required for processing a synapse are the current synaptic potential which is written via a data bus 94 into register 103, the incremental rise written into register 106, the incremental decay written into register 107, the hold time written into register 108, the post-synaptic weight written into register 109, the pre-synaptic weight written into register 111 and the current state, which is held in register 110 and which indicates on which portion of the curve shown in Figure 8 the synapse is currently working. These parameters and the synaptic weights are used to perform one iteration of the synaptic processing described above. A new value for the synaptic potential is generated by the adder 98. This has inputs 99 and 100. The input 99 is an input from the internal bus which adds the post-synaptic weight 109 conditionally, if the pRAM formed by the random number generator and the comparator 93 fires. The input 100 is used to accumulate the synaptic potential for one leaky integrator neuron, from all synapses assigned to it. A control unit 114 controls the processing state of a given neuron, i.e. initial ramp, holding state and trailing ramp. Dependent on the pRAM firing, control unit 114 adds the post-synaptic weight 109 to the synaptic potential 103 using the adder 98. The new synaptic potential 101 is written back into the register 103 and thence to the data bus 104 and into the RAM via the port 94. Processing of one iteration of the next synapse commences at this time.

Synapses are assigned to neurons as defined in the synapse assignment table so that neurons may be formed having a variable number of synapses. The number of neurons is limited by the architecture since the combined post-synaptic potential of each neuron must be stored, the output state of the neuron after thresholding must be stored and the refractory period state must be stored. There is a practical limit on the storage assigned for these purposes on-chip or in RAM. A bank of registers 112 holds the post-synaptic potentials of all neurons within a serial-update module. One of the registers 112 is accessed and incremented as each synapse is processed, the selection of one of the registers 112 being determined by the synapse assignment table.

For each iteration of a synapse the adder 97 adds the synaptic potential which forms the input data 47 to the post-synaptic potential 37 which is determined by one of the registers 112 according to the neuron to which the synapse is designated by the synapse assignment table. The resulting combined post synaptic potential 40 is written back into the selected register 112 using the port 113. In this way, the post-synaptic potential of a designated- neuron accumulates as each synapse is processed. When all synapses have been iterated, each post-synaptic potential' in the register bank 112 is in turn presented to the adder 97 with input 47 set to zero. For each post-synaptic potential, the output 40 of the adder is applied to the multiplier 20 where it is multiplied by the decay rate 21 and written back into the register under consideration in the register bank 112. At the same time the output 40 of the adder 97 is taken to the threshold circuit 22 and the refractory period gate 29 and the output state of the neuron is determined and stored in the output list in the RAM 96.

The connection table in the RAM 96 is used to assign neuron outputs 24 to neuron inputs 11 as described in the above mentioned WO93/00653 and U patent application No. 9226192.4.

Figures 11 and 12 show two alternative ways of modelling the refractory period. A short refractory period can conveniently be modelled using an n-stage shift register as shown in Figure 11. The output 130 of the comparator 131 is high whenever the combined post-synaptic potential 132 is above the threshold 140 (i.e. A>B) . This signal is taken to a gate 133 and then to a shift register 134. The number of stages of the shift register is equal to the length of the refractory period • in terms of unit clock periods of the clock input 135. Thus when output 130 is high and a refractory period is not in operation, output 130 is allowed to pass through the gate 133 and into the shift register 134. A pulse is generated at the neuron output 136 at this time. An OR-gate 137 is used to generate an inhibitory signal 138 whenever a logic '1' level is present in any of the stages of the shift register 134. This inhibitory signal prevents further signals 130 from reaching the neuron output 136. Until the logic '1' level has completely passed through the shift register 134, the gate 133 will remain inhibited. The refractory period is therefore the number of shift register stages used multiplied by the period of the clock input 135. ' In order to make the refractory period variable, mask bits 139 are added to the circuit so that the length of the shift register may be effectively reduced. For example, if all the most-significant three mask bits 139 are at a '0' level, then the shift register 134 is effectively shortened by three bits.

When the refractory period is very long, the shift register approach is unsuitable since a long shift register will be required. In such a case a counter can be utilised as shown in Figure 12. The action of the circuit is similar to that of Figure 11 except that a counter 150 is used to inhibit gate 133. Thus when output 130 is high and a refractory period is not in operation, output 130 is allowed to pass through the gate 133 to the neuron output 136. At the same time a preset input 151 of the counter 150 is activated so that the refractory period count 152 is loaded into the counter. Counter 150 operates as a down- counter and on each cycle of the refractory period clock 135, the counter is decremented. Whilst the counter contains a non-zero number, its output 153 is low, which inhibits gate 133. When the counter reaches a count of zero, output 153 goes high and stays high until signal 130 goes high once more. When output 153 goes- high gate 133 is again enabled as the refractory period has ended.

The methods by which the training of the pRAM memory contents or the post-synaptic effects may be performed include reinforcement training, as already mentioned, gradient descent, back-propagation, Kohonen topographic maps and Hebbian learning, all of which are established techniques in the field of neural networks.

The biological neuron shown diagrammatically in Figure 1 has both a excitatory synapse and an inhibitory synapse. In the foregoing description of various embodiments of artificial neuron according to the- invention, the inputs are all described as being excitatory. However, at least some of the inputs would be' inhibitory, and appropriate modifications could be made to achieve this.

Thus, in Figures 5 and 6 the post-synaptic weight 41 could be a 2-s complement number, so that it can take positive and negative values. The same applies to the post-synaptic weight 92 in Figure 7.

In Figure 8, the height of the pulse may be negative, so that ramp 82 may decrease rather than increase, and ramp 85 may increase rather than decrease. Correspondingly, in Figure 9, the incremental rise 60 may be a 2's complement number such that the ramp may increase or decrease in value with each addition, the incremental decay 61 may be a 2's complement number such that the ramp may decrease or increase in value with each addition, and the hold level 78 may be a 2's complement number such that the height of the pulse may be positive or negative. Similarly, in Figure 9 the values of 106, 107, 109 and 112 may be 2^Λs complement numbers.

As an alternative to using 2's complement numbers, the relevant ADDERS could be changed to SUBTRACTORS to generate an inhibitory response.

Claims

CLAIMS :

1. A neural device which comprises a plurality of elements each of which has at least one input and a digital output, the elements having their outputs connected to a digital pulse accumulating means and a digital decay circuit operable to cause the contents of the pulse accumulating means to decay, the output of the decay circuit being connected to the input of a thresholding device, the thresholding device generating an output signal at its output of a thresholding device, the thresholding device generating an output signal at its output when the signal on its input exceeds a predetermined level.

2. A device as claimed in claim l, wherein each said element has a memory having a plurality of storage locations at each of which locations a number representing a probability is stored, first means for addressing the storage locations, and second means for causing to appear at the output of the element, when the storage locations are addressed, an element output signal, the probability of the element output signal having a given one of first and second values being determined by the number at the addressed location.

3. A device as claimed in claim 2, wherein said second means comprises a comparator having first and second inputs and an output forming the output of the said elements, and a noise generator for inputting to one of the inputs of the comparator a random number representing noise, the contents of the addressed storage location being inputted to the other input of the comparator.

4. A device as claimed in claim 2 or 3 , wherein each said element has two storage locations.

5. A device as claimed in any preceding claim, comprising refractory means for preventing the device from generating a device output signal at a device output, during a refractory period following the generation of a previous such signal.

6. A device according to claim 5, wherein the refractory means comprises a shift register, each device output signal passing to the device output via the shift register.

7. A device according to claim 6, wherein the shift register is provided with a mask bit input for varying the effective length of the shift register and thereby the refractory period.

8. A device according to claim 5, wherein the refractory means comprises a clock pulse generator and a counter for counting the clock pulses, the counter serving to prevent generation of the device output signal for a period determined by that required to count a predetermined number of pulses.

9. A device according to any preceding claim, wherein the digital pulse accumulating means comprises, weighting means for according a given weight to each of the digital outputs received from the said elements.

10. A device according to claim 9, wherein the digital pulse accumulating means is a digital counter, and the weighting means comprises a count gate with which the counter is provided, the count gate receiving for each element a count gate signal denoting the number of times the output of the element is to be counted.

11. A device according to claim 9, wherein the digital pulse accumulating means comprises an adder, and for each element a weight is stored, means being provided for supplying the said weight as an input to the adder when the output of the element is non-zero.

12. A device according to any preceding claim, wherein means are provided for causing each element to generate an output a number of times for each signal at its input, and means causing the resulting input to the digital pulse accumulating means to be divided by the said number.

13. A device according to any preceding claim, wherein the outputs of each of the said elements means is connected to the digital pulse accumulating means via a pulse shape generating circuit which for each output signal from the element generates a shaped pulse and applies the successive digital values of the amplitude of the shaped pulse to the digital pulse accumulating means.

14. A device according to claim 13, wherein for at least one element the shaped pulse has successively a first ramp portion, a constant amplitude hold portion, and a second ramp portion opposite in direction to the first ramp portion.

15. A device according to claim 14, wherein for at least one element the first ramp portion is of increasing amplitude and the second ramp portion is of decreasing amplitude.

16. A device according to claim 14, wherein for at least one element the first ramp portion is of decreasing amplitude and the second ramp portion is of increasing- amplitude.

17. A device according to claim 3, or any one of claims 4 to 16 as dependent on claim 3, comprising a data storage member containing any or all of the following namely:

(a) a data relating to the contents of the storage locations of all the said elements,

(b) the current output of the device and of any other devices connected directly or indirectly thereto,

(c) the connectivity of the device, and

(d) where the data storage member holds information relating to more than one such neural device, data identifying which of the said elements forms part of which such device.