WO2018004966A1 - Neural integrated circuit with biological behaviors - Google Patents

Neural integrated circuit with biological behaviors Download PDF

Info

Publication number
WO2018004966A1
WO2018004966A1 PCT/US2017/035323 US2017035323W WO2018004966A1 WO 2018004966 A1 WO2018004966 A1 WO 2018004966A1 US 2017035323 W US2017035323 W US 2017035323W WO 2018004966 A1 WO2018004966 A1 WO 2018004966A1
Authority
WO
WIPO (PCT)
Prior art keywords
circuit
coupled
input
output
comparator
Prior art date
Application number
PCT/US2017/035323
Other languages
French (fr)
Inventor
Jose Cruz-Albrecht
Timothy Derosier
Narayan Srinivasa
Original Assignee
Hrl Laboratories, Llc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hrl Laboratories, Llc filed Critical Hrl Laboratories, Llc
Priority to EP17820834.4A priority Critical patent/EP3479303A4/en
Priority to CN201780040366.3A priority patent/CN109416759B/en
Publication of WO2018004966A1 publication Critical patent/WO2018004966A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F7/00—Methods or arrangements for processing data by operating upon the order or content of the data handled
    • G06F7/38—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation
    • G06F7/48—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F7/00—Methods or arrangements for processing data by operating upon the order or content of the data handled
    • G06F7/38—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation
    • G06F7/48—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices
    • G06F7/50—Adding; Subtracting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F7/00—Methods or arrangements for processing data by operating upon the order or content of the data handled
    • G06F7/38—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation
    • G06F7/48—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices
    • G06F7/52—Multiplying; Dividing
    • G06F7/523—Multiplying only
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/088—Non-supervised learning, e.g. competitive learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2207/00—Indexing scheme relating to methods or arrangements for processing data by operating upon the order or content of the data handled
    • G06F2207/38—Indexing scheme relating to groups G06F7/38 - G06F7/575
    • G06F2207/48—Indexing scheme relating to groups G06F7/48 - G06F7/575
    • G06F2207/4802—Special implementations
    • G06F2207/4818—Threshold devices
    • G06F2207/4824—Neural networks

Definitions

  • This disclosure relates to neural network circuits.
  • Reference [4] which is incorporated herein by reference, some biological behaviors are presented such as homeostatic plasticity (HP) , kinetic dynamics (KD) , and axonal delay (AD) in a single neuron; however, Reference [4] does not include synapses with Short Term Plasticity (STP) and Spike Timing Dependent Plasticity (STDP) , and does not integrate behaviors in a circuit with multiple neurons and synapses.
  • STP Short Term Plasticity
  • STDP Spike Timing Dependent Plasticity
  • Patent Application 13/679,727 filed 11/16/2012, which is now U.S. Patent No. 8,996,431, issued 3/31/2015, titled "Spike Domain Neuron Circuit with Programmable Kinetic Dynamics, Homeostatic Plasticity and Axonal Delays”.
  • a circuit for emulating the behavior of biological neural circuits comprises a plurality of nodes wherein each node comprises a neuron circuit, a time multiplexed synapse circuit coupled to an input of the neuron circuit, a time multiplexed short term plasticity (STP) circuit coupled to an input of the node and to the synapse circuit, a time multiplexed Spike Timing Dependent Plasticity (STDP) circuit coupled to the input of the node and to the synapse circuit, and an output of the node coupled to the neuron circuit, and an interconnect fabric coupled between the plurality of nodes for providing coupling from the output of any node of the plurality of nodes to any input of any other node of the plurality of nodes
  • a method for emulating the behavior of biological neural circuits comprises providing a plurality of nodes, providing a neuron circuit in each node, providing a synapse circuit in each node coupled to an input of the neuron circuit, providing a short term
  • STP plasticity
  • STDP Spike Timing Dependent Plasticity
  • FIG. 1A shows a top level diagram of neural circuit
  • FIG. IB shows a diagram of a node in accordance with the present disclosure
  • FIG. 2 shows a diagram of the core of a node and connections among the components of the node in accordance with the present disclosure
  • FIG. 3 shows a detail of the short term plasticity (STP) circuit with time multiplexing in accordance with the present disclosure
  • FIG. 4 shows a timing diagram of the STP circuit and shows one example prei spike in accordance with the present disclosure ;
  • FIG. 5A shows a typical waveform at the input of the input of the STP circuit for a given slot i
  • FIG. 5B shows a typical output of the circuit for signal xi for the circuit operating in a first mode
  • FIG. 5C shows a typical output of the circuit for xi for the circuit operating in a second mode in accordance with the present disclosure
  • FIG. 6 shows a detail of one embodiment of an spike timing dependent plasticity (STDP) circuit in accordance with the present disclosure
  • STDP spike timing dependent plasticity
  • FIG. 8A shows a detail of a "S-to-E” (Spike-to- Exponential) circuit
  • FIG. 8B shows a detail of a "E-to-S”
  • FIG. 8C shows a detail of the synapse
  • FIG. 8D shows a detail of a kinetic dynamics (KD) circuit in accordance with the present disclosure
  • FIG. 9A shows a neuron
  • FIG. 9B shows a homeostatic plasticity (HP) circuit
  • FIG. 9C shows an Axonal Delay (AD) circuit in accordance with the present disclosure
  • FIG. 10A shows the results of simulations of a network with neurons and synapses
  • FIG. 10B shows the results of simulations of a STP circuit
  • FIG. IOC shows the results of simulations of a HP circuit in accordance with the present disclosure.
  • the present disclosure describes a neural circuit 10 that has a plurality of processing nodes 12, as shown in FIG. 1A, the processing nodes 12 may be arranged in an array and in one embodiment the array may be an array of 32 x 32 nodes for a total of 1024 nodes.
  • the processing nodes 12 may be
  • the neural circuit 10 includes electronic neuron and synapse circuits that have spiking signals at biological rates.
  • the electronic neurons and synapses may mimic multiple behaviors of biological neurons and synapses, including Short Term Plasticity (STP) , different types of Spike Timing Dependent Plasticity (STDP) , synaptic Kinetic Dynamics (KD) , neural Homeostatic Plasticity (HP) , and Axonal Delays (AD) .
  • STP Short Term Plasticity
  • STDP Spike Timing Dependent Plasticity
  • KD synaptic Kinetic Dynamics
  • HP neural Homeostatic Plasticity
  • AD Axonal Delays
  • a node 14 as shown in FIG. IB, is described that implements a synapse 16 and neuron 18 and includes a Short Term Plasticity (STP) circuit 20, a Spike Timing Dependent Plasticity (STDP) circuit 22, a synaptic Kinetic Dynamics (KD) circuit 24, a neural Homeostatic Plasticity (HP) circuit 26, and an Axonal Delay (AD) circuit 28.
  • the node 14 uses time multiplexing to reduce hardware complexity.
  • the STP circuit 20 has a time multiplexed circuit together with a memory 1 that is used to store STP states, as further described below.
  • the STDP circuit 22 also has a time multiplexed circuit together with the memory 1 that is used to store STDP states, as further described below.
  • the neural circuit 10 can be used to perform real-time low-power biological-type neural
  • the neural circuit 10 has an array of nodes 14 and data input/output circuitry 40.
  • FIG. IB shows a diagram of one node 14 of the array of nodes 14 in FIG. 1A.
  • the node 14 shown in FIG. IB has periphery circuitry and core circuitry 42.
  • the periphery circuitry includes an interconnect fabric 44 that may be used to send data from any node 14 to another node 14.
  • the interconnect fabric 44 may include interconnect buses 44 and routing switches. The routing switches, which are described in Reference [3], determine which node 14 at any one time is connected to another node 14, and this
  • the periphery circuitry also includes buses 46 that are used to power and bias the nodes 14 and buses 48 that are used to send initial parameter data into the memories 1, 2 and 3 of the core circuitry 42.
  • Memory 3 stores parameters that are used by the circuitry in the node 14, as further described below.
  • the buses 48 may also be used to monitor some of the internal signals of the core circuity 42.
  • the core circuitry 42 of the node 14 includes the synapse circuit 16, the STP (Short Term Plasticity) circuit 20, which adjusts the behavior of the synapse circuit 16, the STDP (Spike Timing Dependent Plasticity) circuit 22 that also adjusts the behavior of the synapse circuit 16, and a memory 1, which may also be a referred to as a weight memory that stores weights used by the synapse 16, the STP circuit 20, and the STDP circuit 22, as explained below.
  • STP Short Term Plasticity
  • STDP Sespike Timing Dependent Plasticity
  • the core circuitry 42 also includes the neuron circuit 18, the KD (Kinetic dynamics) circuit 24, which pre-processes signals that are provided into the neuron circuit 18, the HP (Homeostatic plasticity) circuit 26 that regulates the behavior of the neuron circuit 18, and the AD (Axonal Delay) circuit 28 that post-processes signals generated by the neuron circuit 18.
  • the core circuitry 42 further includes memory 2 that stores the connectivity states for the interconnect fabric 44, as further described in Reference [3], and memory 3 that stores parameters that regulate the behavior of the neuron circuit 18, the STP circuit 20, the STDP circuit 22, the synapse circuit 16, the KD circuit 24, the AD circuit 28, the HP circuit 26, and the Spike to Exponential (S-to-E) circuits 72 and 74. [0027] FIG.
  • FIG. 2 is a diagram showing the connectivity of different components of the core circuitry 42 of the node 14.
  • Data arriving from the interconnect fabric 44 is fed into the STDP circuit 22, and also into an Exponential to Spike (E-to- S) circuit 52.
  • the input 50 to the node 14 is an input signal P ⁇ that comes from the interconnect fabric 44.
  • the signal P ⁇ contains signals that each have a falling exponential tail.
  • the signal Pi is time multiplexed, and in each time slot the interconnect fabric 44 may be controlled via the routing switches so that in each time slot the signal Pi comes from the output P 73 from another node 14.
  • the E-to-S (Exponential to Spike) circuit 52 has an output 54 prei, which is connected to the synapse circuit 16, the STP circuit 20, and to the STDP circuit 22.
  • the E-to-S circuit 52 and the prei 54 are time multiplexed, in the same manner as Pi.
  • FIG. 4 shows a particular prei 54 spike, which is aligned in FIG. 4 with time slot 110 number 1.
  • time slot 110 there may or may not be any prei 54 spike, because the prei 54 is a result of P 73 from another node 14, which may or may not have had its neuron 18 spike, as further described below.
  • the synapse circuit 16 scales the time multiplexed prei 54 from the E-to-S circuit 52 on each time slot 110.
  • a synaptic gain or weight determines the amount of scaling.
  • This synaptic weight is equal to a product of a Wi and an Xi value.
  • the value Xi is calculated for each time slot 110 by the STP circuit 20, and the value of w ⁇ is calculated for each time slot 110 by the STDP circuit 22.
  • the time multiplexed outputs 60 of the synapse circuit 16 are scaled spike signals that are applied to the KD circuit 24.
  • the KD circuit 24 converts the scaled spike signals to step signals with a falling exponential tail.
  • the output 62 of the KD circuit 24 is input to a multiplier circuit 64 that is used to scale the output 62.
  • the amount of scaling is determined by the HP circuit 26, whose output 66 is connected to the multiplier circuit 64.
  • the output 68 of the multiplier circuit 64 is then input to the neuron circuit 18.
  • the neuron circuit 18 receives the time multiplexed input 68, which is the result of the time multiplexed signal Pi 50, and the time multiplexed STP 20, SDTP 22, synapse 16, KD 24, and multiplier 64 circuits, and produces an output 70, which is a spike-type signal when it occurs.
  • the output 70 of the neuron circuit 18 is not time multiplexed and may
  • the output 70 from the neuron circuit 18 is sent to the AD circuit 28, then to a S-to-E (Spike to Exponential) converter circuit 72 to produce output P 73, and finally to the interconnect fabric 44.
  • the output P 73 is an falling exponential type signal that is .1 to 700 milliseconds long depending on parameters in the S-to-E circuit 72.
  • the output 70 of the neuron circuit 18 is also sent to the HP circuit 26, which regulates the spike rate of the neuron circuit 18. Also, the output 70, which may be referred to as the post signal, of the neuron circuit 18 is sent to the STDP circuit 22, and to a S-to-E (Spike to Exponential) converter circuit 74, whose output 76 is sent to the STDP circuit 22.
  • S-to-E Spike to Exponential
  • the memory 1, shown in FIG. IB, may have two
  • MemorylB 78 in FIG. 2 stores the values xi that are used by the STP circuit 20 and the synapse 16.
  • the other part of memory 1 which is denoted as memorylA 80 in FIG. 2 stores the Wi values that are used by the STDP circuit 22 and the synapse 16.
  • Memory 2 shown in FIG. IB is similar to a connectivity memory described in Reference [3], and is used to control the routing switches in interconnect fabric 44.
  • Memory 3 shown in FIG. IB is used to locally store control parameters for the neuron 18, synapse 16, STP 20, STDP 22, neuron 18, KD 24, AD 28, and HP 26 circuits, and S-to-E circuit 72 and 74.
  • FIG. 3 shows a detail of the STP circuit 20 and associated STP memory 78, which may be MemorylB 78, shown in FIG. 2.
  • the STP computation circuit is composed of an adder 90 that receives the time multiplexed prei 54, a bias term -b 92 which is from memory 3, and data 94 read from STP memory 78.
  • the pre input 54 is a spiking signal that has only two
  • the data 94 read from the STP memory 78 encodes a positive value. It is stored digitally as a multi-bit word, which in one embodiment has 6 bits. One word 94 is read from STP memory 78 for each STP computation .
  • the output of the adder 90 is input to a multiplier 96, which multiplies the output of adder 90 by the parameter k 98.
  • the output of the multiplier 96 is input to an adder 100, which adds a bias term +b 102.
  • the value of b is the same for both -b 92 and +b 102.
  • the output of the adder 100 is a signal x ⁇ 56 that is provided to the synapse 16, and also written into STP memory 78 at the end of each computation, which occurs during each time slot 110, as shown in FIG. 4.
  • the STP memory 78 can store a number of words with each row of the memory storing one word, and each column of the memory storing different bits of the words, in the usual manner.
  • each row of the memory stores one word, and each column of the memory storing different bits of the words, in the usual manner.
  • the memory has an address that is denoted by slot address 104. In one embodiment the slot address 104 increments by 1 from 1 to 128 for each cycle 108 and then repeats.
  • the slot address 104 is kept constant, so that an x ⁇ 94 can be read from the memory 78, the STP computation performed, and the result Xi 56 written back into the memory 78, as shown in FIGs. 3 and 4. Then the slot address is incremented to a next value for the next computation.
  • a counter circuit 106 may be used to store and increment the slot address 104. When the counter reaches the maximum address value, which is 128 in the embodiment of FIG. 3, then the counter 106 is reset to the first value, which is 1 in the embodiment of the FIG. 3.
  • FIG. 4 shows a timing diagram of how the STP circuit 20 is operated. Time is divided into time slots 110. In one embodiment the duration of each slot is 8 microseconds. For each slot there is one operation of the STP circuit 20. As described above, during each time slot I, a value x ⁇ 94 is read from the STP memory 78, an updated value of x ⁇ 56 is derived using the STP circuit 20, and then the updated value of xi 56 is written and stored into STP memory 78 in the same STP memory address 104 from which the value x ⁇ 94 was read. In other words, the address 104 of the STP memory 78 stays the same for a time slot, so the updated value of x ⁇ 56 is written into the same memory address or slot address from which the x ⁇ 94 was read. The STP memory addresses are accessed
  • a group of 128 slots is defined as a cycle 108.
  • the duration of a cycle 108 is 1.024
  • FIG. 5A shows a typical waveform of the prei input 54 of the STP circuit 20 for a given time slot i with the
  • the output prei 54 of the E-to-S circuit 52 has only two possible values: a high value, which is denoted as VH, or a low value, which is denoted as VL . VL may be zero.
  • VH a high value
  • VL a low value
  • VL may be zero.
  • the time multiplexed prei 54 is valid for a particular time slot 110, as shown in FIG. 4, and updated once per cycle, which as described above may in one embodiment be 1.024 milliseconds long.
  • Neural information is encoded in the timing of the prei 54 spikes.
  • the interval between prei 54 spikes is
  • the interval between spikes may be ⁇ 20 milliseconds, but it can be less, for example 10 milliseconds, or higher, for example 100 milliseconds, with a typical 1 millisecond resolution in the actual value. It is also possible that a particular pre ⁇ 54 never has a spike.
  • the STP circuit 20 has two modes of operation. In one mode the STP circuit produces positive xi steps every time that there is a pre spike. In the second mode the STP circuit produces negative xi steps every time that there is a pre spike.
  • FIG. 5B shows a typical output of the STP circuit 20 for xi 56 for the STP circuit 20 operating in the first mode. Every time that there is a spike in the prei signal 54 then there a positive step in the signal xi 56. After the step, the signal x ⁇ 56 decays exponentially towards value b, unless another pre ⁇ signal 54 causes another positive step. When there is no step the amplitude of the signal x ⁇ 56 decays
  • the time scale for the plot of FIG. 5B is much longer than the time scale of FIG. 4.
  • the plots of FIGs. 5A, 5B and 5C span several 100' s of milliseconds, while the plot of FIG. 4 spans only 2 cycles 108 of 1.024 milliseconds each for the embodiment of 128 8 microsecond time slots 110. In the plot of FIG. 5B the individual time slots are not visible.
  • the Xi signal 56 of FIG. 5B is input to the synapse 16 only once in a particular time slot 110 i each cycle 108, or for the embodiment of FIGs. 3 and 4, once about every 1.024 milliseconds.
  • FIG. 5C shows a typical output of the STP circuit 20 for Xi 56 for the STP circuit 20 operating in the second mode. Every time that there is a spike in the prei signal 54 then there is a negative step in the signal Xi 56. When there is no step the amplitude of the signal x ⁇ 56 increases exponentially towards the value b.
  • FIG. 6 shows a diagram of one STDP circuit 22, which has two comparators 112 and 114, two adders 116 and 118, and an STDP memory 80, which may be memorylA 80, shown in FIG. 2.
  • the four data signals input to the STDP circuit 22 are: pre ⁇ 54, Pi 50, post 70, which is the output of neuron 18, and D 76, as shown in FIGs 2 and 6.
  • the signals pre ⁇ 54 and post 70 are spiking type signals.
  • the P ⁇ 50 and the D 76 signals consists of positive steps and falling exponentials.
  • the STDP circuit 22 produces the signal wi 58, which as described above is used as a weight in the synapse circuit 16.
  • the signals prei 54, ⁇ ⁇ 50, and w ⁇ 58 are time multiplexed, and are updated every time slot 110.
  • the signals post 70 and D 76 are updated every cycle 108.
  • the output 70 of the neuron circuit 18 is not time multiplexed and may typically occur at a rate of 10-100 Hz, and the rate may range from 0 Hz to less than 1 kHz.
  • the output 70 is input to S-to-E 74, the output of S-to-E 74 is D 76, which is a falling exponential type signal that may be .1 to 700 milliseconds long depending on parameters in the S-to-E circuit 74.
  • the first comparator 112 receives the input signal P 50 and a bias reference constant 120.
  • the first comparator 112 is enabled by the spiking signal post 70, and produces a signal ul 122.
  • the signal ul 122 is high when the signal P 50 is higher than the bias reference constant 120 and when there is a spike at the enable post signal 70.
  • Signal ul 122 is low or zero otherwise.
  • the signal ul 122 is only high when a step in Pi 50 has happened recently before a spike happens at enable post signal 70.
  • the second comparator 114 receives the input signal D 76 and a bias reference constant 124, which is the same as bias reference constant 120.
  • the second comparator 114 is enabled by the spiking signal pre ⁇ 54, and produces a signal u2 126.
  • the signal u2 126 is high when the signal D 76 is higher than the bias reference constant 124 and when there is a spike at enable signal pre ⁇ 54.
  • the signal u2 126 is low or zero otherwise.
  • the signal u2 126 is only high when a step in D 76 has happened recently before a spike happens at pre ⁇ 54.
  • the adder 116 sums the results of the two
  • the STDP memory 80 is used in one embodiment to store 128 weights for the 128 time slots.
  • the STDP memory 80 is denoted as memorylA 80 in FIG. 2.
  • the timing of addressing this memory is similar to the timing of the STP memory 78, as shown in FIG. 3.
  • the slot address 104 for the STDP memory 80 may be the same as used for the STP memory 78 in FIG. 3 and can be generated by the counter 106 in FIG. 3.
  • At every time slot 110 one weight wi 130 is read from the STDP memory 80 and added to the signal Awi 128 by adder 118.
  • the updated value of Wi 58 is given by:
  • Wi (updated) Wi(read from memory 80) + AWi (Equation 1)
  • the updated w ⁇ 58 is the output of the STDP circuit 22, and the updated w ⁇ 58 is also written back into the STDP memory 80.
  • FIG. 7A shows an example of waveforms of signals Pi 50, post 70, and Aw x 128 in one example node.
  • the index i is used for the Pi 50 and Aw x 128 signals to indicate that the signal is time multiplexed and that is only valid during each time slot i. Note that in the plot the Pi signal 50 appears continuous. However, the signal is produced only once each cycle, that is, once per each 1.024 milliseconds for the embodiment shown in FIGs. 3, 4 and 6, which each have 128 time slots .
  • the signal Pi 50 consists of positive edges followed by decaying or falling exponentials. The timing of the
  • positive steps corresponds to the times at which one pre ⁇ synaptic neuron circuit 18, which is a neuron circuit 18 located in a different node 14 and connected to the example node 14 by the interconnect fabric 44, is spiking.
  • the signal post 70 consists of spikes produced by a post-synaptic neuron circuit 18, which is the neuron circuit 18 located in the example node 14.
  • the signal Awi 128 is calculated by the circuitry shown in FIG. 6.
  • the signal Awi 128 is only updated when there is a spike in either the pre-synaptic neuron circuit 18 (enabled by pre ⁇ 54) or the post-synaptic neuron circuit 18 (enabled by post 70) .
  • the value of Awi 128 depends on the relative timing of spikes at the pre-synaptic neuron circuit 18 and the post-synaptic neuron circuit 18.
  • FIG. 7A is shown an example of the values of Awi 128 for cases in which two spikes happen at different times at the post signal 70. The first spike that happens at post
  • post-synaptic spike 70 in the example occurs shortly after the first positive step of signal Pi 50, which corresponds to a presynaptic spike from a pre-synaptic neuron circuit 18.
  • the value of Pi 50 is higher than bias 120.
  • the resulting Aw x is positive.
  • the second spike that happens at post 70 in the example occurs further after the first positive step of signal Pi 50.
  • the value of Pi 50 is lower than bias 120.
  • the resulting Aw x is negative.
  • FIG. 7B shows the output-input transfer
  • FIG. 7C shows the transfer function of the STP circuit 22 of FIG. 6.
  • the horizontal axis is the timing difference between a presynaptic and a postsynaptic spike.
  • the vertical axis is the weight change Awi 128 produced by the STDP circuit 22 of FIG. 6.
  • the weight change Awi 128 increases when the time difference is small, and the weight change Awi 128 decreases when the time difference is large.
  • the polarity of either one or both of the comparators 112 and 114 may be changed. This may change the STDP transfer characteristic from symmetrical to anti-symmetrical.
  • the Pi 50 input can be connected to the negative input of the comparator 112, rather than the positive input, and the bias 120 may be connected to the positive input of the comparator 112, rather than the negative input.
  • the STDP circuit 22 may also have exponential characteristics, as described in Reference [3].
  • FIG. 8A shows a detail of S-to-E circuits 72 and 74 shown in FIG. 2.
  • the "S-to-E” Spike-to-Exponential )
  • the "S-to-E" (Spike-to-Exponential) circuit of FIG. 8A has an adder 140, a multiplier 142, a parameter k 144, which is used to set the time constant, and a memory 146, which may be a register.
  • the parameter k 144 may be stored in memory 3 and typically k has a value slightly smaller than 1.
  • the adder 140 is connected to the input 138 and to a value read from memory 146, which may begin at zero. Then the adder output is multiplied by parameter k 144, and the result is written back into memory 146 and output.
  • FIG. 8B shows a detail of "E-to-S" (Exponential-to- Spike) circuit 52 shown in FIG. 2.
  • the "E-to-S” circuit converts positive steps with falling exponentials into spikes.
  • the bias 148 has a value slightly smaller than a maximum amplitude of the input signal 147.
  • FIG. 8C shows a detail of the synapse circuit 16.
  • the amplifier 61 has a variable gain, which is set by the output of multiplier 57, which multiplies x ⁇ 56 times Wi 58
  • the signals x ⁇ 56 and Wi 58 are produced by the STP 20 and STDP 22 circuits, respectively, as described above.
  • FIG. 8D shows a detail of the KD (kinetic dynamics) circuit 24. It is composed of a 1:N multiplexer 150, an array of N "S-to-E" circuits 152, and a summer 154. The number N may be typically 4. Each one of the S-to-E circuits 152 may be implemented as shown in FIG. 8A; however each S-to-E circuit 152 may have a different internal parameter k 144.
  • the kinetic dynamics circuit 24 is used convert spike signals 60 produced by the synapse circuit 16 of FIG. 8C into positive steps with falling exponentials with a set of N possible time decay constants. The kinetic dynamics circuit 24 is used to mimic the output behavior of multiple types of biological synapses.
  • FIG. 9A shows an embodiment of a neuron circuit 18, which as shown has an accumulator 160, which may have 9 bits, whose output is compared by a comparator 162 to a threshold Vth 164. As shown, if the sum of the accumulator 160 exceeds the threshold Vth 164, the neuron output 70 is a spike and the accumulator is reset.
  • the neuron input 68 is from all the time multiplexed inputs 68.
  • a spike at the neuron output 70 may occur at any time, or there may be no spike at all.
  • the rate of spikes at the output 70 may range from 0 Hz to less than 1 kHz; however, depending on the implementation of the neural circuit, the rate may be much higher.
  • An embodiment of a neuron circuit 18 is further described in Reference [3] .
  • FIG. 9B shows a detail of the HP (Homeostatic
  • Plasticity circuit 26 which is connected between the output 70 of the neuron circuit 18 and the multiplier 64, as shown in FIG. 2.
  • the HP circuit 26 has a digital or an analog low pass
  • LP filter 170 a comparator 172, a summer 174, and a bias term 176, which may have a value 1 in one embodiment.
  • Parameters for the low pass filter may be stored in memory 3.
  • the output 70 is filtered by the low pass filter 170.
  • the result is input to comparator 172 and compared to the ft a r ge t 173.
  • the output of the comparator 172 is negative when the output of the LP filter 170 exceeds the ftarget 173.
  • This allows the HP circuit 26 to regulate the spiking rate of the neuron circuit 18 to a rate defined by ftarget 173, which may be stored in memory 3.
  • Some parts of this circuit are similar to a circuit described in Reference [4]; however, the circuit of FIG. 9B is intended to drive a digital multiplier instead than an amplifier as described in Reference [4] .
  • FIG. 9C shows an AD (Axonal Delay) circuit 28.
  • the AD circuit 28 is connected to the output 70 of the neuron circuit 18, as shown in FIG. 2.
  • the AD circuit 28 has a set of digital delay stages 180 and a multiplexer (MUX) 182.
  • the AD circuit 28 of FIG. 9C produces delays that are aligned to a clock 184.
  • An AD circuit is also described in Reference [4]; however, the AD circuit described in Reference [4] produces asynchronous delays.
  • the AD circuit 28 delays the spikes produced by the neuron circuit 18, so that the neuron circuit 18 output is aligned to the cycle times indicated in FIG. 3, which makes the spike timing compatible with the time
  • the circuit of the invention may be implemented with many integrated circuit technologies, including CMOS.
  • CMOS complementary metal-oxide-semiconductor
  • a reduction to practice design has been made using 90nm CMOS. This design has over 1000 nodes, over 1000 neurons and over 100,000 synapses with STP, STDP, KD, HP and AD.
  • the neural circuit 10 of this 90nm design has an area of less than 2 cm 2 and a power of less than 1 Watt.
  • FIG. 10A shows an end-to-end simulation a network consisting of a 16 neurons receiving 16 external inputs. These 16 neurons are connected via 16 synapses with STDP and STP to an output neuron.
  • the neural circuit can determine which inputs are correlated to each other.
  • the top graph of FIG. 10A shows the 16 inputs to the chip, which are labeled as Pre 1 to Pre 16, and the spiking output post of the output neuron.
  • the bottom graph of FIG. 10A shows the weights wi*xi of the 16 synapses of the output neuron. All the weights of synapses receiving correlated inputs evolve to a high state, as
  • FIG. 10B shows a simulation of the STP (Short Term Plasticity) circuit 20, which is used to calculate a signal x that is used to further adjust the gain of a synapse according to the recent spikes received at the input of the synapse.
  • This STP circuit 20 is used in addition to the STDP (Spike Timing Dependent Plasticity) circuit 22.
  • the top graph of FIG. 10B shows the spike signal at the input of synapse.
  • the bottom graph of FIG. 10B shows the output signal produced by the STP circuit 22. This output signal consists of a step happening at the time of the input spike, followed by an exponential.
  • the transistor-level simulation matched the expected behaviors .
  • FIG. IOC shows a simulation of a HP (Homoestatic Plasticity) circuit 26.
  • the HP circuit 26 is used to adjust the spiking rate of a neuron circuit 18 to a given target rate.
  • the target rate is 50 Hz.
  • the top graph of FIG. IOC shows the spiking output of the neuron circuit.
  • the bottom graph of FIG. IOC shows actual measured spiking rate, and is close to the target rate as desired.
  • a circuit for emulating the behavior of biological neural circuits including a plurality of nodes wherein each node comprises a neuron circuit, a time multiplexed synapse circuit coupled to an input of the neuron circuit, a time multiplexed short term plasticity (STP) circuit coupled to an input of the node and to the synapse circuit, a time multiplexed Spike Timing Dependent Plasticity (STDP) circuit coupled to the input of the node and to the synapse circuit, an output of the node coupled to the neuron circuit; and an interconnect fabric coupled between the plurality of nodes for providing coupling from the output of any node of the
  • circuit comprising:
  • each node comprises:
  • a time multiplexed synapse circuit coupled to an input of the neuron circuit
  • STP short term plasticity
  • STDP Time multiplexed Spike Timing Dependent Plasticity
  • an interconnect fabric coupled between the plurality of nodes for providing coupling from the output of any node of the plurality of nodes to any input of any other node of the plurality of nodes.
  • time multiplexed STP circuit calculates a weight xi for the synapse circuit on each time slot i; and wherein the time multiplexed STDP circuit calculates ght wi for the synapse circuit on each time slot i.
  • the circuit of concept 1 further comprising: an exponential to spike (E-to-S) conversion circuit coupled between the input of the node and the synapse circuit, the STP circuit, and the STDP circuit.
  • E-to-S exponential to spike
  • STP memory has N addresses
  • the STP computation circuit comprises:
  • a stored parameter b coupled to the adder; wherein the first adder adds the input, subtracts the parameter b, and adds a value xi read from address i of the STP memory for time slot i;
  • a second adder for adding a product of the multiplier and the parameter b to produce a result x ⁇ for time slot i;
  • the STDP memory has N addresses; and wherein the STDP computation circuit comprises: a first input coupled to the input of the node;
  • S-to-E spike to exponential
  • a first comparator having a first input coupled to the input of the node, a second input coupled to a stored bias parameter, and an enable input coupled to the output of the neuron circuit, wherein the enable input enables a first comparator output;
  • a second comparator having a first input coupled to an output of the first S-to-E conversion circuit, a second input coupled to the stored bias parameter, and an enable input coupled to an output of the E-to-S conversion circuit, wherein the enable input enables a second comparator output;
  • a second adder for adding an output of the first adder and a value w ⁇ read from address i of the STDP memory for time slot i to produce a result wi for time slot i;
  • the first input of the first comparator is coupled to a positive input of the first comparator
  • the second input of the first comparator is coupled to a negative input of the first comparator
  • the first input of the second comparator is coupled to a positive input of the second comparator
  • the first input of the first comparator is coupled to a negative input of the first comparator
  • the second input of the first comparator is coupled to a positive input of the first comparator
  • the first input of the second comparator is coupled to a positive input of the second comparator
  • the second input of the second comparator is coupled to a negative input of the second comparator.
  • the first input of the first comparator is coupled to a negative input of the first comparator
  • the second input of the first comparator is coupled to a positive input of the first comparator
  • the first input of the second comparator is coupled to a negative input of the second comparator
  • the second input of the second comparator is coupled to a positive input of the second comparator.
  • a multiplier for multiplying Xi from the STP circuit times the Wi from the STDP circuit to calculate a product x ⁇ Wi; and a variable gain amplifier that has a gain that depends on the product XiWi .
  • a comparator for comparing an output of the accumulator to a stored threshold parameter; wherein if the output of the accumulator is greater than the threshold parameter, the output of the neuron circuit is spike and the accumulator is reset to zero.
  • an adder coupled to the input of the node and to the memory for adding the input on time slot i to a value read from the memory
  • a multiplier for multiplying an output of the adder and stored parameter k for calculating a product
  • the circuit of concept 1 further comprising: a kinetic dynamics (KD) circuit coupled to the output of the synapse circuit;
  • KD kinetic dynamics
  • HP homeostatic plasticity
  • a multiplier coupled to an output of the KD circuit an output of the HP circuit for multiplying the output of the KD circuit times the output of the HP circuit
  • the homeostatic plasticity (HP) circuit comprises:
  • a low pass filter coupled to the output of the neuron circuit
  • a comparator coupled to an output of the low pass filter and to a stored parameter f target and an adder coupled the an output of the comparator for adding the output of the comparator and a second stored parameter
  • the KD circuit comprises:
  • a 1 to J multiplexer coupled to the output of the synapse circuit
  • J spike to exponential (S-to-E) conversion circuits coupled to an output of 1 to J multiplexer, wherein each of the J spike to exponential (S-to-E) conversion circuits may have a different time decay constant;
  • the circuit of concept 1 further comprising: an axonal delay (AD) circuit coupled to the output of the neuron circuit; and
  • AD axonal delay
  • S-to-E spike to exponential
  • each delay element has an output and wherein each delay element is coupled to a clock that is aligned to the time slots; and a multiplexer coupled to the output of each delay
  • STP short term plasticity
  • STDP Spike Timing Dependent Plasticity
  • time multiplexed STDP circuit calculates a weight wi for the synapse circuit on each time slot i.
  • the method of concept 17 further comprising: converting the input of the node from an exponential to a spike using an exponential to spike (E-to-S) conversion circuit coupled between the input of the node and the synapse circuit, the STP circuit, and the STDP circuit.
  • E-to-S exponential to spike
  • STP memory has N addresses
  • the STDP memory has N addresses
  • the STDP computation circuit compares a first input coupled to the input of the node to a second input coupled to a stored bias parameter, wherein a first enable input coupled to the output of the neuron circuit enables a first comparison output;
  • HP homeostatic plasticity
  • Providing the homeostatic plasticity (HP) circuit comprises :
  • circuits may have a different time decay constant
  • concept 27 The method of concept 17 further comprising: providing an axonal delay (AD) coupled to the output of the neuron circuit; and

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computational Mathematics (AREA)
  • Pure & Applied Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Mathematical Optimization (AREA)
  • Neurology (AREA)
  • Complex Calculations (AREA)
  • Image Analysis (AREA)
  • Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
  • Logic Circuits (AREA)

Abstract

A circuit for emulating the behavior of biological neural circuits, the circuit including a plurality of nodes wherein each node comprises a neuron circuit, a time multiplexed synapse circuit coupled to an input of the neuron circuit, a time multiplexed short term plasticity (STP) circuit coupled to an input of the node and to the synapse circuit, a time multiplexed Spike Timing Dependent Plasticity (STDP) circuit coupled to the input of the node and to the synapse circuit, an output of the node coupled to the neuron circuit; and an interconnect fabric coupled between the plurality of nodes for providing coupling from the output of any node of the plurality of nodes to any input of any other node of the plurality of nodes.

Description

NEURAL INTEGRATED CIRCUI T WI TH BIOLOGICAL BEHAVIORS
CROSS REFERENCE TO RELATED APPLICATIONS
[ 0001 ] This application claims priority to and claims the benefit of U.S. S.N. 15/199,800 filed June 30, 2016, which is incorporated herein by reference.
[ 0002 ] This application is related to U.S. Patent
Application 14/453,154, filed 8/6/2014 and U.S. Patent
Application 13/679,727, filed 11/16/2012, which is now U.S. Patent No. 8,996,431, issued 3/31/2015, which are incorporated herein as though set forth in full.
STATEMENT REGARDING FEDERAL FUNDING
[ 0003 ] This invention was made under U.S. Government contract HRL0011-09-C-001. The U.S. Government has certain rights in this invention.
TECHNICAL FIELD
[ 0004 ] This disclosure relates to neural network circuits.
BACKGROUND
[ 0005 ] In the prior art, circuits have been used to emulate the behavior of biological neural circuits. In Reference [1] below, which is incorporated herein by reference, models of neural circuits are given. However, no circuit implementation is presented. In References [2], [3] and [4] below, which are incorporated herein by reference, circuits are described that implement neurons and synapses. However the circuits of References [2] and [3] only include some biological behaviors. In Reference [4] below, which is incorporated herein by reference, some biological behaviors are presented such as homeostatic plasticity (HP) , kinetic dynamics (KD) , and axonal delay (AD) in a single neuron; however, Reference [4] does not include synapses with Short Term Plasticity (STP) and Spike Timing Dependent Plasticity (STDP) , and does not integrate behaviors in a circuit with multiple neurons and synapses.
[0006] References
[0007] The following references are incorporated herein by reference .
[1] E. M. Izhikevich and G. M. Edelman, "Large-scale model of mammalian thalamocortical systems (Supporting
Information, Appendix) ", Proceedings National Academy of
Sciences (PNAS), vo . 105, no. 9, pp. 3593-3598, 2008.
[2] J. M. Cruz-Albrecht , T. Derosier, and N. Srinivasa, "A scalable neural chip with synaptic electronics using CMOS integrated memristors," Nanotechnology, vo . 24, no. 38, Sep 2013.
[3] J. Cruz-Albrecht, T. Derosier, and N. Srinivasa, U.S. Patent Application 14/453,154, filed 8/6/2014, titled "Scalable Integrated Circuit with Synaptic Electronics and CMOS integrated Memristors".
[4] J. Cruz-Albrecht, M. Yung and N. Srinivasa, U.S.
Patent Application 13/679,727, filed 11/16/2012, which is now U.S. Patent No. 8,996,431, issued 3/31/2015, titled "Spike Domain Neuron Circuit with Programmable Kinetic Dynamics, Homeostatic Plasticity and Axonal Delays".
[0008] What is needed is an electronic circuit that is compatible with integrated circuit technology and that
emulates the behavior of biological neural circuits. The embodiments of the present disclosure answer these and other needs .
SUMMARY
[0009] In a first embodiment disclosed herein, a circuit for emulating the behavior of biological neural circuits, the circuit comprises a plurality of nodes wherein each node comprises a neuron circuit, a time multiplexed synapse circuit coupled to an input of the neuron circuit, a time multiplexed short term plasticity (STP) circuit coupled to an input of the node and to the synapse circuit, a time multiplexed Spike Timing Dependent Plasticity (STDP) circuit coupled to the input of the node and to the synapse circuit, and an output of the node coupled to the neuron circuit, and an interconnect fabric coupled between the plurality of nodes for providing coupling from the output of any node of the plurality of nodes to any input of any other node of the plurality of nodes
[0010] In another embodiment disclosed herein, a method for emulating the behavior of biological neural circuits comprises providing a plurality of nodes, providing a neuron circuit in each node, providing a synapse circuit in each node coupled to an input of the neuron circuit, providing a short term
plasticity (STP) circuit in each node coupled to an input of the node and to the synapse circuit, providing a Spike Timing Dependent Plasticity (STDP) circuit in each node coupled to the input of the node and to the synapse circuit, providing an output of the node coupled to the neuron circuit, time
multiplexing the synapse, the STP, and the STDP circuits; and providing an interconnect fabric coupled between the plurality of nodes for providing coupling from the output of any node of the plurality of nodes to any input of any other node of the plurality of nodes. [0011] These and other features and advantages will become further apparent from the detailed description and
accompanying figures that follow. In the figures and
description, numerals indicate the various features, like numerals referring to like features throughout both the drawings and the description.
BRIEF DESCRIPTION OF THE DRAWINGS
[0012] FIG. 1A shows a top level diagram of neural circuit, and FIG. IB shows a diagram of a node in accordance with the present disclosure;
[0013] FIG. 2 shows a diagram of the core of a node and connections among the components of the node in accordance with the present disclosure;
[0014] FIG. 3 shows a detail of the short term plasticity (STP) circuit with time multiplexing in accordance with the present disclosure;
[0015] FIG. 4 shows a timing diagram of the STP circuit and shows one example prei spike in accordance with the present disclosure ;
[0016] FIG. 5A shows a typical waveform at the input of the input of the STP circuit for a given slot i, FIG. 5B shows a typical output of the circuit for signal xi for the circuit operating in a first mode, and FIG. 5C shows a typical output of the circuit for xi for the circuit operating in a second mode in accordance with the present disclosure;
[0017] FIG. 6 shows a detail of one embodiment of an spike timing dependent plasticity (STDP) circuit in accordance with the present disclosure; [0018] FIG. 7A shows typical waveforms of input signals and of an Awi signal generated by the STDP circuit, FIG. 7B shows an transfer characteristic of the internal comparator, and FIG. 7C shows an overall STDP transfer characteristic in accordance with the present disclosure;
[0019] FIG. 8A shows a detail of a "S-to-E" (Spike-to- Exponential) circuit, FIG. 8B shows a detail of a "E-to-S"
(Exponential-to-Spike ) circuit, FIG. 8C shows a detail of the synapse, and FIG. 8D shows a detail of a kinetic dynamics (KD) circuit in accordance with the present disclosure;
[0020] FIG. 9A shows a neuron, FIG. 9B shows a homeostatic plasticity (HP) circuit, and FIG. 9C shows an Axonal Delay (AD) circuit in accordance with the present disclosure; and
[0021] FIG. 10A shows the results of simulations of a network with neurons and synapses, FIG. 10B shows the results of simulations of a STP circuit, and FIG. IOC shows the results of simulations of a HP circuit in accordance with the present disclosure.
DETAILED DESCRIPTION
[0022] In the following description, numerous specific details are set forth to clearly describe various specific embodiments disclosed herein. One skilled in the art, however, will understand that the presently claimed invention may be practiced without all of the specific details discussed below. In other instances, well known features have not been
described so as not to obscure the invention.
[0023] The present disclosure describes a neural circuit 10 that has a plurality of processing nodes 12, as shown in FIG. 1A, the processing nodes 12 may be arranged in an array and in one embodiment the array may be an array of 32 x 32 nodes for a total of 1024 nodes. The processing nodes 12 may be
arranged in any arbitrary topology, rather than the array shown in FIG. 1A. The neural circuit 10 includes electronic neuron and synapse circuits that have spiking signals at biological rates. The electronic neurons and synapses may mimic multiple behaviors of biological neurons and synapses, including Short Term Plasticity (STP) , different types of Spike Timing Dependent Plasticity (STDP) , synaptic Kinetic Dynamics (KD) , neural Homeostatic Plasticity (HP) , and Axonal Delays (AD) . The neural circuit 10 saves hardware cost and complexity by using time multiplexing.
[0024] A node 14, as shown in FIG. IB, is described that implements a synapse 16 and neuron 18 and includes a Short Term Plasticity (STP) circuit 20, a Spike Timing Dependent Plasticity (STDP) circuit 22, a synaptic Kinetic Dynamics (KD) circuit 24, a neural Homeostatic Plasticity (HP) circuit 26, and an Axonal Delay (AD) circuit 28. The node 14 uses time multiplexing to reduce hardware complexity. The STP circuit 20 has a time multiplexed circuit together with a memory 1 that is used to store STP states, as further described below. The STDP circuit 22 also has a time multiplexed circuit together with the memory 1 that is used to store STDP states, as further described below. The neural circuit 10 can be used to perform real-time low-power biological-type neural
computations, and can implement over 1000 neurons and over 100,000 synapses.
[0025] The neural circuit 10 has an array of nodes 14 and data input/output circuitry 40. FIG. IB shows a diagram of one node 14 of the array of nodes 14 in FIG. 1A. The node 14 shown in FIG. IB has periphery circuitry and core circuitry 42. The periphery circuitry includes an interconnect fabric 44 that may be used to send data from any node 14 to another node 14. The interconnect fabric 44 may include interconnect buses 44 and routing switches. The routing switches, which are described in Reference [3], determine which node 14 at any one time is connected to another node 14, and this
connectivity is determined by connectivity data in memory 2. The periphery circuitry also includes buses 46 that are used to power and bias the nodes 14 and buses 48 that are used to send initial parameter data into the memories 1, 2 and 3 of the core circuitry 42. Memory 3 stores parameters that are used by the circuitry in the node 14, as further described below. The buses 48 may also be used to monitor some of the internal signals of the core circuity 42.
[ 0026 ] The core circuitry 42 of the node 14 includes the synapse circuit 16, the STP (Short Term Plasticity) circuit 20, which adjusts the behavior of the synapse circuit 16, the STDP (Spike Timing Dependent Plasticity) circuit 22 that also adjusts the behavior of the synapse circuit 16, and a memory 1, which may also be a referred to as a weight memory that stores weights used by the synapse 16, the STP circuit 20, and the STDP circuit 22, as explained below. The core circuitry 42 also includes the neuron circuit 18, the KD (Kinetic dynamics) circuit 24, which pre-processes signals that are provided into the neuron circuit 18, the HP (Homeostatic plasticity) circuit 26 that regulates the behavior of the neuron circuit 18, and the AD (Axonal Delay) circuit 28 that post-processes signals generated by the neuron circuit 18. The core circuitry 42 further includes memory 2 that stores the connectivity states for the interconnect fabric 44, as further described in Reference [3], and memory 3 that stores parameters that regulate the behavior of the neuron circuit 18, the STP circuit 20, the STDP circuit 22, the synapse circuit 16, the KD circuit 24, the AD circuit 28, the HP circuit 26, and the Spike to Exponential (S-to-E) circuits 72 and 74. [0027] FIG. 2 is a diagram showing the connectivity of different components of the core circuitry 42 of the node 14. Data arriving from the interconnect fabric 44 is fed into the STDP circuit 22, and also into an Exponential to Spike (E-to- S) circuit 52. The input 50 to the node 14 is an input signal P± that comes from the interconnect fabric 44. The signal P± contains signals that each have a falling exponential tail. The signal Pi is time multiplexed, and in each time slot the interconnect fabric 44 may be controlled via the routing switches so that in each time slot the signal Pi comes from the output P 73 from another node 14.
[0028] For example, the interconnect fabric 44 may be controlled so that during a cycle 108, as shown in FIG. 4, a particular node 14 receives a Pi for i = 1 to 128 from each of 128 other nodes 14. Each P± is sent on the interconnect fabric 44 in a time slot 110, as shown in FIG. 4.
[0029] The E-to-S (Exponential to Spike) circuit 52 has an output 54 prei, which is connected to the synapse circuit 16, the STP circuit 20, and to the STDP circuit 22. The E-to-S circuit 52 and the prei 54 are time multiplexed, in the same manner as Pi. FIG. 4 shows a particular prei 54 spike, which is aligned in FIG. 4 with time slot 110 number 1. In any particular time slot 110 there may or may not be any prei 54 spike, because the prei 54 is a result of P 73 from another node 14, which may or may not have had its neuron 18 spike, as further described below.
[0030] The synapse circuit 16 scales the time multiplexed prei 54 from the E-to-S circuit 52 on each time slot 110. A synaptic gain or weight determines the amount of scaling.
This synaptic weight is equal to a product of a Wi and an Xi value. The value Xi is calculated for each time slot 110 by the STP circuit 20, and the value of w± is calculated for each time slot 110 by the STDP circuit 22. [0031] The time multiplexed outputs 60 of the synapse circuit 16 are scaled spike signals that are applied to the KD circuit 24. The KD circuit 24 converts the scaled spike signals to step signals with a falling exponential tail. The output 62 of the KD circuit 24 is input to a multiplier circuit 64 that is used to scale the output 62. The amount of scaling is determined by the HP circuit 26, whose output 66 is connected to the multiplier circuit 64. The output 68 of the multiplier circuit 64 is then input to the neuron circuit 18.
[0032] The neuron circuit 18 receives the time multiplexed input 68, which is the result of the time multiplexed signal Pi 50, and the time multiplexed STP 20, SDTP 22, synapse 16, KD 24, and multiplier 64 circuits, and produces an output 70, which is a spike-type signal when it occurs. The output 70 of the neuron circuit 18 is not time multiplexed and may
typically occur at a rate of 10-100 Hz or there may be no output spike at all. The rate may range from 0 Hz to less than 1 kHz. The output 70 from the neuron circuit 18 is sent to the AD circuit 28, then to a S-to-E (Spike to Exponential) converter circuit 72 to produce output P 73, and finally to the interconnect fabric 44. The output P 73 is an falling exponential type signal that is .1 to 700 milliseconds long depending on parameters in the S-to-E circuit 72.
[0033] The output 70 of the neuron circuit 18 is also sent to the HP circuit 26, which regulates the spike rate of the neuron circuit 18. Also, the output 70, which may be referred to as the post signal, of the neuron circuit 18 is sent to the STDP circuit 22, and to a S-to-E (Spike to Exponential) converter circuit 74, whose output 76 is sent to the STDP circuit 22.
[0034] The memory 1, shown in FIG. IB, may have two
subparts. One part, which is denoted as memorylB 78 in FIG. 2, stores the values xi that are used by the STP circuit 20 and the synapse 16. The other part of memory 1, which is denoted as memorylA 80 in FIG. 2, stores the Wi values that are used by the STDP circuit 22 and the synapse 16. Memory 2, shown in FIG. IB, is similar to a connectivity memory described in Reference [3], and is used to control the routing switches in interconnect fabric 44. Memory 3 shown in FIG. IB is used to locally store control parameters for the neuron 18, synapse 16, STP 20, STDP 22, neuron 18, KD 24, AD 28, and HP 26 circuits, and S-to-E circuit 72 and 74.
[0035] FIG. 3 shows a detail of the STP circuit 20 and associated STP memory 78, which may be MemorylB 78, shown in FIG. 2. The STP computation circuit is composed of an adder 90 that receives the time multiplexed prei 54, a bias term -b 92 which is from memory 3, and data 94 read from STP memory 78. The pre input 54 is a spiking signal that has only two
possible values: high or zero. The bias -b 92 is a constant. In one embodiment b=l, so -b equals -1. The data 94 read from the STP memory 78 encodes a positive value. It is stored digitally as a multi-bit word, which in one embodiment has 6 bits. One word 94 is read from STP memory 78 for each STP computation .
[0036] The output of the adder 90 is input to a multiplier 96, which multiplies the output of adder 90 by the parameter k 98. The value of k 98 is typically a number slightly less than 1. In one embodiment k=0.9. The output of the multiplier 96 is input to an adder 100, which adds a bias term +b 102. The value of b is the same for both -b 92 and +b 102. The output of the adder 100 is a signal x± 56 that is provided to the synapse 16, and also written into STP memory 78 at the end of each computation, which occurs during each time slot 110, as shown in FIG. 4.
[0037] The STP memory 78 can store a number of words with each row of the memory storing one word, and each column of the memory storing different bits of the words, in the usual manner. In the embodiment shown in FIG. 3 stores 128 words, which corresponds to the example above in which the output P 73 of 128 nodes 14 are time multiplexed to provide P± 50 for I = 1 to 128 to a particular node 14 on each cycle 108. The memory has an address that is denoted by slot address 104. In one embodiment the slot address 104 increments by 1 from 1 to 128 for each cycle 108 and then repeats. For each time slot 110 the slot address 104 is kept constant, so that an x± 94 can be read from the memory 78, the STP computation performed, and the result Xi 56 written back into the memory 78, as shown in FIGs. 3 and 4. Then the slot address is incremented to a next value for the next computation. In one embodiment a counter circuit 106 may be used to store and increment the slot address 104. When the counter reaches the maximum address value, which is 128 in the embodiment of FIG. 3, then the counter 106 is reset to the first value, which is 1 in the embodiment of the FIG. 3.
[ 0038 ] FIG. 4 shows a timing diagram of how the STP circuit 20 is operated. Time is divided into time slots 110. In one embodiment the duration of each slot is 8 microseconds. For each slot there is one operation of the STP circuit 20. As described above, during each time slot I, a value x± 94 is read from the STP memory 78, an updated value of x± 56 is derived using the STP circuit 20, and then the updated value of xi 56 is written and stored into STP memory 78 in the same STP memory address 104 from which the value x± 94 was read. In other words, the address 104 of the STP memory 78 stays the same for a time slot, so the updated value of x± 56 is written into the same memory address or slot address from which the x± 94 was read. The STP memory addresses are accessed
sequentially and always in the same order. In the embodiment shown in FIGs. 3 and 4, a group of 128 slots is defined as a cycle 108. In the embodiment with 128 time slots 110 of 8 microseconds, the duration of a cycle 108 is 1.024
milliseconds .
[0039] A person skilled in the art would understand that other embodiment may have a different number and duration of time slots, with the resulting cycle time being the product of the number of time slots multipled by the time slot duration. Each time slot has the same duration.
[0040] FIG. 5A shows a typical waveform of the prei input 54 of the STP circuit 20 for a given time slot i with the
horizontal axis being time and the vertical axis being
amplitude. The output prei 54 of the E-to-S circuit 52 has only two possible values: a high value, which is denoted as VH, or a low value, which is denoted as VL . VL may be zero. In the STP circuit 20, the time multiplexed prei 54 is valid for a particular time slot 110, as shown in FIG. 4, and updated once per cycle, which as described above may in one embodiment be 1.024 milliseconds long.
[0041] Neural information is encoded in the timing of the prei 54 spikes. The interval between prei 54 spikes is
typically not constant, but rather changes. On average, the interval between spikes may be ~20 milliseconds, but it can be less, for example 10 milliseconds, or higher, for example 100 milliseconds, with a typical 1 millisecond resolution in the actual value. It is also possible that a particular pre± 54 never has a spike.
[0042] The STP circuit 20 has two modes of operation. In one mode the STP circuit produces positive xi steps every time that there is a pre spike. In the second mode the STP circuit produces negative xi steps every time that there is a pre spike.
[0043] FIG. 5B shows a typical output of the STP circuit 20 for xi 56 for the STP circuit 20 operating in the first mode. Every time that there is a spike in the prei signal 54 then there a positive step in the signal xi 56. After the step, the signal x± 56 decays exponentially towards value b, unless another pre± signal 54 causes another positive step. When there is no step the amplitude of the signal x± 56 decays
exponentially towards value b.
[0044] The time scale for the plot of FIG. 5B is much longer than the time scale of FIG. 4. The plots of FIGs. 5A, 5B and 5C span several 100' s of milliseconds, while the plot of FIG. 4 spans only 2 cycles 108 of 1.024 milliseconds each for the embodiment of 128 8 microsecond time slots 110. In the plot of FIG. 5B the individual time slots are not visible.
In the plot of FIG. 5B the xi signal 56 appears as a
continuous waveform. However, the Xi signal 56 of FIG. 5B is input to the synapse 16 only once in a particular time slot 110 i each cycle 108, or for the embodiment of FIGs. 3 and 4, once about every 1.024 milliseconds.
[0045] FIG. 5C shows a typical output of the STP circuit 20 for Xi 56 for the STP circuit 20 operating in the second mode. Every time that there is a spike in the prei signal 54 then there is a negative step in the signal Xi 56. When there is no step the amplitude of the signal x± 56 increases exponentially towards the value b.
[0046] FIG. 6 shows a diagram of one STDP circuit 22, which has two comparators 112 and 114, two adders 116 and 118, and an STDP memory 80, which may be memorylA 80, shown in FIG. 2. The four data signals input to the STDP circuit 22 are: pre± 54, Pi 50, post 70, which is the output of neuron 18, and D 76, as shown in FIGs 2 and 6. The signals pre± 54 and post 70 are spiking type signals. The P± 50 and the D 76 signals consists of positive steps and falling exponentials. The STDP circuit 22 produces the signal wi 58, which as described above is used as a weight in the synapse circuit 16. The signals prei 54, Ρ± 50, and w± 58 are time multiplexed, and are updated every time slot 110. The signals post 70 and D 76 are updated every cycle 108. As discussed above, the output 70 of the neuron circuit 18 is not time multiplexed and may typically occur at a rate of 10-100 Hz, and the rate may range from 0 Hz to less than 1 kHz. The output 70 is input to S-to-E 74, the output of S-to-E 74 is D 76, which is a falling exponential type signal that may be .1 to 700 milliseconds long depending on parameters in the S-to-E circuit 74.
[0047] The first comparator 112 receives the input signal P 50 and a bias reference constant 120. The first comparator 112 is enabled by the spiking signal post 70, and produces a signal ul 122. The signal ul 122 is high when the signal P 50 is higher than the bias reference constant 120 and when there is a spike at the enable post signal 70. Signal ul 122 is low or zero otherwise. The signal ul 122 is only high when a step in Pi 50 has happened recently before a spike happens at enable post signal 70.
[0048] The second comparator 114 receives the input signal D 76 and a bias reference constant 124, which is the same as bias reference constant 120. The second comparator 114 is enabled by the spiking signal pre± 54, and produces a signal u2 126. The signal u2 126 is high when the signal D 76 is higher than the bias reference constant 124 and when there is a spike at enable signal pre± 54. The signal u2 126 is low or zero otherwise. The signal u2 126 is only high when a step in D 76 has happened recently before a spike happens at pre± 54.
[0049] The adder 116 sums the results of the two
comparators 112 and 114, and the output of the adder 116 is the signal Awi 128, which represents a change in a weight wi .
[0050] The STDP memory 80 is used in one embodiment to store 128 weights for the 128 time slots. The STDP memory 80 is denoted as memorylA 80 in FIG. 2. The timing of addressing this memory is similar to the timing of the STP memory 78, as shown in FIG. 3. The slot address 104 for the STDP memory 80 may be the same as used for the STP memory 78 in FIG. 3 and can be generated by the counter 106 in FIG. 3. At every time slot 110 one weight wi 130 is read from the STDP memory 80 and added to the signal Awi 128 by adder 118. The updated value of Wi 58 is given by:
[0051] Wi (updated) = Wi(read from memory 80) + AWi (Equation 1)
[0052] The updated w± 58 is the output of the STDP circuit 22, and the updated w± 58 is also written back into the STDP memory 80.
[0053] FIG. 7A shows an example of waveforms of signals Pi 50, post 70, and Awx 128 in one example node. The index i is used for the Pi 50 and Awx 128 signals to indicate that the signal is time multiplexed and that is only valid during each time slot i. Note that in the plot the Pi signal 50 appears continuous. However, the signal is produced only once each cycle, that is, once per each 1.024 milliseconds for the embodiment shown in FIGs. 3, 4 and 6, which each have 128 time slots .
[0054] The signal Pi 50 consists of positive edges followed by decaying or falling exponentials. The timing of the
positive steps corresponds to the times at which one pre¬ synaptic neuron circuit 18, which is a neuron circuit 18 located in a different node 14 and connected to the example node 14 by the interconnect fabric 44, is spiking.
[0055] The signal post 70 consists of spikes produced by a post-synaptic neuron circuit 18, which is the neuron circuit 18 located in the example node 14.
[0056] The signal Awi 128 is calculated by the circuitry shown in FIG. 6. The signal Awi 128 is only updated when there is a spike in either the pre-synaptic neuron circuit 18 (enabled by pre± 54) or the post-synaptic neuron circuit 18 (enabled by post 70) . The value of Awi 128 depends on the relative timing of spikes at the pre-synaptic neuron circuit 18 and the post-synaptic neuron circuit 18.
[0057] In FIG. 7A is shown an example of the values of Awi 128 for cases in which two spikes happen at different times at the post signal 70. The first spike that happens at post
(post-synaptic spike) 70 in the example occurs shortly after the first positive step of signal Pi 50, which corresponds to a presynaptic spike from a pre-synaptic neuron circuit 18.
At the time of the first spike of post 70 the value of Pi 50 is higher than bias 120. The resulting Awx is positive. The second spike that happens at post 70 in the example occurs further after the first positive step of signal Pi 50. At the time of the second spike of post the value of Pi 50 is lower than bias 120. The resulting Awx is negative.
[0058] FIG. 7B shows the output-input transfer
characteristic of the comparators 112 and 114 of the STDP circuit 22 shown in FIG. 6.
[0059] FIG. 7C shows the transfer function of the STP circuit 22 of FIG. 6. The horizontal axis is the timing difference between a presynaptic and a postsynaptic spike. The vertical axis is the weight change Awi 128 produced by the STDP circuit 22 of FIG. 6. In this embodiment of a STDP circuit 22 the weight change Awi 128 increases when the time difference is small, and the weight change Awi 128 decreases when the time difference is large.
[0060] In another embodiment of the STDP circuit 22 of FIG. 6 the polarity of either one or both of the comparators 112 and 114 may be changed. This may change the STDP transfer characteristic from symmetrical to anti-symmetrical. For example the Pi 50 input can be connected to the negative input of the comparator 112, rather than the positive input, and the bias 120 may be connected to the positive input of the comparator 112, rather than the negative input.
[0061] The STDP circuit 22 may also have exponential characteristics, as described in Reference [3].
[0062] FIG. 8A shows a detail of S-to-E circuits 72 and 74 shown in FIG. 2. The "S-to-E" ( Spike-to-Exponential )
circuits 72 and 74 convert spikes into positive steps followed by falling exponentials. The "S-to-E" (Spike-to-Exponential) circuit of FIG. 8A has an adder 140, a multiplier 142, a parameter k 144, which is used to set the time constant, and a memory 146, which may be a register. The parameter k 144 may be stored in memory 3 and typically k has a value slightly smaller than 1. The adder 140 is connected to the input 138 and to a value read from memory 146, which may begin at zero. Then the adder output is multiplied by parameter k 144, and the result is written back into memory 146 and output.
[0063] FIG. 8B shows a detail of "E-to-S" (Exponential-to- Spike) circuit 52 shown in FIG. 2. The "E-to-S" circuit converts positive steps with falling exponentials into spikes. Typically the bias 148 has a value slightly smaller than a maximum amplitude of the input signal 147.
[0064] FIG. 8C shows a detail of the synapse circuit 16. The amplifier 61 has a variable gain, which is set by the output of multiplier 57, which multiplies x± 56 times Wi 58 The signals x± 56 and Wi 58 are produced by the STP 20 and STDP 22 circuits, respectively, as described above. The output signal 60 of the synapse circuit 16 is time multiplexed and updated in every time slot 110 i, which as described above may be for I = 1 to 128 for one embodiment. This time multiplexing allows a single synapse circuit 16 in a node 14 to emulate the behavior of multiple virtual synapses each driven by the output P 73 of neurons 18 in other nodes 14, which in one embodiment may number 128. [ 0065 ] FIG. 8D shows a detail of the KD (kinetic dynamics) circuit 24. It is composed of a 1:N multiplexer 150, an array of N "S-to-E" circuits 152, and a summer 154. The number N may be typically 4. Each one of the S-to-E circuits 152 may be implemented as shown in FIG. 8A; however each S-to-E circuit 152 may have a different internal parameter k 144. The kinetic dynamics circuit 24 is used convert spike signals 60 produced by the synapse circuit 16 of FIG. 8C into positive steps with falling exponentials with a set of N possible time decay constants. The kinetic dynamics circuit 24 is used to mimic the output behavior of multiple types of biological synapses.
[ 0066 ] FIG. 9A shows an embodiment of a neuron circuit 18, which as shown has an accumulator 160, which may have 9 bits, whose output is compared by a comparator 162 to a threshold Vth 164. As shown, if the sum of the accumulator 160 exceeds the threshold Vth 164, the neuron output 70 is a spike and the accumulator is reset. The neuron input 68 is from all the time multiplexed inputs 68. A spike at the neuron output 70 may occur at any time, or there may be no spike at all. The rate of spikes at the output 70 may range from 0 Hz to less than 1 kHz; however, depending on the implementation of the neural circuit, the rate may be much higher. An embodiment of a neuron circuit 18 is further described in Reference [3] .
[ 0067 ] FIG. 9B shows a detail of the HP (Homeostatic
Plasticity) circuit 26, which is connected between the output 70 of the neuron circuit 18 and the multiplier 64, as shown in FIG. 2. The HP circuit 26 has a digital or an analog low pass
(LP) filter 170, a comparator 172, a summer 174, and a bias term 176, which may have a value 1 in one embodiment.
Parameters for the low pass filter may be stored in memory 3. The output 70 is filtered by the low pass filter 170. The result is input to comparator 172 and compared to the ftarget 173. The output of the comparator 172 is negative when the output of the LP filter 170 exceeds the ftarget 173. This allows the HP circuit 26 to regulate the spiking rate of the neuron circuit 18 to a rate defined by ftarget 173, which may be stored in memory 3. Some parts of this circuit are similar to a circuit described in Reference [4]; however, the circuit of FIG. 9B is intended to drive a digital multiplier instead than an amplifier as described in Reference [4] .
[ 0068 ] FIG. 9C shows an AD (Axonal Delay) circuit 28. The AD circuit 28 is connected to the output 70 of the neuron circuit 18, as shown in FIG. 2. The AD circuit 28 has a set of digital delay stages 180 and a multiplexer (MUX) 182. The AD circuit 28 of FIG. 9C produces delays that are aligned to a clock 184. An AD circuit is also described in Reference [4]; however, the AD circuit described in Reference [4] produces asynchronous delays. The AD circuit 28 delays the spikes produced by the neuron circuit 18, so that the neuron circuit 18 output is aligned to the cycle times indicated in FIG. 3, which makes the spike timing compatible with the time
multiplexed circuitry in the E-to-S 52, the STP 20, the STDP 22, the synapse 16 circuitry.
[ 0069 ] The circuit of the invention may be implemented with many integrated circuit technologies, including CMOS. A reduction to practice design has been made using 90nm CMOS. This design has over 1000 nodes, over 1000 neurons and over 100,000 synapses with STP, STDP, KD, HP and AD. The neural circuit 10 of this 90nm design has an area of less than 2 cm2 and a power of less than 1 Watt.
[ 0070 ] FIG. 10A shows an end-to-end simulation a network consisting of a 16 neurons receiving 16 external inputs. These 16 neurons are connected via 16 synapses with STDP and STP to an output neuron. The neural circuit can determine which inputs are correlated to each other. The top graph of FIG. 10A shows the 16 inputs to the chip, which are labeled as Pre 1 to Pre 16, and the spiking output post of the output neuron. The bottom graph of FIG. 10A shows the weights wi*xi of the 16 synapses of the output neuron. All the weights of synapses receiving correlated inputs evolve to a high state, as
desired. All the weights of synapses receiving uncorrelated inputs evolve to a low state, as desired. This simulation shows proper operation of the core neuron and synapse circuits as well as other circuitry.
[0071] FIG. 10B shows a simulation of the STP (Short Term Plasticity) circuit 20, which is used to calculate a signal x that is used to further adjust the gain of a synapse according to the recent spikes received at the input of the synapse. This STP circuit 20 is used in addition to the STDP (Spike Timing Dependent Plasticity) circuit 22. The top graph of FIG. 10B shows the spike signal at the input of synapse. The bottom graph of FIG. 10B shows the output signal produced by the STP circuit 22. This output signal consists of a step happening at the time of the input spike, followed by an exponential. The transistor-level simulation matched the expected behaviors .
[0072] FIG. IOC shows a simulation of a HP (Homoestatic Plasticity) circuit 26. The HP circuit 26 is used to adjust the spiking rate of a neuron circuit 18 to a given target rate. In the example the target rate is 50 Hz. The top graph of FIG. IOC shows the spiking output of the neuron circuit. The bottom graph of FIG. IOC shows actual measured spiking rate, and is close to the target rate as desired.
[0073] Having now described the invention in accordance with the requirements of the patent statutes, those skilled in this art will understand how to make changes and modifications to the present invention to meet their specific requirements or conditions. Such changes and modifications may be made without departing from the scope and spirit of the invention as disclosed herein.
[ 0074 ] The foregoing Detailed Description of exemplary and preferred embodiments is presented for purposes of
illustration and disclosure in accordance with the
requirements of the law. It is not intended to be exhaustive nor to limit the invention to the precise form(s) described, but only to enable others skilled in the art to understand how the invention may be suited for a particular use or
implementation. The possibility of modifications and
variations will be apparent to practitioners skilled in the art. No limitation is intended by the description of
exemplary embodiments which may have included tolerances, feature dimensions, specific operating conditions, engineering specifications, or the like, and which may vary between implementations or with changes to the state of the art, and no limitation should be implied therefrom. Applicant has made this disclosure with respect to the current state of the art, but also contemplates advancements and that adaptations in the future may take into consideration of those advancements, namely in accordance with the then current state of the art. It is intended that the scope of the invention be defined by the Claims as written and equivalents as applicable.
Reference to a claim element in the singular is not intended to mean "one and only one" unless explicitly so stated.
Moreover, no element, component, nor method or process step in this disclosure is intended to be dedicated to the public regardless of whether the element, component, or step is explicitly recited in the Claims. No claim element herein is to be construed under the provisions of 35 U.S.C. Sec. 112, sixth paragraph, unless the element is expressly recited using the phrase "means for. . ." and no method or process step herein is to be construed under those provisions unless the step, or steps, are expressly recited using the phrase
"comprising the step(s) of. . .."
[0075] All elements, parts and steps described herein are preferably included. It is to be understood that any of these elements, parts and steps may be replaced by other elements, parts and steps or deleted altogether as will be obvious to those skilled in the art.
Abstract
[0076] Broadly, this writing discloses at least the following :
A circuit for emulating the behavior of biological neural circuits, the circuit including a plurality of nodes wherein each node comprises a neuron circuit, a time multiplexed synapse circuit coupled to an input of the neuron circuit, a time multiplexed short term plasticity (STP) circuit coupled to an input of the node and to the synapse circuit, a time multiplexed Spike Timing Dependent Plasticity (STDP) circuit coupled to the input of the node and to the synapse circuit, an output of the node coupled to the neuron circuit; and an interconnect fabric coupled between the plurality of nodes for providing coupling from the output of any node of the
plurality of nodes to any input of any other node of the plurality of nodes. Concepts
This writing also presents at least the following concepts.
Concept 1. A circuit for emulating the behavior of
biological neural circuits, the circuit comprising:
a plurality of nodes wherein each node comprises:
a neuron circuit;
a time multiplexed synapse circuit coupled to an input of the neuron circuit;
a time multiplexed short term plasticity (STP) circuit coupled to an input of the node and to the synapse circuit ;
a time multiplexed Spike Timing Dependent Plasticity (STDP) circuit coupled to the input of the node and to the synapse circuit; and
an output of the node coupled to the neuron circuit; and
an interconnect fabric coupled between the plurality of nodes for providing coupling from the output of any node of the plurality of nodes to any input of any other node of the plurality of nodes.
Concept 2. The circuit of concept 1:
wherein the input of a respective node is time
multiplexed between the outputs of N nodes of the plurality of nodes, wherein an output of node i of the N nodes is coupled to the input of the respective node for a time slot i, wherein N time slots is a cycle, and wherein an output of each of the N nodes is coupled to the input of the respective node for a same time slot in each cycle;
wherein the time multiplexed STP circuit calculates a weight xi for the synapse circuit on each time slot i; and wherein the time multiplexed STDP circuit calculates ght wi for the synapse circuit on each time slot i.
Concept 3. The circuit of concept 1 further comprising: an exponential to spike (E-to-S) conversion circuit coupled between the input of the node and the synapse circuit, the STP circuit, and the STDP circuit.
Concept 4. The circuit of concept 2 wherein the STP circuit comprises:
an STP computation circuit; and
an STP memory;
wherein the STP memory has N addresses; and
wherein the STP computation circuit comprises:
an input coupled to the (E-to-S) conversion circuit; a first adder coupled to the input and to the STP memory;
a stored parameter b coupled to the adder; wherein the first adder adds the input, subtracts the parameter b, and adds a value xi read from address i of the STP memory for time slot i;
a multiplier for multiplying the output of the first adder by a stored parameter k; and
a second adder for adding a product of the multiplier and the parameter b to produce a result x± for time slot i;
wherein the result x± for time slot i is written into address i of the STP memory.
Concept 5. The circuit of concept 2 wherein the STDP circuit comprises:
an STDP computation circuit; and
an STDP memory;
wherein the STDP memory has N addresses; and wherein the STDP computation circuit comprises: a first input coupled to the input of the node;
a second input coupled to an output of the neuron circuit ;
a first spike to exponential (S-to-E) conversion circuit coupled to the output of the neuron circuit;
a first comparator having a first input coupled to the input of the node, a second input coupled to a stored bias parameter, and an enable input coupled to the output of the neuron circuit, wherein the enable input enables a first comparator output;
a second comparator having a first input coupled to an output of the first S-to-E conversion circuit, a second input coupled to the stored bias parameter, and an enable input coupled to an output of the E-to-S conversion circuit, wherein the enable input enables a second comparator output;
a first adder for adding the outputs of the first and second comparator; and
a second adder for adding an output of the first adder and a value w± read from address i of the STDP memory for time slot i to produce a result wi for time slot i;
wherein the result Wi for time slot i is written into address i of the STDP memory.
Concept 6. The circuit of concept 5 wherein:
the first input of the first comparator is coupled to a positive input of the first comparator;
the second input of the first comparator is coupled to a negative input of the first comparator;
the first input of the second comparator is coupled to a positive input of the second comparator; and
the second input of the second comparator is coupled to a negative input of the second comparator. Concept 7. The circuit of concept 5 wherein:
the first input of the first comparator is coupled to a negative input of the first comparator;
the second input of the first comparator is coupled to a positive input of the first comparator;
the first input of the second comparator is coupled to a positive input of the second comparator; and
the second input of the second comparator is coupled to a negative input of the second comparator.
Concept 8. The circuit of concept 5 wherein:
the first input of the first comparator is coupled to a negative input of the first comparator;
the second input of the first comparator is coupled to a positive input of the first comparator;
the first input of the second comparator is coupled to a negative input of the second comparator; and
the second input of the second comparator is coupled to a positive input of the second comparator.
Concept 9. The circuit of concept 2 wherein the synapse circuit comprises:
a multiplier for multiplying Xi from the STP circuit times the Wi from the STDP circuit to calculate a product x±Wi; and a variable gain amplifier that has a gain that depends on the product XiWi .
Concept 10. The circuit of concept 1 wherein the neuron circuit comprises:
an accumulator for summing an input to the neuron
circuit; and
a comparator for comparing an output of the accumulator to a stored threshold parameter; wherein if the output of the accumulator is greater than the threshold parameter, the output of the neuron circuit is spike and the accumulator is reset to zero.
Concept 11. The circuit of concept 3 wherein the E-to-S conversion circuit comprises:
a memory;
an adder coupled to the input of the node and to the memory for adding the input on time slot i to a value read from the memory; and
a multiplier for multiplying an output of the adder and stored parameter k for calculating a product;
wherein at each time slot I the product is written into the memory.
Concept 12. The circuit of concept 1 further comprising: a kinetic dynamics (KD) circuit coupled to the output of the synapse circuit;
a homeostatic plasticity (HP) circuit coupled to the output of the neuron circuit; and
a multiplier coupled to an output of the KD circuit an output of the HP circuit for multiplying the output of the KD circuit times the output of the HP circuit;
wherein an output of the multiplier is an input to the neuron circuit.
Concept 13. The circuit of concept 12 wherein:
the homeostatic plasticity (HP) circuit comprises:
a low pass filter coupled to the output of the neuron circuit;
a comparator coupled to an output of the low pass filter and to a stored parameter f target and an adder coupled the an output of the comparator for adding the output of the comparator and a second stored parameter;
wherein an output of the adder is coupled to the multiplier .
Concept 14. The circuit of concept 12 wherein:
the KD circuit comprises:
a 1 to J multiplexer coupled to the output of the synapse circuit;
J spike to exponential (S-to-E) conversion circuits coupled to an output of 1 to J multiplexer, wherein each of the J spike to exponential (S-to-E) conversion circuits may have a different time decay constant; and
an adder coupled to outputs of the J S-to-E conversion circuits;
wherein an an output of the adder is coupled to the multiplier .
Concept 15. The circuit of concept 1 further comprising: an axonal delay (AD) circuit coupled to the output of the neuron circuit; and
a second spike to exponential (S-to-E) conversion circuit coupled to the output of the AD circuit;
wherein the output of the node is the output of the second S-to-E conversion circuit.
Concept 16. The circuit of concept 15 wherein the axonal delay (AD) circuit comprises:
an input coupled to the output of the neuron circuit; a plurality of delay elements connected in series, wherein each delay element has an output and wherein each delay element is coupled to a clock that is aligned to the time slots; and a multiplexer coupled to the output of each delay
element ;
wherein an output of the multiplexer to the second spike to exponential (S-to-E) conversion circuit.
Concept 17. A method for emulating the behavior of
biological neural circuits, the method comprising:
providing a plurality of nodes;
providing a neuron circuit in each node;
providing a synapse circuit in each node coupled to an input of the neuron circuit;
providing a short term plasticity (STP) circuit in each node coupled to an input of the node and to the synapse circuit ;
providing a Spike Timing Dependent Plasticity (STDP) circuit in each node coupled to the input of the node and to the synapse circuit;
providing an output of the node coupled to the neuron circuit ;
time multiplexing the synapse, the STP, and the STDP circuits; and
providing an interconnect fabric coupled between the plurality of nodes for providing coupling from the output of any node of the plurality of nodes to any input of any other node of the plurality of nodes.
Concept 18. The method of concept 17:
wherein the input of a respective node is time
multiplexed between the outputs of N nodes of the plurality of nodes, wherein an output of node i of the N nodes is coupled to the input of the respective node for a time slot i, wherein N time slots is a cycle, and wherein an output of each of the N nodes is coupled to the input of the respective node for a same time slot in each cycle; wherein the time multiplexed STP circuit calculates a weight x± for the synapse circuit on each time slot i; and
wherein the time multiplexed STDP circuit calculates a weight wi for the synapse circuit on each time slot i.
Concept 19. The method of concept 17 further comprising: converting the input of the node from an exponential to a spike using an exponential to spike (E-to-S) conversion circuit coupled between the input of the node and the synapse circuit, the STP circuit, and the STDP circuit.
Concept 20. The method of concept 18 wherein providing the STP circuit comprises:
providing an STP computation circuit; and
providing an STP memory;
wherein the STP memory has N addresses; and
wherein the STP computation circuit:
adds an input coupled to the (E-to-S) conversion circuit to the negative of stored parameter b and adds a value Xi read from address i of the STP memory for time slot i;
multiplies a sum of the adding by a stored parameter k; and
adds a product of the multiplier and the parameter b to produce a result x± for time slot i;
wherein the result x± for time slot i is written into address i of the STP memory.
Concept 21. The method of concept 18 wherein providing the STDP circuit comprises:
providing an STDP computation circuit; and
providing an STDP memory;
wherein the STDP memory has N addresses; and
wherein the STDP computation circuit: compares a first input coupled to the input of the node to a second input coupled to a stored bias parameter, wherein a first enable input coupled to the output of the neuron circuit enables a first comparison output;
compares a third input coupled to an output of a S-to-E conversion circuit coupled to the output of the neuron circuit to a fourth input coupled to the stored bias
parameter, wherein a second enable input coupled to an output of the E-to-S conversion circuit enables a second comparator output ;
adding the outputs of the first and second comparator to produce a sum; and
adding the sum and a value w± read from address i of the STDP memory for time slot i to produce a result w± for time slot i;
wherein the result w± for time slot i is written into address i of the STDP memory.
Concept 22. The method of concept 18 wherein providing the synapse circuit comprises:
multiplying Xi from the STP circuit times the Wi from the STDP circuit to calculate a product x±Wi; and
amplifying the input to the synapse by an amount that depends on the product x±Wi .
Concept 23. The method of concept 17 wherein providing the neuron circuit comprises:
accumulating inputs to the neuron circuit; and
comparing a value of the accumulating to a stored
threshold parameter;
wherein if the value is greater than the threshold parameter, the output of the neuron circuit is a spike and the accumulator is reset to zero. Concept 24. The method of concept 17 further comprising: providing a kinetic dynamics (KD) circuit coupled to the output of the synapse circuit;
providing a homeostatic plasticity (HP) circuit coupled to the output of the neuron circuit; and
multiplying an output of the KD circuit times an output of the HP circuit; and
providing the product of the multiplying to the neuron circuit .
Concept 25. The method of concept 24 wherein:
Providing the homeostatic plasticity (HP) circuit comprises :
low pass filtering outputs of the neuron circuit; comparing an output of the low pass filter to a stored parameter f target / and
adding an output of the comparator and a second stored parameter to form a sum that is provided for the step of multiplying.
Concept 26. The method of concept 24 wherein providing the KD circuit comprises:
providing J spike to exponential (S-to-E) conversion circuits coupled to an output of a 1 to J multiplexer, wherein each of the J spike to exponential (S-to-E) conversion
circuits may have a different time decay constant; and
adding outputs of the J S-to-E conversion circuits to form a sum that is provided for the step of multiplying.
Concept 27. The method of concept 17 further comprising: providing an axonal delay (AD) coupled to the output of the neuron circuit; and
converting a spike output of the AD circuit to an
exponential .

Claims

WHAT IS CLAIMED IS:
1. A circuit for emulating the behavior of biological neural circuits, the circuit comprising:
a plurality of nodes wherein each node comprises:
a neuron circuit;
a time multiplexed synapse circuit coupled to an input of the neuron circuit;
a time multiplexed short term plasticity (STP) circuit coupled to an input of the node and to the synapse circuit ;
a time multiplexed Spike Timing Dependent Plasticity (STDP) circuit coupled to the input of the node and to the synapse circuit; and
an output of the node coupled to the neuron circuit; and
an interconnect fabric coupled between the plurality of nodes for providing coupling from the output of any node of the plurality of nodes to any input of any other node of the plurality of nodes.
2. The circuit of claim 1:
wherein the input of a respective node is time
multiplexed between the outputs of N nodes of the plurality of nodes, wherein an output of node i of the N nodes is coupled to the input of the respective node for a time slot i, wherein N time slots is a cycle, and wherein an output of each of the N nodes is coupled to the input of the respective node for a same time slot in each cycle;
wherein the time multiplexed STP circuit calculates a weight x± for the synapse circuit on each time slot i; and
wherein the time multiplexed STDP circuit calculates a weight w± for the synapse circuit on each time slot i.
3. The circuit of claim 1 further comprising:
an exponential to spike (E-to-S) conversion circuit coupled between the input of the node and the synapse circuit, the STP circuit, and the STDP circuit.
4. The circuit of claim 2 wherein the STP circuit comprises: an STP computation circuit; and
an STP memory;
wherein the STP memory has N addresses; and
wherein the STP computation circuit comprises:
an input coupled to the (E-to-S) conversion circuit; a first adder coupled to the input and to the STP memory;
a stored parameter b coupled to the adder; wherein the first adder adds the input, subtracts the parameter b, and adds a value xi read from address i of the STP memory for time slot i;
a multiplier for multiplying the output of the first adder by a stored parameter k; and
a second adder for adding a product of the multiplier and the parameter b to produce a result x± for time slot i;
wherein the result x± for time slot i is written into address i of the STP memory.
5. The circuit of claim 2 wherein the STDP circuit
comprises :
an STDP computation circuit; and
an STDP memory;
wherein the STDP memory has N addresses; and
wherein the STDP computation circuit comprises:
a first input coupled to the input of the node;
a second input coupled to an output of the neuron circuit ; a first spike to exponential (S-to-E) conversion circuit coupled to the output of the neuron circuit;
a first comparator having a first input coupled to the input of the node, a second input coupled to a stored bias parameter, and an enable input coupled to the output of the neuron circuit, wherein the enable input enables a first comparator output;
a second comparator having a first input coupled to an output of the first S-to-E conversion circuit, a second input coupled to the stored bias parameter, and an enable input coupled to an output of the E-to-S conversion circuit, wherein the enable input enables a second comparator output;
a first adder for adding the outputs of the first and second comparator; and
a second adder for adding an output of the first adder and a value w± read from address i of the STDP memory for time slot i to produce a result w± for time slot i;
wherein the result w± for time slot i is written into address i of the STDP memory.
6. The circuit of claim 5 wherein:
the first input of the first comparator is coupled to a positive input of the first comparator;
the second input of the first comparator is coupled to negative input of the first comparator;
the first input of the second comparator is coupled to positive input of the second comparator; and
the second input of the second comparator is coupled to negative input of the second comparator.
7. The circuit of claim 5 wherein :
the first input of the first comparator is coupled to a negative input of the first comparator; the second input of the first comparator is coupled to positive input of the first comparator;
the first input of the second comparator is coupled to positive input of the second comparator; and
the second input of the second comparator is coupled to negative input of the second comparator.
8. The circuit of claim 5 wherein:
the first input of the first comparator is coupled to a negative input of the first comparator;
the second input of the first comparator is coupled to positive input of the first comparator;
the first input of the second comparator is coupled to negative input of the second comparator; and
the second input of the second comparator is coupled to positive input of the second comparator.
9. The circuit of claim 2 wherein the synapse circuit comprises :
a multiplier for multiplying x± from the STP circuit times the Wi from the STDP circuit to calculate a product x±Wi; and a variable gain amplifier that has a gain that depends on the product XiWi .
10. The circuit of claim 1 wherein the neuron circuit
comprises :
an accumulator for summing an input to the neuron
circuit; and
a comparator for comparing an output of the accumulator to a stored threshold parameter;
wherein if the output of the accumulator is greater than the threshold parameter, the output of the neuron circuit is a spike and the accumulator is reset to zero.
11. The circuit of claim 3 wherein the E-to-S conversion circuit comprises:
a memory;
an adder coupled to the input of the node and to the memory for adding the input on time slot i to a value read from the memory; and
a multiplier for multiplying an output of the adder and a stored parameter k for calculating a product;
wherein at each time slot I the product is written into the memory.
12. The circuit of claim 1 further comprising:
a kinetic dynamics (KD) circuit coupled to the output of the synapse circuit;
a homeostatic plasticity (HP) circuit coupled to the output of the neuron circuit; and
a multiplier coupled to an output of the KD circuit an output of the HP circuit for multiplying the output of the KD circuit times the output of the HP circuit;
wherein an output of the multiplier is an input to the neuron circuit.
13. The circuit of claim 12 wherein:
the homeostatic plasticity (HP) circuit comprises:
a low pass filter coupled to the output of the neuron circuit;
a comparator coupled to an output of the low pass filter and to a stored parameter f target and
an adder coupled the an output of the comparator for adding the output of the comparator and a second stored parameter;
wherein an output of the adder is coupled to the multiplier .
14. The circuit of claim 12 wherein:
the KD circuit comprises:
a 1 to J multiplexer coupled to the output of the synapse circuit;
J spike to exponential (S-to-E) conversion circuits coupled to an output of 1 to J multiplexer, wherein each of the J spike to exponential (S-to-E) conversion circuits may have a different time decay constant; and
an adder coupled to outputs of the J S-to-E conversion circuits;
wherein an an output of the adder is coupled to the multiplier .
15. The circuit of claim 1 further comprising:
an axonal delay (AD) circuit coupled to the output of the neuron circuit; and
a second spike to exponential (S-to-E) conversion circuit coupled to the output of the AD circuit;
wherein the output of the node is the output of the second S-to-E conversion circuit.
16. The circuit of claim 15 wherein the axonal delay (AD) circuit comprises:
an input coupled to the output of the neuron circuit; a plurality of delay elements connected in series, wherein each delay element has an output and wherein each delay element is coupled to a clock that is aligned to the time slots; and
a multiplexer coupled to the output of each delay
element ;
wherein an output of the multiplexer to the second spike to exponential (S-to-E) conversion circuit.
17. A method for emulating the behavior of biological neural circuits, the method comprising:
providing a plurality of nodes;
providing a neuron circuit in each node;
providing a synapse circuit in each node coupled to an input of the neuron circuit;
providing a short term plasticity (STP) circuit in each node coupled to an input of the node and to the synapse circuit ;
providing a Spike Timing Dependent Plasticity (STDP) circuit in each node coupled to the input of the node and to the synapse circuit;
providing an output of the node coupled to the neuron circuit ;
time multiplexing the synapse, the STP, and the STDP circuits; and
providing an interconnect fabric coupled between the plurality of nodes for providing coupling from the output of any node of the plurality of nodes to any input of any other node of the plurality of nodes.
18. The method of claim 17:
wherein the input of a respective node is time
multiplexed between the outputs of N nodes of the plurality of nodes, wherein an output of node i of the N nodes is coupled to the input of the respective node for a time slot i, wherein N time slots is a cycle, and wherein an output of each of the N nodes is coupled to the input of the respective node for a same time slot in each cycle;
wherein the time multiplexed STP circuit calculates a weight x± for the synapse circuit on each time slot i; and
wherein the time multiplexed STDP circuit calculates a weight w± for the synapse circuit on each time slot i.
19. The method of claim 17 further comprising:
converting the input of the node from an exponential to a spike using an exponential to spike (E-to-S) conversion circuit coupled between the input of the node and the synapse circuit, the STP circuit, and the STDP circuit.
20. The method of claim 18 wherein providing the STP circuit comprises :
providing an STP computation circuit; and
providing an STP memory;
wherein the STP memory has N addresses; and
wherein the STP computation circuit:
adds an input coupled to the (E-to-S) conversion circuit to the negative of stored parameter b and adds a value Xi read from address i of the STP memory for time slot i;
multiplies a sum of the adding by a stored parameter k; and
adds a product of the multiplier and the parameter b to produce a result Xi for time slot i;
wherein the result Xi for time slot i is written into address i of the STP memory.
21. The method of claim 18 wherein providing the STDP circuit comprises :
providing an STDP computation circuit; and
providing an STDP memory;
wherein the STDP memory has N addresses; and
wherein the STDP computation circuit:
compares a first input coupled to the input of the node to a second input coupled to a stored bias parameter, wherein a first enable input coupled to the output of the neuron circuit enables a first comparison output;
compares a third input coupled to an output of a S-to-E conversion circuit coupled to the output of the neuron circuit to a fourth input coupled to the stored bias
parameter, wherein a second enable input coupled to an output of the E-to-S conversion circuit enables a second comparator output ;
adding the outputs of the first and second comparator to produce a sum; and
adding the sum and a value w± read from address i of the STDP memory for time slot i to produce a result w± for time slot i;
wherein the result w± for time slot i is written into address i of the STDP memory.
22. The method of claim 18 wherein providing the synapse circuit comprises:
multiplying x± from the STP circuit times the w± from the STDP circuit to calculate a product XiWi; and
amplifying the input to the synapse by an amount that depends on the product XiWi .
23. The method of claim 17 wherein providing the neuron circuit comprises:
accumulating inputs to the neuron circuit; and
comparing a value of the accumulating to a stored
threshold parameter;
wherein if the value is greater than the threshold parameter, the output of the neuron circuit is a spike and the accumulator is reset to zero.
24. The method of claim 17 further comprising:
providing a kinetic dynamics (KD) circuit coupled to the output of the synapse circuit;
providing a homeostatic plasticity (HP) circuit coupled to the output of the neuron circuit; and multiplying an output of the KD circuit times an output of the HP circuit; and
providing the product of the multiplying to the neuron circuit .
25. The method of claim 24 wherein:
Providing the homeostatic plasticity (HP) circuit comprises :
low pass filtering outputs of the neuron circuit; comparing an output of the low pass filter to a stored parameter f target / and
adding an output of the comparator and a second stored parameter to form a sum that is provided for the step of multiplying.
26. The method of claim 24 wherein providing the KD circuit comprises :
providing J spike to exponential (S-to-E) conversion circuits coupled to an output of a 1 to J multiplexer, wherein each of the J spike to exponential (S-to-E) conversion
circuits may have a different time decay constant; and
adding outputs of the J S-to-E conversion circuits to form a sum that is provided for the step of multiplying.
27. The method of claim 17 further comprising:
providing an axonal delay (AD) coupled to the output of the neuron circuit; and
converting a spike output of the AD circuit to an
exponential .
PCT/US2017/035323 2016-06-30 2017-05-31 Neural integrated circuit with biological behaviors WO2018004966A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP17820834.4A EP3479303A4 (en) 2016-06-30 2017-05-31 Neural integrated circuit with biological behaviors
CN201780040366.3A CN109416759B (en) 2016-06-30 2017-05-31 Neural integrated circuit with biological behavior

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US15/199,800 US10147035B2 (en) 2016-06-30 2016-06-30 Neural integrated circuit with biological behaviors
US15/199,800 2016-06-30

Publications (1)

Publication Number Publication Date
WO2018004966A1 true WO2018004966A1 (en) 2018-01-04

Family

ID=60785269

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2017/035323 WO2018004966A1 (en) 2016-06-30 2017-05-31 Neural integrated circuit with biological behaviors

Country Status (4)

Country Link
US (1) US10147035B2 (en)
EP (1) EP3479303A4 (en)
CN (1) CN109416759B (en)
WO (1) WO2018004966A1 (en)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11501143B2 (en) 2013-10-11 2022-11-15 Hrl Laboratories, Llc Scalable integrated circuit with synaptic electronics and CMOS integrated memristors
JP7070239B2 (en) * 2018-08-23 2022-05-18 株式会社明電舎 Neural network input / output control method
CN109376851A (en) * 2018-08-29 2019-02-22 博瓦（武汉）科技有限公司 The spiking neuron signal generating circuit of bionic system is based on the implementation method of memristor
CN109934338B (en) * 2019-03-03 2024-03-19 广西师范大学 Hardware circuit for realizing neuron model
KR20200111584A (en) * 2019-03-19 2020-09-29 한국전자통신연구원 Electronic device for encoding event indicating spatial-temporal input signals and operating method thereof
US20210103801A1 (en) * 2019-10-03 2021-04-08 International Business Machines Corporation Artificial intelligence hardware with synaptic reuse

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140032465A1 (en) * 2012-03-29 2014-01-30 International Business Machines Corporation Synaptic, dendritic, somatic, and axonal plasticity in a network of neural cores using a plastic multi-stage crossbar switching
US20140032460A1 (en) * 2012-07-25 2014-01-30 Hrl Labroratories, Llc Spike domain neuron circuit with programmable kinetic dynamic, homeostatic plasticity and axonal delays
US20140250037A1 (en) * 2011-09-21 2014-09-04 Brain Corporation Methods for memory management in parallel networks
US20150220831A1 (en) * 2014-02-06 2015-08-06 Qualcomm Incorporated Short-term synaptic memory based on a presynaptic spike

Family Cites Families (36)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3569842A (en) 1968-07-29 1971-03-09 Bendix Corp Pulse delay circuit
US3824411A (en) 1972-06-19 1974-07-16 Rca Corp Pulse delay circuit
NL9201053A (en) 1992-06-15 1994-01-03 Koninkl Philips Electronics Nv SWITCHED CAPACITOR LOADING PUMP AND SAW Tooth Oscillator equipped with such a SWITCHED CAPACITOR LOADING PUMP.
US5650739A (en) 1992-12-07 1997-07-22 Dallas Semiconductor Corporation Programmable delay lines
US5717947A (en) 1993-03-31 1998-02-10 Motorola, Inc. Data processing system and method thereof
JP3460913B2 (en) 1995-09-29 2003-10-27 旭化成マイクロシステム株式会社 Variable delay time generating circuit and method
US5907250A (en) 1997-05-16 1999-05-25 International Business Machines Corporation Transition detector with timer
FR2766993B1 (en) 1997-07-31 1999-10-15 Sgs Thomson Microelectronics ADJUSTABLE DELAY CIRCUIT
US6097212A (en) 1997-10-09 2000-08-01 Lattice Semiconductor Corporation Variable grain architecture for FPGA integrated circuits
US6124730A (en) 1998-12-15 2000-09-26 Vantis Corporation Methods for configuring FPGA's having variable grain components for providing time-shared access to interconnect resources
US6581046B1 (en) 1997-10-10 2003-06-17 Yeda Research And Development Co. Ltd. Neuronal phase-locked loops
US6052011A (en) 1997-11-10 2000-04-18 Tritech Microelectronics, Ltd. Fractional period delay circuit
JP3144398B2 (en) 1998-10-27 2001-03-12 日本電気株式会社 Variable delay circuit
JP2001326562A (en) 2000-05-15 2001-11-22 Nec Corp Variable delay circuit
US20040217794A1 (en) 2003-04-30 2004-11-04 Mark Strysko Propagation delay adjustment circuit
US7515084B1 (en) 2007-03-22 2009-04-07 Hrl Laboratories, Llc Analog to digital converter using asynchronous pulse technology
US7822698B1 (en) 2007-03-23 2010-10-26 Hrl Laboratories, Llc Spike domain and pulse domain non-linear processors
US7724168B1 (en) 2007-10-31 2010-05-25 Hrl Laboratories, Llc Pulse domain linear programming circuit
CN101453313B (en) 2007-12-06 2013-03-20 鸿富锦精密工业（深圳）有限公司 Communication circuit for master-slave equipment
US20110004579A1 (en) 2008-03-14 2011-01-06 Greg Snider Neuromorphic Circuit
US8600919B2 (en) 2008-08-28 2013-12-03 Massachusetts Institute Of Technology Circuits and methods representative of spike timing dependent plasticity of neurons
JP5326804B2 (en) 2008-09-29 2013-10-30 富士電機株式会社 Power factor improving power supply device, control circuit used for power supply device, and control method
CN101771413B (en) 2008-12-31 2013-04-24 南方医科大学 Analog-digital decoding method and device based on neuron working principle
US20120011092A1 (en) 2010-07-07 2012-01-12 Qualcomm Incorporated Methods and systems for memristor-based neuron circuits
US9129220B2 (en) * 2010-07-07 2015-09-08 Qualcomm Incorporated Methods and systems for digital neural processing with discrete-level synapes and probabilistic STDP
US8892485B2 (en) 2010-07-08 2014-11-18 Qualcomm Incorporated Methods and systems for neural processor training by encouragement of correct output
US8606732B2 (en) 2010-08-04 2013-12-10 Qualcomm Incorporated Methods and systems for reward-modulated spike-timing-dependent-plasticity
US20120110891A1 (en) 2010-11-05 2012-05-10 Minnameier Kip W Plant pest trap and methodology of use
US8473439B2 (en) 2010-12-08 2013-06-25 International Business Machines Corporation Integrate and fire electronic neurons
US8856055B2 (en) * 2011-04-08 2014-10-07 International Business Machines Corporation Reconfigurable and customizable general-purpose circuits for neural networks
US8812414B2 (en) 2011-05-31 2014-08-19 International Business Machines Corporation Low-power event-driven neural computing architecture in neural networks
US8843425B2 (en) 2011-07-29 2014-09-23 International Business Machines Corporation Hierarchical routing for two-way information flow and structural plasticity in neural networks
US9092735B2 (en) * 2011-09-21 2015-07-28 Qualcomm Incorporated Method and apparatus for structural delay plasticity in spiking neural networks
US8930291B1 (en) * 2012-05-03 2015-01-06 Hrl Laboratories, Llc Cortical neuromorphic network, system and method
US9924490B2 (en) 2013-10-09 2018-03-20 International Business Machines Corporation Scaling multi-core neurosynaptic networks across chip boundaries
CN103580668B (en) * 2013-10-28 2016-04-20 华中科技大学 A kind of associative memory circuit based on memristor

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140250037A1 (en) * 2011-09-21 2014-09-04 Brain Corporation Methods for memory management in parallel networks
US20140032465A1 (en) * 2012-03-29 2014-01-30 International Business Machines Corporation Synaptic, dendritic, somatic, and axonal plasticity in a network of neural cores using a plastic multi-stage crossbar switching
US20140032460A1 (en) * 2012-07-25 2014-01-30 Hrl Labroratories, Llc Spike domain neuron circuit with programmable kinetic dynamic, homeostatic plasticity and axonal delays
US20150220831A1 (en) * 2014-02-06 2015-08-06 Qualcomm Incorporated Short-term synaptic memory based on a presynaptic spike

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
JOSE M CRUZ-ALBRECHT ET AL.: "A scalable neural chip with synaptic electronics using CMOS integrated memristors", NANOTECHNOLOGY, September 2013 (2013-09-01), pages 1 - 11, XP020250887 *
See also references of EP3479303A4 *

Also Published As

Publication number Publication date
CN109416759B (en) 2020-04-28
US10147035B2 (en) 2018-12-04
EP3479303A1 (en) 2019-05-08
EP3479303A4 (en) 2020-02-26
US20180005108A1 (en) 2018-01-04
CN109416759A (en) 2019-03-01

Similar Documents

Publication Publication Date Title
US10147035B2 (en) Neural integrated circuit with biological behaviors
CN105229675B (en) The hardware-efficient of shunt peaking is realized
KR20170041724A (en) Artificial neurons and spiking neurons with asynchronous pulse modulation
WO2015112643A1 (en) Monitoring neural networks with shadow networks
EP2877958A1 (en) Neuron circuit and method
KR20160123309A (en) Event-based inference and learning for stochastic spiking bayesian networks
US10552734B2 (en) Dynamic spatial target selection
KR101825933B1 (en) Phase-coding for coordinate transformation
KR20160125967A (en) Method and apparatus for efficient implementation of common neuron models
KR102574887B1 (en) Electronic circuit for implementing generative adversarial network using spike neural network
WO1993018474A1 (en) Devices for use in neural processing
WO2015065850A2 (en) Doppler effect processing in a neural network model
Kaynak Sliding mode algorithm for online learning in analog multilayer feedforward neural networks
Takahashi Remarks on feedforward-feedback controller using simple recurrent quaternion neural network
Oliveri et al. An application of spike-timing-dependent plasticity to readout circuit for liquid state machine
CN113011572B (en) Axon change amount determining method and device and weight processing method and device
US5553197A (en) Devices for use in neural processing
EP3108412A2 (en) Imbalanced cross-inhibitory mechanism for spatial target selection
Belhadj et al. Digital mapping of a realistic spike timing plasticity model for real-time neural simulations
Petlenkov Model reference control of nonlinear systems by dynamic output feedback linearization of neural network based ANARX models
Chen et al. Optimizing Supervised Learning of Deep Spiking Neural Network towards Memristor Crossbar Implementation
Alhassan et al. Forecasting Nigeria Foreign Exchange Rates Using Artificial Neural Network
Mertzios et al. Predicting a Chaotic Time Series Using a Dynamical Recurrent Neural Network
Moreno et al. Physical mapping of spiking neural networks models on a bio-inspired scalable architecture
Ng Stand-alone hardware-based learning system

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17820834

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2017820834

Country of ref document: EP

Effective date: 20190130