US20190304538A1 - In-cell differential read-out circuitry for reading signed weight values in resistive processing unit architecture - Google Patents
In-cell differential read-out circuitry for reading signed weight values in resistive processing unit architecture Download PDFInfo
- Publication number
- US20190304538A1 US20190304538A1 US16/352,230 US201916352230A US2019304538A1 US 20190304538 A1 US20190304538 A1 US 20190304538A1 US 201916352230 A US201916352230 A US 201916352230A US 2019304538 A1 US2019304538 A1 US 2019304538A1
- Authority
- US
- United States
- Prior art keywords
- current
- read
- rpu
- weight
- cell
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11C—STATIC STORES
- G11C13/00—Digital stores characterised by the use of storage elements not covered by groups G11C11/00, G11C23/00, or G11C25/00
- G11C13/0002—Digital stores characterised by the use of storage elements not covered by groups G11C11/00, G11C23/00, or G11C25/00 using resistive RAM [RRAM] elements
- G11C13/0021—Auxiliary circuits
- G11C13/004—Reading or sensing circuits or methods
-
- G06N3/0635—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
- G06N3/065—Analogue means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11C—STATIC STORES
- G11C13/00—Digital stores characterised by the use of storage elements not covered by groups G11C11/00, G11C23/00, or G11C25/00
- G11C13/0002—Digital stores characterised by the use of storage elements not covered by groups G11C11/00, G11C23/00, or G11C25/00 using resistive RAM [RRAM] elements
- G11C13/0021—Auxiliary circuits
- G11C13/0038—Power supply circuits
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11C—STATIC STORES
- G11C13/00—Digital stores characterised by the use of storage elements not covered by groups G11C11/00, G11C23/00, or G11C25/00
- G11C13/0002—Digital stores characterised by the use of storage elements not covered by groups G11C11/00, G11C23/00, or G11C25/00 using resistive RAM [RRAM] elements
- G11C13/0021—Auxiliary circuits
- G11C13/004—Reading or sensing circuits or methods
- G11C2013/0042—Read using differential sensing, e.g. bit line [BL] and bit line bar [BLB]
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11C—STATIC STORES
- G11C13/00—Digital stores characterised by the use of storage elements not covered by groups G11C11/00, G11C23/00, or G11C25/00
- G11C13/0002—Digital stores characterised by the use of storage elements not covered by groups G11C11/00, G11C23/00, or G11C25/00 using resistive RAM [RRAM] elements
- G11C13/0021—Auxiliary circuits
- G11C13/004—Reading or sensing circuits or methods
- G11C2013/0045—Read using current through the cell
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11C—STATIC STORES
- G11C13/00—Digital stores characterised by the use of storage elements not covered by groups G11C11/00, G11C23/00, or G11C25/00
- G11C13/0002—Digital stores characterised by the use of storage elements not covered by groups G11C11/00, G11C23/00, or G11C25/00 using resistive RAM [RRAM] elements
- G11C13/0021—Auxiliary circuits
- G11C13/004—Reading or sensing circuits or methods
- G11C2013/0054—Read is performed on a reference element, e.g. cell, and the reference sensed value is used to compare the sensed value of the selected cell
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11C—STATIC STORES
- G11C2213/00—Indexing scheme relating to G11C13/00 for features not covered by this group
- G11C2213/70—Resistive array aspects
- G11C2213/77—Array wherein the memory element being directly connected to the bit lines and word lines without any access device being used
Definitions
- This disclosure generally relates to resistive processing unit (RPU) frameworks and, in particular, techniques for reading weight values stored in RPU storage cells.
- RPU resistive processing unit
- DNN deep neural network
- DNNs are utilized in various applications including, for example, object/speech recognition, language translation, pattern extraction, and image processing.
- the quality of the DNN models depends on the processing of a large amount of training data and an increased complexity of the neural network.
- training a complex DNN model is a time consuming and computationally intensive task which can require many days or weeks to perform using parallel and distributed computing frameworks with many computing nodes (e.g., datacenter-scale computational resources) to complete the training of the DNN model.
- DNN training generally relies on a backpropagation algorithm which includes three repeating cycles: forward, backward and weight update.
- RPU accelerator devices which are based on a two-dimensional (2D) crossbar array of RPU storage cells, can be configured to perform all three cycles of the backpropagation algorithm in parallel, thus potentially providing significant acceleration in DNN training with lower power and reduced computation resources compared to state-of-the-art implementations using central processing units (CPUs) and graphics processing units (GPUs).
- An RPU accelerator can store and update weight values locally, thereby minimizing data movement during training and fully exploiting the locality and the parallelism of the DNN training process.
- Analog weight storage elements in RPU storage cells are capable of storing weight values without sign (e.g., positive, negative, zero).
- RPU operations for DNN training and other applications require processing of positive, zero and negative weight values.
- Embodiments of the invention include in-cell differential read-out circuits and methods to enable signed weight values to be directly read out from RPU cells of an RPU system.
- one embodiment includes an RPU device, which comprises: a weight storage device configured to store a weight voltage which corresponds to a weight value of the RPU device; a read transistor comprising a gate terminal, a first source/drain terminal, and a second source/drain terminal, wherein the gate terminal is connected to the weight storage device, wherein the first source/drain terminal is connected to a first control port of the RPU device, and wherein the second source/drain terminal is connected to a second control port of the RPU device; and a current source connected to the second source/drain terminal of the read transistor.
- the current source is configured to generate a fixed reference current.
- the read transistor is configured to generate a weight current in response to the weight voltage applied to the gate terminal of the read transistor.
- the RPU device is configured to output a read current from the second control port, wherein the read current comprises a magnitude and sign which represents a signed weight value of the RPU device.
- the magnitude of the read current is equal to a difference between the weight current generated by the read transistor and the fixed reference current of the current source.
- the sign of the read current is deemed positive when the weight current is greater than the fixed reference current, and the sign of the read current is deemed negative when the weight current is less than the fixed reference current.
- an RPU device which comprises: a weight storage device configured to store a weight voltage which corresponds to a weight value of the RPU device; a current integration circuit; a first read transistor comprising a gate terminal, a first source/drain terminal, and a second source/drain terminal, wherein the gate terminal is connected to the weight storage device, wherein the first source/drain terminal is connected to a first control port of the RPU device, and wherein the second source/drain terminal is connected to the current integration circuit; and a second read transistor comprising a gate terminal, a first source/drain terminal, and a second source/drain terminal, wherein the gate terminal is connected to the weight storage device, wherein the first source/drain terminal is connected to a second control port of the RPU device, and wherein the second source/drain terminal is connected to the current integration circuit.
- the first and second read transistors are configured to generate first and second weight currents, respectively, in response to the weight voltage applied to the gate terminals of the first and second read transistors, wherein the first and second weight currents are complementary currents.
- the current integration circuit is configured to generate a read current, which is output from a third port of the RPU device, wherein the read current comprises a magnitude and sign which represents a signed weight value of the RPU device.
- the magnitude of the read current is equal to a difference between the first weight current generated by the first read transistor and the second weight current generated by the second read transistor.
- the sign of the read current is deemed positive when the first weight current is greater than the second weight current, and the sign of the read current is deemed negative when the first weight current is less than the second weight current.
- an RPU device which comprises: a current mirror circuit comprising a reference input node and an output node; a variable resistor device having a resistance value which corresponds to a weight value of the RPU device, wherein the variable resistor device is connected between a first port and a second port of the RPU device, and connected to the output node of the current mirror circuit; and a fixed resistor device having a fixed resistance value, wherein the fixed resistor device is connected between the first port of the RPU device and the reference input node of the current mirror circuit.
- the variable resistor device is configured to generate a first current in response to a control voltage applied to the first port of the RPU device.
- the fixed resistor device is configured to generate a second current in response to the control voltage applied to the first port of the RPU device.
- the current mirror circuit is configured to mirror the second current, and sink the mirrored second current into the output node of the current mirror circuit and cause a read current to be generated and output from the second port of the RPU device.
- the read current comprises a magnitude and sign which represents a signed weight value of the RPU device.
- the magnitude of the read current is equal to a difference between the first current the second mirrored current.
- the sign of the read current is deemed positive when the first current is greater than the second mirrored current, and the sign of the read current is deemed negative when the first current is less than the second mirrored current.
- FIG. 1 schematically illustrates a resistive processing unit system which can be implemented using RPU cell architectures according to embodiments of the invention.
- FIG. 2A schematically illustrates a forward pass operation of a backpropagation process, which can be performed using the RPU system of FIG. 1 .
- FIG. 2B schematically illustrates a backward pass operation of a backpropagation process, which can be performed using the RPU system of FIG. 1 .
- FIG. 2C schematically illustrates a weight update operation of a backpropagation process, which can be performed using the RPU system of FIG. 1 .
- FIG. 3 conceptually illustrates a process for utilizing a reference current source within an RPU cell to implement a differential read scheme which enables signed weight values to be directly read out from the RPU cell, according to an embodiment of the invention.
- FIG. 4 schematically illustrates an RPU cell framework which implements a differential read scheme using a reference current source to enable signed weight values to be directly read out from the RPU cell, according to an embodiment of the invention.
- FIG. 5 conceptually illustrates an in-cell complementary differential read scheme to enable signed weight values to be directly read out from an RPU cell, according to an embodiment of the invention.
- FIG. 6 schematically illustrates an RPU cell which comprises an in-cell complementary reading circuit that implements a differential weight read-out process to enable signed weight values to be directly read out from the RPU cell, according to an embodiment of the invention.
- FIG. 7 schematically illustrates an RPU cell which comprises an in-cell complementary reading circuit that implements a differential weight read-out process to enable signed weight values to be directly read out from the RPU cell, according to another embodiment of the invention.
- FIG. 8 graphically illustrates computer simulation results showing a complementary differential weight read-out process that is implemented based on the RPU cell architecture shown in FIG. 7 .
- FIG. 9 schematically illustrates a resistor-type RPU cell which comprises an in-cell differential read-out system to enable signed weight values to be directly read out from the RPU cell, according to an embodiment of the invention.
- FIG. 10 schematically illustrates a resistor-type RPU cell which comprises an in-cell differential read-out system to enable signed weight values to be directly read out from the RPU cell, according to another embodiment of the invention.
- FIG. 11 schematically illustrates a resistor-type RPU cell which comprises an in-cell differential read-out system to enable signed weight values to be directly read out from the RPU cell, according to another embodiment of the invention.
- FIG. 1 schematically illustrates a RPU system 100 which can be implemented using RPU cell architectures according to embodiments of the invention.
- the RPU system 100 comprises a two-dimensional (2D) crossbar array of RPU cells 110 arranged in a plurality of rows R 1 , R 2 , R 3 , . . . , Rm, and a plurality of columns C 1 , C 2 , C 3 , . . . , Cn.
- the RPU cells 110 in each row R 1 , R 2 , R 3 , . . . , Rm are commonly connected to respective row control lines RL 1 , RL 2 , RL 3 , . . . , RLm (collectively, row control lines RL).
- the RPU cells 110 in each column C 1 , C 2 , C 3 , . . . , Cn are commonly connected to respective column control lines CL 1 , CL 2 , CL 3 , . . . , CLn (collectively, column control lines CL).
- Each RPU cell 110 is connected at (and between) a cross-point (or intersection) of a respective one of the row and column control lines.
- the RPU system 100 further comprises separate blocks of peripheral circuitry 120 - 1 , 120 - 2 , 120 - 3 , . . . , 120 - m (collectively, peripheral circuitry 120 ) connected to respective row control lines RL 1 , RL 2 , RL 3 , . . . , RLm, as well as separate blocks of peripheral circuitry 130 - 1 , 130 - 2 , 130 - 3 , . . . , 130 - n (collectively, peripheral circuitry 130 ) connected to respective column control lines CL 1 , CL 2 , CL 3 , . . . , CLn.
- each block of peripheral circuitry 120 - 1 , 120 - 2 , 120 - 3 , . . . , 120 - m (for the rows) is connected to a data input/output (I/O) interface block 125 .
- each block of peripheral circuitry 130 - 1 , 130 - 2 , 130 - 3 , . . . , 130 - n (for the columns) is connected to a data I/O interface block 135 .
- the RPU system 100 further comprises reference circuitry 140 which comprises various types of circuit blocks such as power, clock, bias and timing circuitry to provide power distribution and control signals and clocking signals for operation of the RPU system 100 .
- the RPU system 100 is configured to perform DNN computations wherein a conductance of each RPU cell 100 represents a matrix element or weight, which can be updated or accessed through operations of the peripheral circuitry 120 and 130 .
- DNN training generally relies on a backpropagation process which comprises three repeating cycles: a forward cycle, a backward cycle, and a weight update cycles.
- the RPU system 100 can be configured to perform all three cycles of the backpropagation process in parallel, thus potentially providing significant acceleration in DNN training with lower power and reduced computation resources.
- the RPU system 100 can be configured to perform vector-matrix multiplication operations in the analog domain in a parallel manner, which leads to a significant enhancement in performance (e.g., 10K ⁇ or more) as compared to a counterpart CPU/GPU implementation.
- each row control line RL and each column control line CL can include two or more control lines connected to the RPU cells 110 in the respective rows and columns, depending on the implementation.
- each row control line RL and each column control line CL comprises a pair of control lines, including an “Update” control line, and a “Read” control line, wherein the cell conductance (e.g., weight value) of a given RPU cell 100 can be adjusted and sensed through the Update and Read control lines, respectively, for a given row and column connected to the given RPU cell 110 .
- each block of peripheral circuitry 120 and 130 comprises essentially the same set of circuit blocks connected to the respective rows and columns in the 2D array of RPU cells 110 to perform forward, backward and update operations for a backpropagation process.
- each block of peripheral circuitry 120 and 130 comprise a stochastic pulse generator block and an update pulse driver block, which are responsive to update vectors that are received during a weight update cycle, to generate and apply a series of stochastically populated pulses to control circuitry within a given RPU cell 110 to update with weight value of the given RPU cell 110 .
- the stochastic pules are transmitted on the Update control lines to control circuitry within the given RPU cell 110 to perform various operations (e.g., multiplication) to update a stored weight value within the RPU cell 110 using known techniques.
- each block of peripheral circuitry 120 and 130 comprise (i) a pulse-width modulation (PWM) circuit and read pulse driver circuit to generate and apply PWM read pulses to the RPU cells 110 , in response to input vector values (read input values) received during forward/backward cycles, and (ii) a current integrator circuit and analog-to-digital (ADC) circuit to integrate read current that is output and accumulated from the connected RPU cells 110 and convert the integrated current into a digital value (read output value) for subsequent computation.
- PWM pulse-width modulation
- ADC analog-to-digital
- the data I/O interface 125 provides an interface that is configured to (i) receive external control signals and apply update and read input values to the blocks of peripheral circuitry 120 , and (ii) receive digital read output values from the blocks of peripheral circuitry 120 , and send the digital read output signals to a remote element for processing.
- the data I/O interface 135 provides an interface that is configured to (i) receive external control signals and apply update and read input values to the blocks of peripheral circuitry 130 , and (ii) receive digital read output values from the blocks of peripheral circuitry 130 , and send the digital read output signals to a remote element for processing.
- Each RPU cell 110 in the RPU system 100 comprises a resistor-like circuit component with a tunable conductance, wherein the conductance value of each RPU cell 110 represents a matrix element or weight.
- the RPU cells 110 can be implemented using a variable resistance device such as a resistive memory element used in resistive random-access memory (RRAM) systems.
- the RPU cells 110 can be implemented using an analog CMOS framework comprising a capacitor and one or more read transistors, wherein the capacitor serves as a memory element in the RPU cell which stores a weight value in the form of capacitor voltage.
- the capacitor voltage, or weight voltage Vw is directly applied to the gate terminal(s) of the read transistor(s) to modulate the channel resistance of the read transistor(s).
- the charge state stored in the capacitor can be accessed by applying small bias across the read transistor(s) to generate drain currents, which generate a read current, I READ , that flows out of RPU cell.
- Deep fully connected neural networks are composed of stacking of multiple fully connected layers such that the signal propagates from input layer to output layer by going through series of linear and non-linear transformations.
- the entire network expresses a single differentiable error function that maps the input data on to class scores at the output layer.
- the network is trained with simple stochastic gradient decent (SGD), in which the error gradient with respect to each parameter is calculated using the backpropagation algorithm.
- SGD stochastic gradient decent
- the backpropagation algorithm is composed of three cycles, forward, backward and weight update that are repeated many times until a convergence criterion is met.
- FIGS. 2A, 2B, and 2C schematically illustrate respective forward pass, backward pass, and weight update operations of a backpropagation algorithm which can be performed using the RPU system 100 of FIG. 1 .
- the resulting vector y of length M is further processed by performing a non-linear activation on each of the elements and then passed to the next layer. Once the information reaches to the final output layer, the error signal is calculated and back propagated through the network.
- stored conductance values in the crossbar array of RPU cells 110 form a matrix, whereas the input vector is transmitted as voltage pulses through each of the input rows R 1 , R 2 , R 3 , . . . , Rm.
- voltage pulses are supplied to the RPU cells 110 from columns CL 1 , CL 2 , CL 3 , . . . , CLn as an input, and the vector-matrix product is computed on the transpose of the weight matrix W.
- the weight matrix W is updated by performing an outer product of the two vectors that are used in the forward and the backward cycles.
- implementing the weight update on a 2D crossbar array of resistive devices locally and all in parallel, independent of the array size requires calculating a vector-vector outer product which consist of a multiplication operation and an incremental weight update to be performed locally at each cross-point (RPU cell 110 ) in the RPU system of FIG. 1 .
- FIG. 2C As schematically illustrated in FIG.
- the weight update process is computed as: w ij ⁇ w ij + ⁇ x i ⁇ j , where w ij represents the weight value for the i th row and the j th column (for simplicity layer index is omitted), where x i is the activity at the input neuron, ⁇ j is the error computed by the output neuron, and where ⁇ denoted a global learning rate.
- All operations on the weight matrix W can be implemented with using the 2D crossbar array of two-terminal RPU device with M rows and N columns where the stored conductance values in the crossbar array form the matrix W.
- input vector x is transmitted as voltage pulses through each of the columns and the resulting vector y can be read as current signals from the rows.
- a vector-matrix product is computed on the transpose of the weight matrix W T .
- voltage pulses representing vectors x and ⁇ are simultaneously supplied from the columns and the rows.
- each RPU cell 110 performs a local multiplication and summation operation by processing the voltage pulses coming from the column and the row and hence achieving an incremental weight update.
- the analog weight storage element in the RPU cell is configured to store weight values, but not the signs (positive or negative) of the weight values.
- the RPU operations for vector-matrix multiplication and other computations require additional processing outside of the RPU cells to determine if the read-out weight values are positive, zero, or negative weight values.
- RPU cells are designed with in-cell differential reading circuitry that is configured to enable weight reading with sign (positive, negative, zero) so that the weight values that are read out from the RPU cells provide the weight value the sign (e.g., positive, negative, zero) of the weight value.
- FIG. 3 conceptually illustrates a process for utilizing a reference current source within an RPU cell to implement a differential read scheme which enables weight values with sign (positive, negative, zero) to be directly read out from the RPU cell, according to an embodiment of the invention.
- FIG. 3 schematically illustrates that an RPU cell has a tunable conductance 300 which can range from a minimum conductance (G_min) to a maximum conductance (G_max).
- a dynamic range of the tunable conductance can include 1000 values from the lowest conductance state (G_min) to the highest conductance state (G_max), and which can be adjusted in an analog and incremental manner.
- FIG. 1 conceptually illustrates a process for utilizing a reference current source within an RPU cell to implement a differential read scheme which enables weight values with sign (positive, negative, zero) to be directly read out from the RPU cell, according to an embodiment of the invention.
- FIG. 3 schematically illustrates that an RPU cell has a tunable conductance 300 which can range from
- the conductance can be converted to a current by an applied voltage whereby a current Iw corresponds to a weight value of the RPU cell.
- the RPU cell has a variable current 310 which can range from a minimum current (Iw_min) to a maximum current (Iw_max), wherein a reference current I REF generated in the RPU cell can be used to implement weight sign.
- I READ Iw ⁇ I REF .
- the magnitude of I READ will indicate a weight value, and the weight sign will depend on the whether Iw is greater than, equal to, or less than, the reference current I REF .
- a positive sign (I READ >0) will be obtained when Iw>I REF .
- a negative sign (I READ ⁇ 0) will be obtained when Iw ⁇ I REF .
- FIG. 4 schematically illustrates an RPU cell 400 which implements a differential read scheme using a reference current source to enable signed weight values to be directly read out from the RPU cell, according to an embodiment of the invention.
- the RPU cell 400 comprises control circuitry 410 , a weight storage device 420 , a read transistor 430 , a current source 440 , and switch elements S 1 and S 2 .
- FIG. 4 illustrates an implementation of a CMOS-based RPU framework in which the weight storage device 420 serves as a memory element that stores a weight voltage Vw which corresponds a weight value of the RPU cell 400 .
- the weight storage device 420 may comprise a capacitor, a battery element, or any suitable storage device or circuit that is configured to store a weight voltage Vw.
- the read transistor 430 comprises an NMOS FET device.
- the weight storage device 420 is connected to a gate terminal of the read transistor 430 .
- a first source/drain terminal of the read transistor 430 is switchably connected to a first read port P 1 of the RPU cell 400 via the switch element S 1 .
- the current source 440 generates a fixed reference current I REF , and is switchably connected to a second source/drain terminal of the read transistor 430 via the second switch element S 2 .
- the second source/drain terminal of the read transistor 430 is connected to a second read port P 2 of the RPU cell 400 .
- the control circuit 410 can be configured using any suitable circuit architecture that is configured to adjust the weight voltage Vw of the weight storage device 420 .
- the control circuitry 410 within the RPU cell would have input ports connected to “update” control lines of the row and column lines RL and CL to receive control signals from the peripheral circuitry 120 and 130 for updating the weight value of the RPU cell.
- the switch elements would be responsive to sense control signals (generated by the peripheral circuitry 120 and 130 to perform a weight value sensing operation) to connect the first/source drain terminal of the read transistor 430 to the first read port P 1 , and to connect the current source 440 to the second source/drain terminal of the read transistor 430 .
- the weight voltage Vw is applied to a gate terminal of the read transistor 430 to modulate the channel resistance of the read transistor 430 and generate a weight current Iw which corresponds to the weight voltage Vw.
- a read current I READ output from the second port P 2 of the RPU cell 400 comprises a value of Iw ⁇ I REF .
- the weight sign be deemed positive when the read current I READ is greater than zero, and the weight singe will be deemed negative when the read current I READ is less than zero.
- FIGS. 3 and 4 schematically illustrate the concept of implementing a differential read-out scheme using a reference current source to enable signed weight values to be directly read out from the RPU cell
- this concept can be extended to an in-cell complementary read out scheme comprising a differential weight read-out framework, as will be explained in further detail with reference to FIGS. 5 and 6 .
- a differential weight read-out framework two current reading elements are utilized which operate in a complementary manner, which serves to increase the dynamic range 2 X more than the single element reading scheme discussed above with reference to FIGS. 3 and 4 .
- FIG. 5 conceptually illustrates an in-cell complementary reading scheme 500 which implements a differential weight read-out process to enable signed weight values to be directly read out from an RPU cell, according to an embodiment of the invention.
- FIG. 5 conceptually illustrates an RPU cell comprising a first reading element 510 which generates a first (positive) weight current Iw + , and a second reading element 512 which generates a second (negative) weight current Iw ⁇ that is complementary to the first weight current Iw + .
- the first (positive) weight current Iw + can range from a minimum current (Iw + _min) to a maximum current (Iw + _max) based on a value of a weight voltage Vw
- the second (negative) weight current Iw ⁇ can range from a minimum current (Iw ⁇ _min) to a maximum current (Iw ⁇ _max) based on the value of the weight voltage Vw.
- I READ Iw + ⁇ Iw ⁇ ⁇ 0
- FIG. 6 schematically illustrates an RPU cell 600 which comprises an in-cell complementary reading circuit that implements a differential weight read-out process to enable signed weight values to be directly read out from the RPU cell 600 , according to an embodiment of the invention.
- the RPU cell 600 comprises control circuitry 610 , a weight storage device 620 , a first read transistor 630 , a second read transistor 632 , and current integration circuitry 640 .
- the RPU cell 600 of FIG. 6 implements a CMOS-based RPU framework in which the weight storage device 620 serves as a memory element that stores a weight voltage Vw which corresponds a weight value of the RPU cell 600 .
- the weight storage device 620 may comprise a capacitor, a battery element, or any suitable storage device or circuit that is configured to store a weight voltage Vw.
- the first read transistor 630 comprises an NMOS FET device
- the second read transistor 632 comprises a PMOS FET device.
- the first and second read transistors 630 and 632 have gate terminals that are commonly connected to the weight storage device 620 such that during a sensing operation, the weight voltage Vw is concurrently applied to the gate terminals of the first and second read transistors 630 and 632
- the first and second read transistors have first source/drain terminals that are connected to separate read control lines XRN and XRP, respectively, and second source/drain terminals that are connected to the current integration circuit 640 .
- the control circuit 610 can be configured using any suitable circuit architecture that is configured to adjust the weight voltage Vw of the weight storage device 620 .
- the control circuitry 610 within the RPU cell would have input ports connected to “update” control lines of the row and column lines RL and CL to receive control signals from the peripheral circuitry 120 and 130 for updating the weight value of the RPU cell.
- the read sensing lines XRN and XRP would apply complementary sense control signals to the first source/drain terminals of the first and second read transistors 630 and 632 , the complementary sense control signals being generated by the peripheral circuitry 120 and 130 to perform a weight value sensing operation).
- the weight voltage Vw is applied to gate terminals of the first and second read transistors 630 and 632 to modulate the channel resistance of the first and second read transistors 630 and 632 to generate respective weight currents Iw + and Iw ⁇ , which correspond to the weight voltage Vw.
- the current integration circuitry 640 can be implemented using any suitable circuit framework which is configured to add, subtract, or otherwise integrate the input differential weight currents Iw + and Iw- to generate the read current I READ , which provides a signed weight value of the RPU cell 600 .
- FIG. 7 schematically illustrates an RPU cell 700 which comprises an in-cell complementary reading circuit that implements a differential weight read-out process to enable signed weight values to be directly read out from the RPU cell 700 , according to another embodiment of the invention.
- the RPU cell 700 is similar to the RPU cell 600 discussed above with reference to FIG. 6 , except that the RPU cell 700 of FIG. 7 implements a current mirror circuit 710 which is configured to perform an in-cell subtraction function without significantly increasing the footprint of the RPU cell 700 .
- the current mirror circuit 710 provides an example embodiment for implementing the current integration circuitry 640 of FIG. 6 .
- the current mirror circuit 710 comprises a first NMOS transistor M 1 and a second NMOS transistor M 2 .
- the NMOS transistors M 1 and M 2 are the same size so that they have equal drain currents.
- the complementary weight current Iw ⁇ which is generated in the second read transistor 632 in response to the weight voltage Vw applied to the gate terminal of the second read transistor 632 , serves as a reference current for the current mirror circuit 710 and is mirrored in the NMOS transistor M 1 .
- the current mirror circuit 710 serves as a current sink which sinks the mirrored current Iw ⁇ from the first current Iw + which is generated in the first read transistor 630 in response to the weight voltage Vw applied to the gate terminal of the first read transistor 630 .
- the in-cell complementary reading circuit framework for differential weight read-out in the RPU cell 700 of FIG. 7 is advantageous for various reasons.
- the in-cell complementary reading circuit framework effectively enhances the dynamic range by doubling the current difference values due to the differential implementation.
- no additional cross bar array is required for the reading the complementary currents Iw + and Iw ⁇ .
- the average current flowing through column read lines can be significantly reduced, which enables are larger RPU array size.
- the in-cell implementation allows for compensation in transistor variations across the RPU array.
- FIG. 8 graphically illustrates computer simulation results showing a complementary differential weight read-out process that is implemented based on the RPU cell architecture shown in FIG. 7 .
- FIG. 8 comprises a graph 800 which illustrates simulated curves for weight current Iw + and Iw ⁇ and a read current I READ (Iw + ⁇ Iw ⁇ ) (in micro Amps) as a function of weight voltage Vw.
- FIG. 8 illustrates an optimal operating range of the weight voltage Vw of 0.4V to 0.6V, wherein the wherein I READ (Iw + ⁇ Iw ⁇ ) has a linear slope. While FIG. 8 illustrates one example simulation for specific bias and operating voltages, additional computer simulations have shown that the RPU cell architecture of FIG. 7 is robust in operation such that the first and second read transistors can be configured to operate in a saturation region with higher Vds values (e.g., 0.6) while still providing essentially a 1:1 correspondence between current output as a function of weight voltage in a wide operating range.
- Vds e.g., 0.6
- resistor-type analog RPU cells can be implemented using memristors or resistive memory cells that are utilized in resistive random-access memory (RRAM).
- RRAM resistive random-access memory
- a resistive memory cell comprises a metal-insulator-metal (MIM) structure in which the insulator layer (e.g., metal-oxide) exhibits a variable resistance through a resistive switching behavior.
- MIM metal-insulator-metal
- FIG. 9 schematically illustrates a resistor-type RPU cell 900 which comprises an in-cell differential read-out system to enable signed weight values to be directly read out from the RPU cell 900 , according to another embodiment of the invention.
- the RPU cell 900 comprises resistance control circuitry 910 , a variable resistance device 920 , a first resistance device 922 , and a current mirror circuit 930 .
- FIG. 9 illustrates an implementation of a resistor-type RPU cell framework in which the resistance value of the variable resistance device 920 corresponds to the weight value of the RPU cell 900 .
- the variable resistance device 920 may comprise a memristor device or resistive memory cell (e.g., MIM device) having a resistance value that can be modulated (or otherwise adjusted) by the resistance control circuitry 910 using known circuits and methods.
- MIM device resistive memory cell
- the resistance control circuit 910 within the RPU cell would have input ports connected to “update” control lines of the row and column lines RL and CL to receive control signals from the peripheral circuitry 120 and 130 for updating the weight value of the RPU cell 900 by adjusting the resistance value of the variable resistance device 920 .
- the variable resistance device 920 generates a variable current I 1 based on a control voltage (e.g., VDD) applied to a row control line (RL) attached to the RPU cell 900 , wherein the value of current I 1 corresponds to the weight value of the RPU cell 900 .
- the fixed resistance device 930 generates a fixed current I 2 based on the control voltage (e.g., VDD) applied to the row control line (RL) attached to the RPU cell 900 , wherein the fixed current I 2 serves as a fixed reference current.
- the current mirror circuit 930 in FIG. 9 comprises a first NMOS transistor M 1 and a second NMOS transistor M 2 .
- the NMOS transistors M 1 and M 2 are the same size so that they have equal drain currents.
- the fixed reference current I 2 which flows through the fixed resistor device 922 , serves as a reference current for the current mirror circuit 930 and is mirrored in the NMOS transistor M 1 .
- the current mirror circuit 930 serves as a current sink which sinks (subtracts) the mirrored current I 2 from the variable current I 1 which flows from the variable resistance device 920 .
- the sign of RPU cell weight is deemed positive when the read current I READ is greater than zero, and the sign of the RPU cell weight is deemed negative when the read current I READ is less than zero.
- FIG. 10 schematically illustrates a resistor-type RPU cell 1000 which comprises an in-cell differential read-out system to enable signed weight values to be directly read out from the RPU cell 1000 , according to another embodiment of the invention.
- the RPU cell 1000 of FIG. 10 schematically illustrates a resistor-type RPU cell 1000 which comprises an in-cell differential read-out system to enable signed weight values to be directly read out from the RPU cell 1000 , according to another embodiment of the invention.
- RPU cell 10 is similar to the RPU cell 900 of FIG. 9 except that a PMOS transistor M 2 and an operational amplifier 1010 are incorporated with the RPU cell 1000 to prevent or significantly limit the voltage variation between nodes N 1 and N 2 and thereby prevent variation in the fixed reference current I 2 that is input to the current mirror circuit 930 .
- the PMOS transistor M 3 is serially connected between the fixed resistor device 922 and a reference input node of the current mirror circuit 930 .
- a gate terminal of the PMOS transistor M 3 is connected to an output of the operational amplifier 1010 .
- the operational amplifier 1010 comprises a non-inverting input (+) connected to the node N 1 , and an inverting input ( ⁇ ) connected to the node N 2 , wherein the output of the operational amplifier 1010 is proportional to V + ⁇ V ⁇ (i.e., the difference between the voltage applied to the non-inverting and inverting inputs).
- the operational amplifier 1010 serves to maintain the voltage V 2 at node N 2 equal to the voltage V 1 at node N 1 , and maintain the current I 2 which flows through the PMOS transistor M 3 to remain at a fixed reference value (with no or insubstantial variation).
- FIG. 11 schematically illustrates a resistor-type RPU cell 1100 which comprises an in-cell differential read-out system to enable signed weight values to be directly read out from the RPU cell 1100 , according to another embodiment of the invention.
- the RPU cell 1100 of FIG. 11 is similar to the RPU cell 1000 of FIG. 10 , but FIG. 11 illustrates an example circuit configuration of the operational amplifier 1010 of FIG. 10 .
- FIG. 11 schematically illustrates an embodiment of an operational amplifier 1110 comprising transistors M 10 , M 11 , M 12 , M 13 and M 14 .
- the transistors M 10 and M 11 comprise PMOS transistors which form a differential input circuit, wherein a gate terminal of the PMOS transistor M 10 comprises a non-inverting input of the operational amplifier 1110 , which is connected to the node N 1 , and wherein a gate terminal of the PMOS transistor M 11 comprises an inverting input of the operational amplifier 1110 , which is connected to the node N 2 .
- the transistors M 12 and M 13 comprises NMOS transistors which form a current mirror load for the operational amplifier 1110 .
- the transistor M 14 comprises a PMOS transistor which serves as a constant current biasing transistor for the operational amplifier 1110 .
- a bias voltage VBP from a global biasing circuit is applied to the gate terminal of the transistor M 14 .
- An output node N OUT of the operational amplifier 1110 is connected to the gate terminal of the PMOS transistor M 3 .
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Neurology (AREA)
- Read Only Memory (AREA)
Abstract
Description
- This disclosure generally relates to resistive processing unit (RPU) frameworks and, in particular, techniques for reading weight values stored in RPU storage cells.
- In recent years, deep neural network (DNN) based models have made significant progress due to the availability of large labeled datasets and continuous improvements in computation resources. DNNs are utilized in various applications including, for example, object/speech recognition, language translation, pattern extraction, and image processing. The quality of the DNN models depends on the processing of a large amount of training data and an increased complexity of the neural network. In this regard, training a complex DNN model is a time consuming and computationally intensive task which can require many days or weeks to perform using parallel and distributed computing frameworks with many computing nodes (e.g., datacenter-scale computational resources) to complete the training of the DNN model.
- To reduce training time, hardware acceleration techniques for processing DNN workloads have been pursued either in conventional CMOS technologies or by using emerging non-volatile memory (NVM) technologies. However, it has been found that resistive processing unit (RPU) accelerator devices have the potential to accelerate DNN training by orders of magnitude, while using less power, as compared to conventional hardware acceleration techniques. DNN training generally relies on a backpropagation algorithm which includes three repeating cycles: forward, backward and weight update. It has been determined that RPU accelerator devices which are based on a two-dimensional (2D) crossbar array of RPU storage cells, can be configured to perform all three cycles of the backpropagation algorithm in parallel, thus potentially providing significant acceleration in DNN training with lower power and reduced computation resources compared to state-of-the-art implementations using central processing units (CPUs) and graphics processing units (GPUs). An RPU accelerator can store and update weight values locally, thereby minimizing data movement during training and fully exploiting the locality and the parallelism of the DNN training process. Analog weight storage elements in RPU storage cells are capable of storing weight values without sign (e.g., positive, negative, zero). However, RPU operations for DNN training and other applications require processing of positive, zero and negative weight values.
- Embodiments of the invention include in-cell differential read-out circuits and methods to enable signed weight values to be directly read out from RPU cells of an RPU system.
- For example, one embodiment includes an RPU device, which comprises: a weight storage device configured to store a weight voltage which corresponds to a weight value of the RPU device; a read transistor comprising a gate terminal, a first source/drain terminal, and a second source/drain terminal, wherein the gate terminal is connected to the weight storage device, wherein the first source/drain terminal is connected to a first control port of the RPU device, and wherein the second source/drain terminal is connected to a second control port of the RPU device; and a current source connected to the second source/drain terminal of the read transistor. The current source is configured to generate a fixed reference current. The read transistor is configured to generate a weight current in response to the weight voltage applied to the gate terminal of the read transistor. The RPU device is configured to output a read current from the second control port, wherein the read current comprises a magnitude and sign which represents a signed weight value of the RPU device. The magnitude of the read current is equal to a difference between the weight current generated by the read transistor and the fixed reference current of the current source. The sign of the read current is deemed positive when the weight current is greater than the fixed reference current, and the sign of the read current is deemed negative when the weight current is less than the fixed reference current.
- Another embodiment includes an RPU device, which comprises: a weight storage device configured to store a weight voltage which corresponds to a weight value of the RPU device; a current integration circuit; a first read transistor comprising a gate terminal, a first source/drain terminal, and a second source/drain terminal, wherein the gate terminal is connected to the weight storage device, wherein the first source/drain terminal is connected to a first control port of the RPU device, and wherein the second source/drain terminal is connected to the current integration circuit; and a second read transistor comprising a gate terminal, a first source/drain terminal, and a second source/drain terminal, wherein the gate terminal is connected to the weight storage device, wherein the first source/drain terminal is connected to a second control port of the RPU device, and wherein the second source/drain terminal is connected to the current integration circuit. The first and second read transistors are configured to generate first and second weight currents, respectively, in response to the weight voltage applied to the gate terminals of the first and second read transistors, wherein the first and second weight currents are complementary currents. The current integration circuit is configured to generate a read current, which is output from a third port of the RPU device, wherein the read current comprises a magnitude and sign which represents a signed weight value of the RPU device. The magnitude of the read current is equal to a difference between the first weight current generated by the first read transistor and the second weight current generated by the second read transistor. The sign of the read current is deemed positive when the first weight current is greater than the second weight current, and the sign of the read current is deemed negative when the first weight current is less than the second weight current.
- Another embodiment includes an RPU device, which comprises: a current mirror circuit comprising a reference input node and an output node; a variable resistor device having a resistance value which corresponds to a weight value of the RPU device, wherein the variable resistor device is connected between a first port and a second port of the RPU device, and connected to the output node of the current mirror circuit; and a fixed resistor device having a fixed resistance value, wherein the fixed resistor device is connected between the first port of the RPU device and the reference input node of the current mirror circuit. The variable resistor device is configured to generate a first current in response to a control voltage applied to the first port of the RPU device.
- The fixed resistor device is configured to generate a second current in response to the control voltage applied to the first port of the RPU device. The current mirror circuit is configured to mirror the second current, and sink the mirrored second current into the output node of the current mirror circuit and cause a read current to be generated and output from the second port of the RPU device. The read current comprises a magnitude and sign which represents a signed weight value of the RPU device. The magnitude of the read current is equal to a difference between the first current the second mirrored current. The sign of the read current is deemed positive when the first current is greater than the second mirrored current, and the sign of the read current is deemed negative when the first current is less than the second mirrored current.
- Other embodiments will be described in the following detailed description of embodiments, which is to be read in conjunction with the accompanying figures.
-
FIG. 1 schematically illustrates a resistive processing unit system which can be implemented using RPU cell architectures according to embodiments of the invention. -
FIG. 2A schematically illustrates a forward pass operation of a backpropagation process, which can be performed using the RPU system ofFIG. 1 . -
FIG. 2B schematically illustrates a backward pass operation of a backpropagation process, which can be performed using the RPU system ofFIG. 1 . -
FIG. 2C schematically illustrates a weight update operation of a backpropagation process, which can be performed using the RPU system ofFIG. 1 . -
FIG. 3 conceptually illustrates a process for utilizing a reference current source within an RPU cell to implement a differential read scheme which enables signed weight values to be directly read out from the RPU cell, according to an embodiment of the invention. -
FIG. 4 schematically illustrates an RPU cell framework which implements a differential read scheme using a reference current source to enable signed weight values to be directly read out from the RPU cell, according to an embodiment of the invention. -
FIG. 5 conceptually illustrates an in-cell complementary differential read scheme to enable signed weight values to be directly read out from an RPU cell, according to an embodiment of the invention. -
FIG. 6 schematically illustrates an RPU cell which comprises an in-cell complementary reading circuit that implements a differential weight read-out process to enable signed weight values to be directly read out from the RPU cell, according to an embodiment of the invention. -
FIG. 7 schematically illustrates an RPU cell which comprises an in-cell complementary reading circuit that implements a differential weight read-out process to enable signed weight values to be directly read out from the RPU cell, according to another embodiment of the invention. -
FIG. 8 graphically illustrates computer simulation results showing a complementary differential weight read-out process that is implemented based on the RPU cell architecture shown inFIG. 7 . -
FIG. 9 schematically illustrates a resistor-type RPU cell which comprises an in-cell differential read-out system to enable signed weight values to be directly read out from the RPU cell, according to an embodiment of the invention. -
FIG. 10 schematically illustrates a resistor-type RPU cell which comprises an in-cell differential read-out system to enable signed weight values to be directly read out from the RPU cell, according to another embodiment of the invention. -
FIG. 11 schematically illustrates a resistor-type RPU cell which comprises an in-cell differential read-out system to enable signed weight values to be directly read out from the RPU cell, according to another embodiment of the invention. - Embodiments of the invention will now be discussed in further detail with regard to in-cell differential read-out circuits and methods which enable signed weight values to be directly read out from RPU cells of an RPU system. It is to be noted that the same or similar reference numbers are used throughout the drawings to denote the same or similar features, elements, or structures, and thus, a detailed explanation of the same or similar features, elements, or structures will not be repeated for each of the drawings.
-
FIG. 1 schematically illustrates aRPU system 100 which can be implemented using RPU cell architectures according to embodiments of the invention. TheRPU system 100 comprises a two-dimensional (2D) crossbar array ofRPU cells 110 arranged in a plurality of rows R1, R2, R3, . . . , Rm, and a plurality of columns C1, C2, C3, . . . , Cn. TheRPU cells 110 in each row R1, R2, R3, . . . , Rm are commonly connected to respective row control lines RL1, RL2, RL3, . . . , RLm (collectively, row control lines RL). TheRPU cells 110 in each column C1, C2, C3, . . . , Cn are commonly connected to respective column control lines CL1, CL2, CL3, . . . , CLn (collectively, column control lines CL). EachRPU cell 110 is connected at (and between) a cross-point (or intersection) of a respective one of the row and column control lines. - The
RPU system 100 further comprises separate blocks of peripheral circuitry 120-1, 120-2, 120-3, . . . , 120-m (collectively, peripheral circuitry 120) connected to respective row control lines RL1, RL2, RL3, . . . , RLm, as well as separate blocks of peripheral circuitry 130-1, 130-2, 130-3, . . . , 130-n (collectively, peripheral circuitry 130) connected to respective column control lines CL1, CL2, CL3, . . . , CLn. Further, each block of peripheral circuitry 120-1, 120-2, 120-3, . . . , 120-m (for the rows) is connected to a data input/output (I/O)interface block 125. Similarly, each block of peripheral circuitry 130-1, 130-2, 130-3, . . . , 130-n (for the columns) is connected to a data I/O interface block 135. TheRPU system 100 further comprisesreference circuitry 140 which comprises various types of circuit blocks such as power, clock, bias and timing circuitry to provide power distribution and control signals and clocking signals for operation of theRPU system 100. - In some embodiments, the
RPU system 100 is configured to perform DNN computations wherein a conductance of eachRPU cell 100 represents a matrix element or weight, which can be updated or accessed through operations of theperipheral circuitry 120 and 130. As noted above, DNN training generally relies on a backpropagation process which comprises three repeating cycles: a forward cycle, a backward cycle, and a weight update cycles. TheRPU system 100 can be configured to perform all three cycles of the backpropagation process in parallel, thus potentially providing significant acceleration in DNN training with lower power and reduced computation resources. TheRPU system 100 can be configured to perform vector-matrix multiplication operations in the analog domain in a parallel manner, which leads to a significant enhancement in performance (e.g., 10K× or more) as compared to a counterpart CPU/GPU implementation. - While the row control lines RL and column control lines CL are each shown in
FIG. 1 as a single line for ease of illustration, it is to be understood that the row and column control lines can include two or more control lines connected to theRPU cells 110 in the respective rows and columns, depending on the implementation. For example, in one embodiment, each row control line RL and each column control line CL comprises a pair of control lines, including an “Update” control line, and a “Read” control line, wherein the cell conductance (e.g., weight value) of a givenRPU cell 100 can be adjusted and sensed through the Update and Read control lines, respectively, for a given row and column connected to the givenRPU cell 110. - In some embodiments, each block of
peripheral circuitry 120 and 130 comprises essentially the same set of circuit blocks connected to the respective rows and columns in the 2D array ofRPU cells 110 to perform forward, backward and update operations for a backpropagation process. For example, in one embodiment, to support a RPU cell update operation (e.g., update weight value of given RPU cell 110), each block ofperipheral circuitry 120 and 130 comprise a stochastic pulse generator block and an update pulse driver block, which are responsive to update vectors that are received during a weight update cycle, to generate and apply a series of stochastically populated pulses to control circuitry within a givenRPU cell 110 to update with weight value of the givenRPU cell 110. The stochastic pules are transmitted on the Update control lines to control circuitry within the givenRPU cell 110 to perform various operations (e.g., multiplication) to update a stored weight value within theRPU cell 110 using known techniques. - In addition, in one embodiment, to support a RPU cell read and sensing operations (e.g., read a weight value of given RPU cell 110), each block of
peripheral circuitry 120 and 130 comprise (i) a pulse-width modulation (PWM) circuit and read pulse driver circuit to generate and apply PWM read pulses to theRPU cells 110, in response to input vector values (read input values) received during forward/backward cycles, and (ii) a current integrator circuit and analog-to-digital (ADC) circuit to integrate read current that is output and accumulated from theconnected RPU cells 110 and convert the integrated current into a digital value (read output value) for subsequent computation. - The data I/
O interface 125 provides an interface that is configured to (i) receive external control signals and apply update and read input values to the blocks ofperipheral circuitry 120, and (ii) receive digital read output values from the blocks ofperipheral circuitry 120, and send the digital read output signals to a remote element for processing. Similarly, the data I/O interface 135 provides an interface that is configured to (i) receive external control signals and apply update and read input values to the blocks of peripheral circuitry 130, and (ii) receive digital read output values from the blocks of peripheral circuitry 130, and send the digital read output signals to a remote element for processing. - Each
RPU cell 110 in theRPU system 100 comprises a resistor-like circuit component with a tunable conductance, wherein the conductance value of eachRPU cell 110 represents a matrix element or weight. For example, in some embodiments, theRPU cells 110 can be implemented using a variable resistance device such as a resistive memory element used in resistive random-access memory (RRAM) systems. In other embodiments, theRPU cells 110 can be implemented using an analog CMOS framework comprising a capacitor and one or more read transistors, wherein the capacitor serves as a memory element in the RPU cell which stores a weight value in the form of capacitor voltage. As explained in further detail below, the capacitor voltage, or weight voltage Vw, is directly applied to the gate terminal(s) of the read transistor(s) to modulate the channel resistance of the read transistor(s). The charge state stored in the capacitor can be accessed by applying small bias across the read transistor(s) to generate drain currents, which generate a read current, IREAD, that flows out of RPU cell. - Deep fully connected neural networks are composed of stacking of multiple fully connected layers such that the signal propagates from input layer to output layer by going through series of linear and non-linear transformations. The entire network expresses a single differentiable error function that maps the input data on to class scores at the output layer. Typically, the network is trained with simple stochastic gradient decent (SGD), in which the error gradient with respect to each parameter is calculated using the backpropagation algorithm. The backpropagation algorithm is composed of three cycles, forward, backward and weight update that are repeated many times until a convergence criterion is met.
FIGS. 2A, 2B, and 2C schematically illustrate respective forward pass, backward pass, and weight update operations of a backpropagation algorithm which can be performed using theRPU system 100 ofFIG. 1 . - For a single fully connected layer where N input neurons are connected to M output (or hidden) neurons, the forward pass (
FIG. 2A ) involves computing a vector-matrix multiplication y=Wx, where the vector x of length N represents the activities of the input neurons and the matrix W of size M×N stores the weight values between each pair of input and output neurons. The resulting vector y of length M is further processed by performing a non-linear activation on each of the elements and then passed to the next layer. Once the information reaches to the final output layer, the error signal is calculated and back propagated through the network. In forward cycle, stored conductance values in the crossbar array ofRPU cells 110 form a matrix, whereas the input vector is transmitted as voltage pulses through each of the input rows R1, R2, R3, . . . , Rm. - The backward cycle (
FIG. 2B ) on a single layer also involves a vector-matrix multiplication on the transpose of a weight matrix, z=WTδ, where W denotes the weight matrix, where the vector 6 of length M represents the error calculated by the output neurons, and where the vector z of length N is further processed using the derivative of neuron non-linearity and then passed down to the previous layers. In a backward cycle, voltage pulses are supplied to theRPU cells 110 from columns CL1, CL2, CL3, . . . , CLn as an input, and the vector-matrix product is computed on the transpose of the weight matrix W. - Finally, in an update cycle (
FIG. 2C ), the weight matrix W is updated by performing an outer product of the two vectors that are used in the forward and the backward cycles. In particular, implementing the weight update on a 2D crossbar array of resistive devices locally and all in parallel, independent of the array size, requires calculating a vector-vector outer product which consist of a multiplication operation and an incremental weight update to be performed locally at each cross-point (RPU cell 110) in the RPU system ofFIG. 1 . As schematically illustrated inFIG. 2C , the weight update process is computed as: wij←wij+ηxi×δj, where wij represents the weight value for the ith row and the jth column (for simplicity layer index is omitted), where xi is the activity at the input neuron, δj is the error computed by the output neuron, and where η denoted a global learning rate. - All operations on the weight matrix W can be implemented with using the 2D crossbar array of two-terminal RPU device with M rows and N columns where the stored conductance values in the crossbar array form the matrix W. In the forward cycle, input vector x is transmitted as voltage pulses through each of the columns and the resulting vector y can be read as current signals from the rows. Similarly, when voltage pulses are supplied from the rows as an input in the backward cycle, then a vector-matrix product is computed on the transpose of the weight matrix WT. Finally, in the update cycle, voltage pulses representing vectors x and δ are simultaneously supplied from the columns and the rows. At this setting, each
RPU cell 110 performs a local multiplication and summation operation by processing the voltage pulses coming from the column and the row and hence achieving an incremental weight update. - All three operating modes described above allow the arrays of RPU cells that constitute the network to be active in all three cycles and hence enable a very efficient implementation of the backpropagation algorithm. However, in conventional frameworks, the analog weight storage element in the RPU cell is configured to store weight values, but not the signs (positive or negative) of the weight values. The RPU operations for vector-matrix multiplication and other computations require additional processing outside of the RPU cells to determine if the read-out weight values are positive, zero, or negative weight values. In accordance with embodiments of the invention, RPU cells are designed with in-cell differential reading circuitry that is configured to enable weight reading with sign (positive, negative, zero) so that the weight values that are read out from the RPU cells provide the weight value the sign (e.g., positive, negative, zero) of the weight value.
-
FIG. 3 conceptually illustrates a process for utilizing a reference current source within an RPU cell to implement a differential read scheme which enables weight values with sign (positive, negative, zero) to be directly read out from the RPU cell, according to an embodiment of the invention.FIG. 3 schematically illustrates that an RPU cell has atunable conductance 300 which can range from a minimum conductance (G_min) to a maximum conductance (G_max). In one embodiment, depending on the implementation, a dynamic range of the tunable conductance can include 1000 values from the lowest conductance state (G_min) to the highest conductance state (G_max), and which can be adjusted in an analog and incremental manner.FIG. 3 further illustrates that the conductance can be converted to a current by an applied voltage whereby a current Iw corresponds to a weight value of the RPU cell. In this regard, the RPU cell has a variable current 310 which can range from a minimum current (Iw_min) to a maximum current (Iw_max), wherein a reference current IREF generated in the RPU cell can be used to implement weight sign. - In particular, as shown in
FIG. 3 , a read current IREAD can be output from an RPU cell to provide a weight value and weight sign, wherein IREAD=Iw−IREF. With this differential scheme, the magnitude of IREAD will indicate a weight value, and the weight sign will depend on the whether Iw is greater than, equal to, or less than, the reference current IREF. A positive sign (IREAD>0) will be obtained when Iw>IREF. A zero value (IREAD=0) will be obtained when Iw=IREF. A negative sign (IREAD<0) will be obtained when Iw<IREF. -
FIG. 4 schematically illustrates an RPU cell 400 which implements a differential read scheme using a reference current source to enable signed weight values to be directly read out from the RPU cell, according to an embodiment of the invention. The RPU cell 400 comprisescontrol circuitry 410, aweight storage device 420, aread transistor 430, acurrent source 440, and switch elements S1 and S2.FIG. 4 illustrates an implementation of a CMOS-based RPU framework in which theweight storage device 420 serves as a memory element that stores a weight voltage Vw which corresponds a weight value of the RPU cell 400. Theweight storage device 420 may comprise a capacitor, a battery element, or any suitable storage device or circuit that is configured to store a weight voltage Vw. In one embodiment, theread transistor 430 comprises an NMOS FET device. Theweight storage device 420 is connected to a gate terminal of theread transistor 430. A first source/drain terminal of theread transistor 430 is switchably connected to a first read port P1 of the RPU cell 400 via the switch element S1. Thecurrent source 440 generates a fixed reference current IREF, and is switchably connected to a second source/drain terminal of theread transistor 430 via the second switch element S2. The second source/drain terminal of theread transistor 430 is connected to a second read port P2 of the RPU cell 400. - The
control circuit 410 can be configured using any suitable circuit architecture that is configured to adjust the weight voltage Vw of theweight storage device 420. In theexemplary RPU system 100 ofFIG. 1 , thecontrol circuitry 410 within the RPU cell would have input ports connected to “update” control lines of the row and column lines RL and CL to receive control signals from theperipheral circuitry 120 and 130 for updating the weight value of the RPU cell. The switch elements would be responsive to sense control signals (generated by theperipheral circuitry 120 and 130 to perform a weight value sensing operation) to connect the first/source drain terminal of theread transistor 430 to the first read port P1, and to connect thecurrent source 440 to the second source/drain terminal of theread transistor 430. - During a sensing operation, the weight voltage Vw is applied to a gate terminal of the
read transistor 430 to modulate the channel resistance of theread transistor 430 and generate a weight current Iw which corresponds to the weight voltage Vw. With the fixedcurrent source 440 connected to the second source/drain terminal of theread transistor 430, a read current IREAD output from the second port P2 of the RPU cell 400 comprises a value of Iw−IREF. As noted above, with this differential scheme, since the value of the reference current IREF is a known fixed value, the weight sign be deemed positive when the read current IREAD is greater than zero, and the weight singe will be deemed negative when the read current IREAD is less than zero. - While
FIGS. 3 and 4 schematically illustrate the concept of implementing a differential read-out scheme using a reference current source to enable signed weight values to be directly read out from the RPU cell, this concept can be extended to an in-cell complementary read out scheme comprising a differential weight read-out framework, as will be explained in further detail with reference toFIGS. 5 and 6 . With a differential weight read-out framework, two current reading elements are utilized which operate in a complementary manner, which serves to increase the dynamic range 2X more than the single element reading scheme discussed above with reference toFIGS. 3 and 4 . -
FIG. 5 conceptually illustrates an in-cell complementary reading scheme 500 which implements a differential weight read-out process to enable signed weight values to be directly read out from an RPU cell, according to an embodiment of the invention. In particular,FIG. 5 conceptually illustrates an RPU cell comprising afirst reading element 510 which generates a first (positive) weight current Iw+, and asecond reading element 512 which generates a second (negative) weight current Iw− that is complementary to the first weight current Iw+. In this conceptual embodiment, it is assumed that the first (positive) weight current Iw+ can range from a minimum current (Iw+_min) to a maximum current (Iw+_max) based on a value of a weight voltage Vw, and the second (negative) weight current Iw− can range from a minimum current (Iw−_min) to a maximum current (Iw−_max) based on the value of the weight voltage Vw. -
FIG. 5 conceptually illustrates the in-cell complementary reading scheme 500 being configured to generate a read current IREAD having a weight value IREAD=Iw+−Iw−, a weight sign which depends on the whether the first current Iw+ is greater than, equal to, or less than, the second current Iw−. In particular,FIG. 5 conceptually illustrates afirst state 500A of the complementary reading scheme 500 in which a positive sign of the weight value is obtained when Iw+ is greater than Iw− (i.e., IREAD=Iw+−Iw−>0), and a second 500B state in which a negative sign of the weight value is obtained when Iw+ is less than Iw− (i.e., IREAD=Iw+−Iw−<0). A zero value (IREAD=0) will be obtained when Iw+=Iw−. -
FIG. 6 schematically illustrates an RPU cell 600 which comprises an in-cell complementary reading circuit that implements a differential weight read-out process to enable signed weight values to be directly read out from the RPU cell 600, according to an embodiment of the invention. The RPU cell 600 comprisescontrol circuitry 610, aweight storage device 620, afirst read transistor 630, asecond read transistor 632, andcurrent integration circuitry 640. - Similar to the RPU cell frameworks discussed above with reference to
FIG. 5 , the RPU cell 600 ofFIG. 6 implements a CMOS-based RPU framework in which theweight storage device 620 serves as a memory element that stores a weight voltage Vw which corresponds a weight value of the RPU cell 600. Theweight storage device 620 may comprise a capacitor, a battery element, or any suitable storage device or circuit that is configured to store a weight voltage Vw. - In one embodiment, the
first read transistor 630 comprises an NMOS FET device, and thesecond read transistor 632 comprises a PMOS FET device. The first andsecond read transistors weight storage device 620 such that during a sensing operation, the weight voltage Vw is concurrently applied to the gate terminals of the first andsecond read transistors current integration circuit 640. - The
control circuit 610 can be configured using any suitable circuit architecture that is configured to adjust the weight voltage Vw of theweight storage device 620. In theexemplary RPU system 100 ofFIG. 1 , thecontrol circuitry 610 within the RPU cell would have input ports connected to “update” control lines of the row and column lines RL and CL to receive control signals from theperipheral circuitry 120 and 130 for updating the weight value of the RPU cell. The read sensing lines XRN and XRP would apply complementary sense control signals to the first source/drain terminals of the first andsecond read transistors peripheral circuitry 120 and 130 to perform a weight value sensing operation). - During a sensing operation, the weight voltage Vw is applied to gate terminals of the first and
second read transistors second read transistors current integration circuitry 640 is configured to generate and output the read current IREAD (from output port P2) based on a difference between the input differential weight currents Iw+ and Iw− (i.e., IREAD=Iw+−Iw−). Thecurrent integration circuitry 640 can be implemented using any suitable circuit framework which is configured to add, subtract, or otherwise integrate the input differential weight currents Iw+ and Iw- to generate the read current IREAD, which provides a signed weight value of the RPU cell 600. -
FIG. 7 schematically illustrates an RPU cell 700 which comprises an in-cell complementary reading circuit that implements a differential weight read-out process to enable signed weight values to be directly read out from the RPU cell 700, according to another embodiment of the invention. The RPU cell 700 is similar to the RPU cell 600 discussed above with reference toFIG. 6 , except that the RPU cell 700 ofFIG. 7 implements acurrent mirror circuit 710 which is configured to perform an in-cell subtraction function without significantly increasing the footprint of the RPU cell 700. Thecurrent mirror circuit 710 provides an example embodiment for implementing thecurrent integration circuitry 640 ofFIG. 6 . - The
current mirror circuit 710 comprises a first NMOS transistor M1 and a second NMOS transistor M2. In one embodiment, the NMOS transistors M1 and M2 are the same size so that they have equal drain currents. In this regard, the complementary weight current Iw−, which is generated in thesecond read transistor 632 in response to the weight voltage Vw applied to the gate terminal of thesecond read transistor 632, serves as a reference current for thecurrent mirror circuit 710 and is mirrored in the NMOS transistor M1. In this configuration, thecurrent mirror circuit 710 serves as a current sink which sinks the mirrored current Iw− from the first current Iw+ which is generated in thefirst read transistor 630 in response to the weight voltage Vw applied to the gate terminal of thefirst read transistor 630. As such, the RPU cell 700 outputs a read current IREAD (from output port P2) based on a difference between the input differential weight currents Iw+ and Iw− (i.e., IREAD=Iw+−Iw−). - The in-cell complementary reading circuit framework for differential weight read-out in the RPU cell 700 of
FIG. 7 is advantageous for various reasons. For example, the in-cell complementary reading circuit framework effectively enhances the dynamic range by doubling the current difference values due to the differential implementation. Further, no additional cross bar array is required for the reading the complementary currents Iw+ and Iw−. Moreover, the average current flowing through column read lines can be significantly reduced, which enables are larger RPU array size. In addition, the in-cell implementation allows for compensation in transistor variations across the RPU array. -
FIG. 8 graphically illustrates computer simulation results showing a complementary differential weight read-out process that is implemented based on the RPU cell architecture shown inFIG. 7 . In particular,FIG. 8 comprises a graph 800 which illustrates simulated curves for weight current Iw+ and Iw− and a read current IREAD (Iw+−Iw−) (in micro Amps) as a function of weight voltage Vw. The simulation results ofFIG. 8 were obtained with the first andsecond read transistors FIG. 8 , the first weight current Iw+ increases with increasing weight voltage Vw, and the second weight current Iw− decreases with increasing weight voltage Vw.FIG. 8 illustrates an optimal operating range of the weight voltage Vw of 0.4V to 0.6V, wherein the wherein IREAD (Iw+−Iw−) has a linear slope. WhileFIG. 8 illustrates one example simulation for specific bias and operating voltages, additional computer simulations have shown that the RPU cell architecture ofFIG. 7 is robust in operation such that the first and second read transistors can be configured to operate in a saturation region with higher Vds values (e.g., 0.6) while still providing essentially a 1:1 correspondence between current output as a function of weight voltage in a wide operating range. - While the example embodiments of
FIGS. 4, 6 and 7 are based on an analog CMOS-based framework using, e.g., weight storage capacitors and read transistors for storing and reading RPU cell weight values, other embodiments of the invention can be implemented using resistor-type analog RPU cells. For example, resistor-type analog RPU cells can be implemented using memristors or resistive memory cells that are utilized in resistive random-access memory (RRAM). In particular, as is known in the art, a resistive memory cell comprises a metal-insulator-metal (MIM) structure in which the insulator layer (e.g., metal-oxide) exhibits a variable resistance through a resistive switching behavior. -
FIG. 9 schematically illustrates a resistor-type RPU cell 900 which comprises an in-cell differential read-out system to enable signed weight values to be directly read out from theRPU cell 900, according to another embodiment of the invention. TheRPU cell 900 comprisesresistance control circuitry 910, avariable resistance device 920, afirst resistance device 922, and acurrent mirror circuit 930.FIG. 9 illustrates an implementation of a resistor-type RPU cell framework in which the resistance value of thevariable resistance device 920 corresponds to the weight value of theRPU cell 900. Thevariable resistance device 920 may comprise a memristor device or resistive memory cell (e.g., MIM device) having a resistance value that can be modulated (or otherwise adjusted) by theresistance control circuitry 910 using known circuits and methods. - The
resistance control circuit 910 within the RPU cell would have input ports connected to “update” control lines of the row and column lines RL and CL to receive control signals from theperipheral circuitry 120 and 130 for updating the weight value of theRPU cell 900 by adjusting the resistance value of thevariable resistance device 920. - The
variable resistance device 920 generates a variable current I1 based on a control voltage (e.g., VDD) applied to a row control line (RL) attached to theRPU cell 900, wherein the value of current I1 corresponds to the weight value of theRPU cell 900. The fixedresistance device 930 generates a fixed current I2 based on the control voltage (e.g., VDD) applied to the row control line (RL) attached to theRPU cell 900, wherein the fixed current I2 serves as a fixed reference current. - Similar to the
current mirror 710 discussed above with reference toFIG. 7 , thecurrent mirror circuit 930 inFIG. 9 comprises a first NMOS transistor M1 and a second NMOS transistor M2. In one embodiment, the NMOS transistors M1 and M2 are the same size so that they have equal drain currents. In this regard, the fixed reference current I2, which flows through the fixedresistor device 922, serves as a reference current for thecurrent mirror circuit 930 and is mirrored in the NMOS transistor M1. In this configuration, thecurrent mirror circuit 930 serves as a current sink which sinks (subtracts) the mirrored current I2 from the variable current I1 which flows from thevariable resistance device 920. As such, theRPU cell 900 outputs a read current IREAD from the output port P2, wherein the read current IREAD represents a difference between the variable current I1 and the fixed reference current I2 (i.e., IREAD=I1−I2). In this regard, similar to the schemes discussed above, the sign of RPU cell weight is deemed positive when the read current IREAD is greater than zero, and the sign of the RPU cell weight is deemed negative when the read current IREAD is less than zero. - With the circuit configuration shown in
FIG. 9 , it is desirable that a voltage V1 at node N1 be equal to a voltage V2 at node N2 (i.e., V1=V2). However, the voltage V2 at node N2 may vary ±100 mV, which causes a slight variation in the reference current I2. To address this variation, additional circuitry can be implemented in theRPU cell 900 ofFIG. 9 to prevent variation in the fixed reference current I2. For example,FIG. 10 schematically illustrates a resistor-type RPU cell 1000 which comprises an in-cell differential read-out system to enable signed weight values to be directly read out from theRPU cell 1000, according to another embodiment of the invention. TheRPU cell 1000 ofFIG. 10 is similar to theRPU cell 900 ofFIG. 9 except that a PMOS transistor M2 and anoperational amplifier 1010 are incorporated with theRPU cell 1000 to prevent or significantly limit the voltage variation between nodes N1 and N2 and thereby prevent variation in the fixed reference current I2 that is input to thecurrent mirror circuit 930. - The PMOS transistor M3 is serially connected between the fixed
resistor device 922 and a reference input node of thecurrent mirror circuit 930. A gate terminal of the PMOS transistor M3 is connected to an output of theoperational amplifier 1010. Theoperational amplifier 1010 comprises a non-inverting input (+) connected to the node N1, and an inverting input (−) connected to the node N2, wherein the output of theoperational amplifier 1010 is proportional to V+−V− (i.e., the difference between the voltage applied to the non-inverting and inverting inputs). Theoperational amplifier 1010 serves to maintain the voltage V2 at node N2 equal to the voltage V1 at node N1, and maintain the current I2 which flows through the PMOS transistor M3 to remain at a fixed reference value (with no or insubstantial variation). -
FIG. 11 schematically illustrates a resistor-type RPU cell 1100 which comprises an in-cell differential read-out system to enable signed weight values to be directly read out from theRPU cell 1100, according to another embodiment of the invention. TheRPU cell 1100 ofFIG. 11 is similar to theRPU cell 1000 ofFIG. 10 , butFIG. 11 illustrates an example circuit configuration of theoperational amplifier 1010 ofFIG. 10 . In particular,FIG. 11 schematically illustrates an embodiment of anoperational amplifier 1110 comprising transistors M10, M11, M12, M13 and M14. The transistors M10 and M11 comprise PMOS transistors which form a differential input circuit, wherein a gate terminal of the PMOS transistor M10 comprises a non-inverting input of theoperational amplifier 1110, which is connected to the node N1, and wherein a gate terminal of the PMOS transistor M11 comprises an inverting input of theoperational amplifier 1110, which is connected to the node N2. The transistors M12 and M13 comprises NMOS transistors which form a current mirror load for theoperational amplifier 1110. The transistor M14 comprises a PMOS transistor which serves as a constant current biasing transistor for theoperational amplifier 1110. A bias voltage VBP from a global biasing circuit is applied to the gate terminal of the transistor M14. An output node NOUT of theoperational amplifier 1110 is connected to the gate terminal of the PMOS transistor M3. - Although exemplary embodiments have been described herein with reference to the accompanying figures, it is to be understood that the invention is not limited to those precise embodiments, and that various other changes and modifications may be made therein by one skilled in the art without departing from the scope of the appended claims.
Claims (20)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/352,230 US10453527B1 (en) | 2018-03-30 | 2019-03-13 | In-cell differential read-out circuitry for reading signed weight values in resistive processing unit architecture |
US16/353,111 US10468098B2 (en) | 2018-03-30 | 2019-03-14 | In-cell differential read-out circuitry for reading signed weight values in resistive processing unit architecture |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/941,558 US10340002B1 (en) | 2018-03-30 | 2018-03-30 | In-cell differential read-out circuitry for reading signed weight values in resistive processing unit architecture |
US16/352,230 US10453527B1 (en) | 2018-03-30 | 2019-03-13 | In-cell differential read-out circuitry for reading signed weight values in resistive processing unit architecture |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/941,558 Continuation US10340002B1 (en) | 2018-03-30 | 2018-03-30 | In-cell differential read-out circuitry for reading signed weight values in resistive processing unit architecture |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/353,111 Continuation US10468098B2 (en) | 2018-03-30 | 2019-03-14 | In-cell differential read-out circuitry for reading signed weight values in resistive processing unit architecture |
Publications (2)
Publication Number | Publication Date |
---|---|
US20190304538A1 true US20190304538A1 (en) | 2019-10-03 |
US10453527B1 US10453527B1 (en) | 2019-10-22 |
Family
ID=67069382
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/941,558 Active US10340002B1 (en) | 2018-03-30 | 2018-03-30 | In-cell differential read-out circuitry for reading signed weight values in resistive processing unit architecture |
US16/352,230 Active US10453527B1 (en) | 2018-03-30 | 2019-03-13 | In-cell differential read-out circuitry for reading signed weight values in resistive processing unit architecture |
US16/353,111 Expired - Fee Related US10468098B2 (en) | 2018-03-30 | 2019-03-14 | In-cell differential read-out circuitry for reading signed weight values in resistive processing unit architecture |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/941,558 Active US10340002B1 (en) | 2018-03-30 | 2018-03-30 | In-cell differential read-out circuitry for reading signed weight values in resistive processing unit architecture |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/353,111 Expired - Fee Related US10468098B2 (en) | 2018-03-30 | 2019-03-14 | In-cell differential read-out circuitry for reading signed weight values in resistive processing unit architecture |
Country Status (1)
Country | Link |
---|---|
US (3) | US10340002B1 (en) |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11232352B2 (en) * | 2018-07-17 | 2022-01-25 | Hewlett Packard Enterprise Development Lp | Memristor spiking architecture |
US10726895B1 (en) * | 2019-01-07 | 2020-07-28 | International Business Machines Corporation | Circuit methodology for differential weight reading in resistive processing unit devices |
DE102019132067A1 (en) | 2019-01-25 | 2020-07-30 | Taiwan Semiconductor Manufacturing Co., Ltd. | CURRENT LIMITER FOR STORAGE DEVICE |
US10991426B2 (en) * | 2019-01-25 | 2021-04-27 | Taiwan Semiconductor Manufacturing Company, Ltd. | Memory device current limiter |
US11544540B2 (en) * | 2019-05-10 | 2023-01-03 | Hewlett Packard Enterprise Development Lp | Systems and methods for neural network training and deployment for hardware accelerators |
US20220188628A1 (en) * | 2020-12-15 | 2022-06-16 | International Business Machines Corporation | Dynamic configuration of readout circuitry for different operations in analog resistive crossbar array |
US20230317124A1 (en) * | 2022-04-05 | 2023-10-05 | Taiwan Semiconductor Manufacturing Company, Ltd. | Memory system and operating method of memory system |
Family Cites Families (18)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
TWI244264B (en) | 2004-09-20 | 2005-11-21 | Faraday Tech Corp | Current comparator with hysteresis |
US7286429B1 (en) | 2006-04-24 | 2007-10-23 | Taiwan Semiconductor Manufacturing Company, Ltd. | High speed sensing amplifier for an MRAM cell |
US20090109295A1 (en) | 2007-10-30 | 2009-04-30 | Henry King | Method and apparatus for operating at least one camera from a position estimate of a container to estimate its container code |
US9070424B2 (en) | 2012-06-29 | 2015-06-30 | Samsung Electronics Co., Ltd. | Sense amplifier circuitry for resistive type memory |
US9076540B2 (en) * | 2012-08-23 | 2015-07-07 | Infineon Technologies Ag | Symmetrical differential sensing method and system for STT MRAM |
US9202543B2 (en) * | 2012-11-30 | 2015-12-01 | Intel Deutschland Gmbh | System and methods using a multiplexed reference for sense amplifiers |
US9087578B2 (en) | 2013-09-30 | 2015-07-21 | Micron Technology, Inc. | Configurable reference current generation for non volatile memory |
US9472245B2 (en) | 2013-12-31 | 2016-10-18 | Taiwan Semiconductor Manufacturing Company, Ltd. | Sense amplifier and related method |
US9224464B2 (en) | 2014-02-10 | 2015-12-29 | Taiwan Semiconductor Manufacturing Company, Ltd. | Memory circuit and related method |
US9617011B2 (en) | 2014-06-24 | 2017-04-11 | Sikorsky Aircraft Corporation | Probabilistic safe landing area determination |
US9224277B1 (en) | 2014-08-06 | 2015-12-29 | Belkin International, Inc. | Detector devices for presenting notifications and supporting context inferences |
TWI593834B (en) | 2015-08-18 | 2017-08-01 | 昆山納諾新材料科技有限公司 | Polishing system based on the non-newton fluid and polishing method thereof |
TWI668698B (en) | 2016-01-26 | 2019-08-11 | 聯華電子股份有限公司 | Memory current sensor |
KR101912881B1 (en) | 2016-03-18 | 2018-10-30 | 국민대학교 산학협력단 | Neuromorphic memristor crossbar circuit |
US9887351B1 (en) * | 2016-09-30 | 2018-02-06 | International Business Machines Corporation | Multivalent oxide cap for analog switching resistive memory |
US9852790B1 (en) * | 2016-10-26 | 2017-12-26 | International Business Machines Corporation | Circuit methodology for highly linear and symmetric resistive processing unit |
US10755170B2 (en) * | 2017-03-01 | 2020-08-25 | International Business Machines Corporation | Resistive processing unit with hysteretic updates for neural network training |
US11055610B2 (en) * | 2017-06-30 | 2021-07-06 | International Business Machines Corporation | Circuit for CMOS based resistive processing unit |
-
2018
- 2018-03-30 US US15/941,558 patent/US10340002B1/en active Active
-
2019
- 2019-03-13 US US16/352,230 patent/US10453527B1/en active Active
- 2019-03-14 US US16/353,111 patent/US10468098B2/en not_active Expired - Fee Related
Also Published As
Publication number | Publication date |
---|---|
US10453527B1 (en) | 2019-10-22 |
US10468098B2 (en) | 2019-11-05 |
US20190304539A1 (en) | 2019-10-03 |
US10340002B1 (en) | 2019-07-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10453527B1 (en) | In-cell differential read-out circuitry for reading signed weight values in resistive processing unit architecture | |
US11361215B2 (en) | Neural network circuits having non-volatile synapse arrays | |
US11842770B2 (en) | Circuit methodology for highly linear and symmetric resistive processing unit | |
JP6477924B2 (en) | Memristor neuromorphological circuit and method for training memristor neuromorphological circuit | |
US11157810B2 (en) | Resistive processing unit architecture with separate weight update and inference circuitry | |
US11875249B2 (en) | Counter based resistive processing unit for programmable and reconfigurable artificial-neural-networks | |
US9779355B1 (en) | Back propagation gates and storage capacitor for neural networks | |
US11915132B2 (en) | Synaptic weight transfer between conductance pairs with polarity inversion for reducing fixed device asymmetries | |
CN110945535A (en) | System and method for constructing synaptic weights for artificial neural networks from signed simulated conductance pairs with varying significance | |
TWI699711B (en) | Memory devices and manufacturing method thereof | |
US20200286553A1 (en) | In-memory computation device with inter-page and intra-page data circuits | |
US11610101B2 (en) | Formation failure resilient neuromorphic device | |
US11537863B2 (en) | Resistive processing unit cell having multiple weight update and read circuits for parallel processing of data using shared weight value | |
Tripathi et al. | Analog neuromorphic system based on multi input floating gate mos neuron model | |
JP2024500737A (en) | Dynamic configuration of readout circuits for different operations in analog resistive crossbar arrays | |
KR20230005309A (en) | Efficient Tile Mapping for Row-by-Row Convolutional Neural Network Mapping for Analog Artificial Intelligence Network Inference | |
US11556770B2 (en) | Auto weight scaling for RPUs | |
US20220101142A1 (en) | Neural network accelerators resilient to conductance drift | |
US11487990B2 (en) | Resistive crossbar arrays with reduced numbers of elements | |
CN116579392A (en) | Neural network training operation system and method based on conductance adjustable device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KIM, SEYOUNG;LEE, HYUNG-MIN;GOKMEN, TAYFUN;AND OTHERS;REEL/FRAME:048587/0845 Effective date: 20180329 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
FEPP | Fee payment procedure |
Free format text: SURCHARGE FOR LATE PAYMENT, LARGE ENTITY (ORIGINAL EVENT CODE: M1554); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |