WO2021259482A1 - Réalisation matérielle analogique de réseaux neuronaux - Google Patents

Réalisation matérielle analogique de réseaux neuronaux Download PDF

Info

Publication number
WO2021259482A1
WO2021259482A1 PCT/EP2020/067800 EP2020067800W WO2021259482A1 WO 2021259482 A1 WO2021259482 A1 WO 2021259482A1 EP 2020067800 W EP2020067800 W EP 2020067800W WO 2021259482 A1 WO2021259482 A1 WO 2021259482A1
Authority
WO
WIPO (PCT)
Prior art keywords
analog
network
neural network
equivalent
layer
Prior art date
Application number
PCT/EP2020/067800
Other languages
English (en)
Inventor
Aleksandrs TIMOFEJEVS
Boris Maslov
Original Assignee
PolyN Technology Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by PolyN Technology Limited filed Critical PolyN Technology Limited
Priority to PCT/EP2020/067800 priority Critical patent/WO2021259482A1/fr
Priority to US17/189,109 priority patent/US20210406661A1/en
Priority to US17/196,960 priority patent/US20210406662A1/en
Priority to US17/198,198 priority patent/US20210406663A1/en
Priority to US17/199,373 priority patent/US20210406664A1/en
Priority to US17/199,422 priority patent/US20210406666A1/en
Priority to US17/199,407 priority patent/US20210406665A1/en
Priority to US17/200,707 priority patent/US20210406667A1/en
Priority to US17/200,723 priority patent/US20220004861A1/en
Priority to TW110122949A priority patent/TWI773398B/zh
Priority to TW111125566A priority patent/TWI796257B/zh
Publication of WO2021259482A1 publication Critical patent/WO2021259482A1/fr
Priority to US17/733,932 priority patent/US11885271B2/en
Priority to US17/744,565 priority patent/US20220280072A1/en
Priority to US17/902,757 priority patent/US20230081715A1/en
Priority to US18/093,315 priority patent/US20230147781A1/en
Priority to US18/196,412 priority patent/US20230367998A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/10—Interfaces, programming languages or software development kits, e.g. for simulating neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G06N3/065—Analogue means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent

Definitions

  • the disclosed implementations relate generally to neural networks, and more specifically to systems and methods for hardware realization of neural networks.
  • memristor-based architectures that use cross-bar technology remain impractical for manufacturing recurrent and feed-forward neural networks.
  • memristor-based cross-bars have a number of disadvantages, including high latency and leakage of currents during operation, that make them impractical.
  • there are reliability issues in manufacturing memristor-based cross-bars especially when neural networks have both negative and positive weights.
  • memristor-based cross-bars cannot be used for simultaneous propagation of different signals, which in turn complicates summation of signals, when neurons are represented by operational amplifiers.
  • memristor-based analog integrated circuits have a number of limitations, such as a small number of resistive states, first cycle problem when forming memristors, complexity with channel formation when training the memristors, unpredictable dependency on dimensions of the memristors, slow operations of memristors, and drift of state of resistance.
  • a trained neural network is used for specific inferencing tasks, such as classification. Once a neural network is trained, a hardware equivalent is manufactured. When the neural network is retrained, the hardware manufacturing process is repeated, driving up costs.
  • edge environments such as smart-home applications, do not require re-programmability as such. For example, 85% of all applications of neural networks do not require any retraining during operation, so on-chip learning is not that useful. Furthermore, edge applications include noisy environments, that can cause reprogrammable hardware to become unreliable.
  • Analog circuits that model trained neural networks and manufactured according to the techniques described herein, can provide improved performance per watt advantages, can be useful in implementing hardware solutions in edge environments, and can tackle a variety of applications, such as drone navigation and autonomous cars.
  • the cost advantages provided by the proposed manufacturing methods and/or analog network architectures are even more pronounced with larger neural networks.
  • analog hardware implementations of neural networks provide improved parallelism and neuro morphism.
  • neuromorphic analog components are not sensitive to noise and temperature changes, when compared to digital counterparts.
  • Chips manufactured according to the techniques described herein provide order of magnitude improvements over conventional systems in size, power, and performance, and are ideal for edge environments, including for retraining purposes.
  • Such analog neuromorphic chips can be used to implement edge computing applications or in Internet-of-Things (IoT) environments.
  • initial processing e.g., formation of descriptors for image recognition
  • IoT Internet-of-Things
  • Various edge applications can benefit from use of such analog hardware.
  • the techniques described herein can be used to include direct connection to CMOS sensor without digital interface.
  • video processing applications include road sign recognition for automobiles, camera-based true depth and/or simultaneous localization and mapping for robots, room access control without server connection, and always-on solutions for security and healthcare.
  • Such chips can be used for data processing from radars and lidars, and for low-level data fusion.
  • Such techniques can be used to implement battery management features for large battery packs, sound/voice processing without connection to data centers, voice recognition on mobile devices, wake up speech instructions for IoT sensors, translators that translate one language to another, large sensors arrays of IoT with low signal intensity, and/or configurable process control with hundreds of sensors.
  • Neuromorphic analog chips can be mass produced after standard software- based neural network simulations/training, according to some implementations.
  • a client’s neural network can be easily ported, regardless of the structure of the neural network, with customized chip design and production.
  • a library of ready to make on-chip solutions are provided, according to some implementations. Such solutions require only training, one lithographic mask change, following which chips can be mass produced. For example, during chip production, only part of the lithography masks need to be changed.
  • the techniques described herein can be used to design and/or manufacture an analog neuromorphic integrated circuit that is mathematically equivalent to a trained neural network (either feed-forward or recurrent neural networks).
  • the process begins with a trained neural network that is first converted into a transformed network comprised of standard elements. Operation of the transformed network are simulated using software with known models representing the standard elements. The software simulation is used to determine the individual resistance values for each of the resistors in the transformed network. Lithography masks are laid out based on the arrangement of the standard elements in the transformed network. Each of the standard elements are laid out in the masks using an existing library of circuits corresponding to the standard elements to simplify and speed up the process.
  • the resistors are laid out in one or more masks separate from the masks including the other elements (e.g., operational amplifiers) in the transformed network.
  • the other elements e.g., operational amplifiers
  • the lithography masks are then sent to a fab for manufacturing the analog neuromorphic integrated circuit.
  • a method for hardware realization of neural networks, according to some implementations.
  • the method incudes obtaining a neural network topology and weights of a trained neural network.
  • the method also includes transforming the neural network topology to an equivalent analog network of analog components.
  • the method also includes computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent analog network.
  • the method also includes generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components.
  • generating the schematic model includes generating a resistance matrix for the weight matrix.
  • Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value.
  • the method further includes obtaining new weights for the trained neural network, computing a new weight matrix for the equivalent analog network based on the new weights, and generating a new resistance matrix for the new weight matrix.
  • the neural network topology includes one or more layers of neurons, each layer of neurons computing respective outputs based on a respective mathematical function, and transforming the neural network topology to the equivalent analog network of analog components includes: for each layer of the one or more layers of neurons: (i) identifying one or more function blocks, based on the respective mathematical function, for the respective layer.
  • Each function block has a respective schematic implementation with block outputs that conform to outputs of a respective mathematical function; and (ii) generating a respective multilayer network of analog neurons based on arranging the one or more function blocks.
  • Each analog neuron implements a respective function of the one or more function blocks, and each analog neuron of a first layer of the multilayer network is connected to one or more analog neurons of a second layer of the multilayer network.
  • ReLU Rectified Linear Unit (ReLU) activation function or a similar activation function, V t represents an i-th input, W j represents a weight corresponding to the i-th input, and bias represents a bias value, and ⁇ is a summation operator;
  • a signal multiplier block with a block output V out coeff. Vi. V j .
  • V t represents an i-th input and V j represents a j-th input, and coeff is a predetermined coefficient;
  • a sigmoid activation block with a block output V out
  • identifying the one or more function blocks includes selecting the one or more function blocks based on a type of the respective layer.
  • the neural network topology includes one or more layers of neurons, each layer of neurons computing respective outputs based on a respective mathematical function, and transforming the neural network topology to the equivalent analog network of analog components includes: (i) decomposing a first layer of the neural network topology to a plurality of sub-layers, including decomposing a mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions.
  • Each sub-layer implements an intermediate mathematical function; and (ii) for each sub-layer of the first layer of the neural network topology: (a) selecting one or more sub-function blocks, based on a respective intermediate mathematical function, for the respective sub-layer; and (b) generating a respective multilayer analog sub-network of analog neurons based on arranging the one or more sub-function blocks.
  • Each analog neuron implements a respective function of the one or more sub-function blocks, and each analog neuron of a first layer of the multilayer analog sub-network is connected to one or more analog neurons of a second layer of the multilayer analog sub-network.
  • the mathematical function corresponding to the first layer includes one or more weights
  • decomposing the mathematical function includes adjusting the one or more weights such that combining the one or more intermediate functions results in the mathematical function.
  • the method further includes: (i) generating equivalent digital network of digital components for one or more output layers of the neural network topology; and (ii) connecting output of one or more layers of the equivalent analog network to the equivalent digital network of digital components.
  • the analog components include a plurality of operational amplifiers and a plurality of resistors, each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons.
  • selecting component values of the analog components includes performing a gradient descent method to identify possible resistance values for the plurality of resistors.
  • the neural network topology includes one or more
  • GRU or LSTM neurons and transforming the neural network topology includes generating one or more signal delay blocks for each recurrent connection of the one or more GRU or LSTM neurons.
  • the one or more signal delay blocks are activated at a frequency that matches a predetermined input signal frequency for the neural network topology.
  • the neural network topology includes one or more layers of neurons that perform unlimited activation functions
  • transforming the neural network topology includes applying one or more transformations selected from the group consisting of: (i) replacing the unlimited activation functions with limited activation; and (ii) adjusting connections or weights of the equivalent analog network such that, for predetermined one or more inputs, difference in output between the trained neural network and the equivalent analog network is minimized.
  • the method further includes generating one or more lithographic masks for fabricating a circuit implementing the equivalent analog network of analog components based on the resistance matrix.
  • the method further includes: (i) obtaining new weights for the trained neural network; (ii) computing a new weight matrix for the equivalent analog network based on the new weights; (iii) generating a new resistance matrix for the new weight matrix; and (iv) generating a new lithographic mask for fabricating the circuit implementing the equivalent analog network of analog components based on the new resistance matrix.
  • the trained neural network is trained using software simulations to generate the weights.
  • a method for hardware realization of neural networks includes obtaining a neural network topology and weights of a trained neural network.
  • the method also includes calculating one or more connection constraints based on analog integrated circuit (IC) design constraints.
  • the method also includes transforming the neural network topology to an equivalent sparsely connected network of analog components satisfying the one or more connection constraints.
  • the method also includes computing a weight matrix for the equivalent sparsely connected network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent sparsely connected network.
  • transforming the neural network topology to the equivalent sparsely connected network of analog components includes deriving a possible input connection degree N t and output connection degree JV 0 , according to the one or more connection constraints.
  • the neural network topology includes at least one densely connected layer with K inputs and L outputs and a weight matrix U.
  • transforming the at least one densely connected layer includes constructing the equivalent sparsely connected network with K inputs, L outputs, and [ log N. K] + ⁇ log No ] — 1 layers, such that input connection degree does not exceed JV j , and output connection degree does not exceed N 0.
  • the neural network topology includes at least one densely connected layer with K inputs and L outputs and a weight matrix U. .
  • transforming the at least one densely connected layer includes constructing the equivalent sparsely connected network with K inputs, L outputs, and M 3 max( ⁇ log N. L ], ⁇ log No K ) layers.
  • Each layer m is represented by a corresponding weight matrix U m , where absent connections are represented with zeros, such that input connection degree does not exceed JV j , and output connection degree does not exceed N 0.
  • the neural network topology includes a single sparsely connected layer with K inputs and L outputs, a maximum input connection degree of P j , a maximum output connection degree of P 0 , and a weight matrix of U , where absent connections are represented with zeros.
  • transforming the single sparsely connected layer includes constructing the equivalent sparsely connected network with K inputs, L outputs, M 3 max( log N. P i ⁇ , ⁇ log No P 0 ]) layers, each layer m represented by a corresponding weight matrix U m , where absent connections are represented with zeros, such that input connection degree does not exceed JV j , and output connection degree does not exceed JV 0 .
  • the neural network topology includes a convolutional layer with K inputs and L outputs.
  • transforming the neural network topology to the equivalent sparsely connected network of analog components includes decomposing the convolutional layer into a single sparsely connected layer with K inputs, L outputs, a maximum input connection degree of P and a maximum output connection degree of P 0 .
  • the neural network topology includes a recurrent neural layer.
  • transforming the neural network topology to the equivalent sparsely connected network of analog components includes transforming the recurrent neural layer into one or more densely or sparsely connected layers with signal delay connections.
  • the neural network topology includes a recurrent neural layer.
  • transforming the neural network topology to the equivalent sparsely connected network of analog components includes decomposing the recurrent neural layer into several layers, where at least one of the layers is equivalent to a densely or sparsely connected layer with K inputs and L output and a weight matrix U , where absent connections are represented with zeros.
  • the neural network topology includes K inputs, a weight vector U E R K , and a single layer perceptron with a calculation neuron with an activation function F.
  • the equivalent sparsely connected network includes respective one or more analog neurons in each layer of the m layers, each analog neuron of first m- 1 layers implements identity transform, and an analog neuron of last layer implements the activation function F of the calculation neuron of the single layer perceptron.
  • computing the weight matrix for the equivalent sparsely connected network includes calculating a weight vector W for connections of the equivalent sparsely connected network by solving a system of equations based on the weight vector U.
  • the system of equations includes K equations with
  • the neural network topology includes K inputs, a single layer perceptron with L calculation neurons, and a weight matrix V that includes a row of weights for each calculation neuron of the L calculation neurons.
  • Each single layer perceptron network includes a respective calculation neuron of the L calculation neurons; (iv) for each single layer perceptron network of the L single layer perceptron networks: (a) constructing a respective equivalent pyramid-like sub-network for the respective single layer perceptron network with the K inputs, the m layers and the connection degree N.
  • the equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m- ⁇ layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron; and (b) constructing the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating an input of each equivalent pyramid-like sub-network for the L single layer perceptron networks to form an input vector with L*K inputs.
  • the system of equations includes K equations with S variables, and
  • the neural network topology includes K inputs, a multi-layer perceptron with S layers, each layer i of the S layers includes a corresponding set of calculation neurons L and corresponding weight matrices V that includes a row of weights for each calculation neuron of the L calculation neurons.
  • the equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m- 1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron network; and (iv) constructing the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating input of each equivalent pyramid-like sub-network for the Q single layer perceptron networks to form an input vector with Q*K J inputs.
  • the neural network topology includes a
  • CNN Convolutional Neural Network
  • each layer i of the S layers includes a corresponding set of calculation neurons L, and corresponding weight matrices V that includes a row of weights for each calculation neuron of the T, calculation neurons.
  • the equivalent pyramid-like sub- network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m- 1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron network; and (iv) constructing the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating input of each equivalent pyramid-like sub-network for the Q single layer perceptron networks to form an input vector with Q*K l J inputs.
  • the system of equations includes
  • transforming the neural network topology to the equivalent sparsely connected network of analog components includes performing a trapezium transformation that includes: (i) deriving a possible input connection degree N j > 1 and a possible output connection degree N 0 > 1, according to the one or more connection constraints; (ii) in accordance with a determination that K ⁇ L ⁇ L ⁇ N j + K ⁇ N 0 , constructing a three-layered analog network that includes a layer LA P with K analog neurons performing identity activation function, a layer LA / , with
  • M [max ( ana '° & neurons performing identity activation function, and a layer
  • computing the weight matrix for the equivalent sparsely connected network includes generating a sparse weight matrices W 0 and W h by solving a matrix equation W 0 .
  • the sparse weight matrix W 0 E R KxM re p resen s connections between the layers LA P and IA / and the sparse weight matrix W h E R MxL represents connections between the layers LA / , and LA 0 ,.
  • performing the trapezium transformation further includes: in accordance with a determination that K ⁇ L 3 L ⁇ N j + K ⁇ N 0 : (i) splitting the layer L p to obtain a sub-layer L pi with K ' neurons and a sub-layer P? with (K - K’) neurons such that K’ L 3 L ⁇ N j + K' N 0 ; (ii) for the sub-layer L pi with K' neurons, performing the constructing, and generating steps; and (iii) for the sub-layer L P2 with K - K’ neurons, recursively performing the splitting, constructing, and generating steps.
  • the neural network topology includes a multilayer perceptron network.
  • the method further includes, for each pair of consecutive layers of the multilayer perceptron network, iteratively performing the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network.
  • the neural network topology includes a recurrent neural network (RNN) that includes (i) a calculation of linear combination for two fully connected layers, (ii) element-wise addition, and (iii) a non-linear function calculation.
  • RNN recurrent neural network
  • the method further includes performing the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network, for (i) the two fully connected layers, and (ii) the non-linear function calculation.
  • the neural network topology includes a long short term memory (LSTM) network or a gated recurrent unit (GRU) network that includes (i) a calculation of linear combination for a plurality of fully connected layers, (ii) element-wise addition, (iii) a Hadamard product, and (iv) a plurality of non-linear function calculations.
  • the method further includes performing the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network, for (i) the plurality of fully connected layers, and (ii) the plurality of non-linear function calculations.
  • the neural network topology includes a convolutional neural network (CNN) that includes (i) a plurality of partially connected layers and (ii) one or more fully-connected layers.
  • CNN convolutional neural network
  • the method further includes: (i) transforming the plurality of partially connected layers to equivalent fully-connected layers by inserting missing connections with zero weights; and (ii) for each pair of consecutive layers of the equivalent fully-connected layers and the one or more fully-connected layers, iteratively performing the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network.
  • the neural network topology includes K inputs, L output neurons, and a weight matrix U E R LxK , where R is the set of real numbers, each output neuron performs an activation function F.
  • transforming the neural network topology to the equivalent sparsely connected network of analog components includes: for each layer j of the S layers of the multilayer perceptron: (i) constructing a respective pyramid-trapezium network PTNNX j by performing the approximation transformation to a respective single layer perceptron consisting of L j- inputs, L j output neurons, and a weight matrix and (ii) constructing the equivalent sparsely connected network by stacking each pyramid trapezium network.
  • a method for hardware realization of neural networks, according to some implementations.
  • the method includes obtaining a neural network topology and weights of a trained neural network.
  • the method also includes transforming the neural network topology to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors.
  • Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons.
  • the method also includes computing a weight matrix for the equivalent analog network based on the weights of the trained neural network.
  • Each element of the weight matrix represents a respective connection.
  • the method also includes generating a resistance matrix for the weight matrix.
  • Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value.
  • the predetermined range of possible resistance values includes resistances according to nominal series E24 in the range 100 KW to 1 MW.
  • R + and R ⁇ are chosen independently for each layer of the equivalent analog network.
  • R + and R ⁇ are chosen independently for each analog neuron of the equivalent analog network.
  • a first one or more weights of the weight matrix and a first one or more inputs represent one or more connections to a first operational amplifier of the equivalent analog network.
  • the method further includes, prior to generating the resistance matrix: (i) modifying the first one or more weights by a first value; and (ii) configuring the first operational amplifier to multiply, by the first value, a linear combination of the first one or more weights and the first one or more inputs, before performing an activation function.
  • the method further includes: (i) obtaining a predetermined range of weights; and (ii) updating the weight matrix according to the predetermined range of weights such that the equivalent analog network produces similar output as the trained neural network for same input.
  • the trained neural network is trained so that each layer of the neural network topology has quantized weights.
  • the method further includes retraining the trained neural network to reduce sensitivity to errors in the weights or the resistance values that cause the equivalent analog network to produce different output compared to the trained neural network.
  • the method further includes retraining the trained neural network so as to minimize weight in any layer that are more than mean absolute weight for that layer by larger than a predetermined threshold.
  • an integrated circuit is provided, according to some implementations.
  • the integrated circuit includes an analog network of analog components fabricated by a method that includes: (i) obtaining a neural network topology and weights of a trained neural network; (ii) transforming the neural network topology to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors.
  • Each operational amplifier represents a respective analog neuron, and each resistor represents a respective connection between a respective first analog neuron and a respective second analog neuron; (iii) computing a weight matrix for the equivalent analog network based on the weights of the trained neural network.
  • Each element of the weight matrix represents a respective connection; (iv) generating a resistance matrix for the weight matrix.
  • Each element of the resistance matrix corresponds to a respective weight of the weight matrix;
  • the integrated circuit further includes one or more digital to analog converters configured to generate analog input for the equivalent analog network of analog components based on one or more digital.
  • the integrated circuit further includes an analog signal sampling module configured to process 1 -dimensional or 2-dimensional analog inputs with a sampling frequency based on number of inferences of the integrated circuit.
  • the integrated circuit further includes a voltage converter module to scale down or scale up analog signals to match operational range of the plurality of operational amplifiers.
  • the integrated circuit further includes a tact signal processing module configured to process one or more frames obtained from a CCD camera.
  • the trained neural network is a long short-term memory (LSTM) network.
  • the integrated circuit further includes one or more clock modules to synchronize signal tacts and to allow time series processing.
  • the integrated circuit further includes one or more analog to digital converters configured to generate digital signal based on output of the equivalent analog network of analog components.
  • the integrated circuit further includes one or more signal processing modules configured to process 1 -dimensional or 2-dimensional analog signals obtained from edge applications.
  • the trained neural network is trained, using training datasets containing signals of arrays of gas sensors on different gas mixture, for selective sensing of different gases in a gas mixture containing predetermined amounts of gases to be detected.
  • the neural network topology is a 1 -Dimensional Deep Convolutional Neural network (1D-DCNN) designed for detecting 3 binary gas components based on measurements by 16 gas sensors, and includes 16 sensor- wise 1-D convolutional blocks, 3 shared or common 1-D convolutional blocks and 3 dense layers.
  • the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) delay blocks to produce delay by any number of time steps, (iii) a signal limit of 5, (iv) 15 layers, (v) approximately 100,000 analog neurons, and (vi) approximately 4,900,000 connections.
  • the trained neural network is trained, using training datasets containing thermal aging time series data for different MOSFETs, for predicting remaining useful life (RUL) of a MOSFET device.
  • the neural network topology includes 4 LSTM layers with 64 neurons in each layer, followed by two dense layers with 64 neurons and 1 neuron, respectively.
  • the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 18 layers, (iv) between 3,000 and 3,200 analog neurons, and (v) between 123,000 and 124,000 connections.
  • the trained neural network is trained, using training datasets containing time series data including discharge and temperature data during continuous usage of different commercially available Li-Ion batteries, for monitoring state of health (SOH) and state of charge (SOC) of Lithium Ion batteries to use in battery management systems (BMS).
  • the neural network topology includes an input layer, 2 LSTM layers with 64 neurons in each layer, followed by an output dense layer with 2 neurons for generating SOC and SOH values.
  • the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 9 layers, (iv) between 1,200 and 1,300 analog neurons, and (v) between 51,000 and 52,000 connections.
  • the trained neural network is trained, using training datasets containing time series data including discharge and temperature data during continuous usage of different commercially available Li-Ion batteries, for monitoring state of health (SOH) of Lithium Ion batteries to use in battery management systems (BMS).
  • the neural network topology includes an input layer with 18 neurons, a simple recurrent layer with 100 neurons, and a dense layer with 1 neuron.
  • the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 4 layers, (iv) between 200 and 300 analog neurons, and (v) between 2,200 and 2,400 connections.
  • the trained neural network is trained, using training datasets containing speech commands, for identifying voice commands.
  • the neural network topology is a Depthwise Separable Convolutional Neural Network (DS-CNN) layer with 1 neuron.
  • the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 13 layers, (iv) approximately 72,000 analog neurons, and (v) approximately 2.6 million connections.
  • the trained neural network is trained, using training datasets containing photoplethysmography (PPG) data, accelerometer data, temperature data, and electrodermal response signal data for different individuals performing various physical activities for a predetermined period of times and reference heart rate data obtained from ECG sensor, for determining pulse rate during physical exercises based on PPG sensor data and 3-axis accelerometer data.
  • the neural network topology includes two ConvlD layers each with 16 filters and a kernel of 20, performing time series convolution, two LSTM layers each with 16 neurons, and two dense layers with 16 neurons and 1 neuron, respectively.
  • the equivalent analog network includes: (i) delay blocks to produce any number of time steps, (ii) a maximum of 100 input and output connections per analog neuron, (iii) a signal limit of 5, (iv) 16 layers, (v) between 700 and 800 analog neurons, and (vi) between 12,000 and 12,500 connections.
  • the trained neural network is trained to classify different objects based on pulsed Doppler radar signal.
  • the neural network topology includes multi-scale LSTM neural network.
  • the trained neural network is trained to perform human activity type recognition, based on inertial sensor data.
  • the neural network topology includes three channel-wise convolutional networks each with a convolutional layer of 12 filters and a kernel dimension of 64, and each followed by a max pooling layer, and two common dense layers of 1024 neurons and N neurons, respectively, where N is a number of classes.
  • the equivalent analog network includes: (i) delay blocks to produce any number of time steps, (ii) a maximum of 100 input and output connections per analog neuron, (iii) an output layer of 10 analog neurons, (iv) signal limit of 5, (v) 10 layers, (vi) between 1,200 and 1,300 analog neurons, and (vi) between 20,000 and 21,000 connections.
  • the trained neural network is further trained to detect abnormal patterns of human activity based on accelerometer data that is merged with heart rate data using a convolution operation.
  • a computer system has one or more processors, memory, and a display.
  • the one or more programs include instructions for performing any of the methods described herein.
  • a non-transitory computer readable storage medium stores one or more programs configured for execution by a computer system having one or more processors, memory, and a display.
  • the one or more programs include instructions for performing any of the methods described herein.
  • Figure 1 A is a block diagram of a system for hardware realization of trained neural networks using analog components, according to some implementations.
  • Figure IB is a block diagram of an alternative representation of the system of Figure 1A for hardware realization of trained neural networks using analog components, according to some implementations.
  • Figure 1C is a block diagram of another representation of the system of Figure 1A for hardware realization of trained neural networks using analog components, according to some implementations.
  • Figure 2A is a system diagram of a computing device in accordance with some implementations.
  • Figure 2B shows optional modules of the computing device, according to some implementations.
  • Figure 3A shows an example process for generating schematic models of analog networks corresponding to trained neural networks, according to some implementations.
  • Figure 3B shows an example manual prototyping process used for generating a target chip model, according to some implementations.
  • Figures 4A, 4B, and 4C show examples of neural networks that are transformed to mathematically equivalent analog networks, according to some implementations.
  • Figure 5 shows an example of a math model for a neuron, according to some implementations.
  • Figures 6A-6C illustrate an example process for analog hardware realization of a neural network for computing an XOR of input values, according to some implementations.
  • Figure 7 shows an example perceptron, according to some implementations.
  • Figure 8 shows an example Pyramid-Neural Network, according to some implementations.
  • Figure 9 shows an example Pyramid Single Neural Network, according to some implementations.
  • Figure 10 shows an example of a transformed neural network, according to some implementations.
  • Figures 11A-11C show an application of a T-transformation algorithm for a single layer neural network, according to some implementations.
  • FIG 12 shows an example Recurrent Neural Network (RNN), according to some implementations.
  • RNN Recurrent Neural Network
  • Figure 13 A is a block diagram of a LSTM neuron, according to some implementations.
  • Figure 13B shows delay blocks, according to some implementations.
  • Figure 13C is a neuron schema for a LSTM neuron, according to some implementations.
  • Figure 14A is a block diagram of a GRU neuron, according to some implementations.
  • Figure 14B is a neuron schema for a GRU neuron, according to some implementations.
  • Figures 15A and 15B are neuron schema of variants of a single ConvlD filter, according to some implementations.
  • Figure 16 shows an example architecture of a transformed neural network, according to some implementations.
  • Figures 17A - 17C provide example charts illustrating dependency between output error and classification error or weight error, according to some implementations.
  • Figure 18 provides an example scheme of a neuron model used for resistors quantization, according to some implementations.
  • Figure 19A shows a schematic diagram of an operational amplifier made on CMOS, according to some implementations.
  • Figure 19B shows a table of description for the example circuit shown in Figure 19A, according to some implementations.
  • Figures 20A-20E show a schematic diagram of a LSTM block, according to some implementations.
  • Figure 20F shows a table of description for the example circuit shown in Figure 20A-20D, according to some implementations.
  • Figures 21A-21I show a schematic diagram of a multiplier block, according to some implementations.
  • Figure 21 J shows a table of description for the schematic shown in Figures 21 A-21I, according to some implementations.
  • Figure 22 A shows a schematic diagram of a sigmoid neuron, according to some implementations.
  • Figure 22B shows a table of description for the schematic diagram shown in Figure 22A, according to some implementations.
  • Figure 23A shows a schematic diagram of a hyperbolic tangent function block, according to some implementations.
  • Figure 23B shows a table of description for the schematic diagram shown in Figure 23 A, according to some implementations.
  • Figures 24A-24C show a schematic diagram of a single neuron CMOS operational amplifier, according to some implementations.
  • Figure 24D shows a table of description for the schematic diagram shown in Figure 24A-24C, according to some implementations.
  • Figures 25A-25D show a schematic diagram of a variant of a single neuron CMOS operational amplifiers according to some implementations.
  • Figure 25E shows a table of description for the schematic diagram shown in Figure 25A-25D, according to some implementations.
  • Figures 26A-26K show example weight distribution histograms, according to some implementations.
  • Figures 27A-27J show a flowchart of a method for hardware realization of neural networks, according to some implementations.
  • Figures 28A-28S show a flowchart of a method for hardware realization of neural networks according to hardware design constraints, according to some implementations.
  • Figures 29A-29F show a flowchart of a method for hardware realization of neural networks according to hardware design constraints, according to some implementations.
  • Figure 30 is intentionally left blank.
  • Figures 31 A-3 IQ show a flowchart of a method for fabricating an integrated circuit that includes an analog network of analog components, according to some implementations.
  • Figure 32 is intentionally left blank.
  • Figure 33 is intentionally left blank.
  • Figure 34 shows a table describing the MobileNet vl architecture, according to some implementations.
  • FIG. 1A is a block diagram of a system 100 for hardware realization of trained neural networks using analog components, according to some implementations.
  • the system includes transforming (126) trained neural networks 102 to analog neural networks 104.
  • analog integrated circuit constraints 184 constrain (146) the transformation (126) to generate the analog neural networks 104.
  • the system derives (calculates or generates) weights 106 for the analog neural networks 104 by a process that is sometimes called weight quantization (128).
  • the analog neural network includes a plurality of analog neuron, each analog neuron represented by an analog component, such as an operational amplifier, and each analog neuron connected to another analog neuron via a connection.
  • the connections are represented using resistors that reduce the current flow between two analog neurons.
  • the system transforms (148) the weights 106 to resistance values 112 for the connections.
  • the system subsequently generates (130) one or more schematic models 108 for implementing the analog neural networks 104 based on the weights 106.
  • the system generates (132) lithographic masks 110 for the connections and/or generates (136) lithographic masks 120 for the analog neurons.
  • the system fabricates (134 and/or 138) analog integrated circuits 118 that implement the analog neural networks 104.
  • the system regenerates (or recalculates) (144) the resistance values 112 (and/or the weights 106), the schematic model 108, and/or the lithographic masks for connections 110.
  • the system reuses the lithographic masks 120 for the analog neurons 120. In other words, in some implementations, only the weights 106 (or the resistance values 112 corresponding to the changed weights), and/or the lithographic masks for the connections 110 are regenerated.
  • the process for (or the path to) fabricating analog integrated circuits for the retrained neural networks is substantially simplified, and the time to market for re-spinning hardware for neural networks is reduced, when compared to conventional techniques for hardware realization of neural networks.
  • Figure IB is a block diagram of an alternative representation of the system 100 for hardware realization of trained neural networks using analog components, according to some implementations.
  • the system includes training (156) neural networks in software, determining weights of connections, generating (158) electronic circuit equivalent to the neural network, calculating (160) resistor values corresponding to weights of each connection, and subsequently generating (162) lithography mask with resistor values.
  • FIG. 1C is a block diagram of another representation of the system 100 for hardware realization of trained neural networks using analog components, according to some implementations.
  • the system is distributed as a software development kit (SDK) 180, according to some implementations.
  • SDK software development kit
  • a user develops and trains (164) a neural network and inputs the trained neural net 166 to the SDK 180.
  • the SDK estimates (168) complexity of the trained neural net 166.
  • the SDK 180 transforms (170) the trained neural net 166 into a sparse network of analog components (e.g., a pyramid- or a trapezia-shaped network).
  • the SDK 180 also generates a circuit model 172 of the analog network.
  • the SDK estimates (176) a deviation in an output generated by the circuit model 172 relative to the trained neural network for a same input, using software simulations. If the estimated error exceeds a threshold error (e.g., a value set by the user), the SDK 180 prompts the user to reconfigure, redevelop, and/or retrain the neural network. In some implementations, although not shown, the SDK automatically reconfigures the trained neural net 166 so as to reduce the estimated error. This process is iterated multiple times until the error is reduced below the threshold error. In Figure 1C, the dashed line from the block 176 (“Estimation of error raised in circuitry”) to the block 164 (“Development and training of neural network”) indicates a feedback loop.
  • a threshold error e.g., a value set by the user
  • the SDK 180 prompts the user to reconfigure, redevelop, and/or retrain the neural network.
  • the SDK automatically reconfigures the trained neural net 166 so as to reduce the estimated
  • this process includes recalculating the weights, since pruning includes retraining of the whole network.
  • FIG. 2A is a system diagram of a computing device 200 in accordance with some implementations.
  • the term “computing device” includes both personal devices 102 and servers.
  • a computing device 200 typically includes one or more processing units/cores (CPUs) 202 for executing modules, programs, and/or instructions stored in the memory 214 and thereby performing processing operations; one or more network or other communications interfaces 204; memory 214; and one or more communication buses 212 for interconnecting these components.
  • the communication buses 212 may include circuitry that interconnects and controls communications between system components.
  • a computing device 200 may include a user interface 206 comprising a display device 208 and one or more input devices or mechanisms 210.
  • the input device/mechanism 210 includes a keyboard; in some implementations, the input device/mechanism includes a “soft” keyboard, which is displayed as needed on the display device 208, enabling a user to “press keys” that appear on the display 208.
  • the display 208 and input device / mechanism 210 comprise a touch screen display (also called a touch sensitive display).
  • the memory 214 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices.
  • the memory 214 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. In some implementations, the memory 214 includes one or more storage devices remotely located from the CPU(s) 202. The memory 214, or alternatively the non-volatile memory device(s) within the memory 214, comprises a computer readable storage medium. In some implementations, the memory 214, or the computer readable storage medium of the memory 214, stores the following programs, modules, and data structures, or a subset thereof:
  • an operating system 216 which includes procedures for handling various basic system services and for performing hardware dependent tasks;
  • a communications module 218, which is used for connecting the computing device 200 to other computers and devices via the one or more communication network interfaces 204 (wired or wireless) and one or more communication networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so on;
  • trained neural networks 220 that includes weights 222 and neural network topologies 224. Examples of input neural networks are described below in reference to Figures 4A-4C, Figure 12, Figure 13A, and 14A, according to some implementations;
  • a neural network transformation module 226 that includes transformed analog neural networks 228, mathematical formulations 230, the basic function blocks 232, analog models 234 (sometimes called neuron models), and/or analog integrated circuit (IC) design constraints 236.
  • Example operations of the neural network transformation module 226 are described below in reference to at least Figures 5, 6A-6C, 7, 8, 9, 10, and 11 A-l 1C, and the flowcharts shown in Figures 27A-27J, and Figures 28A-28S; and/or
  • a weight matrix computation (sometimes called a weight quantization) module 238 that includes weights 272 of transformed networks, and optionally includes resistance calculation module 240, resistance values 242.
  • Example operations of the weight matrix computation module 238 and/or weight quantization are described in reference to at least Figures 17A-17C, Figure 18, and Figures 29 A- 29F, according to some implementations.
  • Some implementations include one or more optional modules 244 as shown in Figure 2B.
  • Some implementations include a lithographic mask generation module 248 that further includes lithographic masks 250 for resistances (corresponding to connections), and/or lithographic masks for analog components (e.g., operational amplifiers, multipliers, delay blocks, etc.) other than the resistances (or connections).
  • lithographic masks are generated based on chip design layout following chip design using Cadence, Synopsys, or Mentor Graphics software packages.
  • Some implementations use a design kit from a silicon wafer manufacturing plant (sometimes called a fab). Lithographic masks are intended to be used in that particular fab that provides the design kit (e.g., TSMC 65 nm design kit). The lithographic mask files that are generated are used to fabricate the chip at the fab.
  • the Cadence, Mentor Graphics, or Synopsys software packages-based chip design is generated semi-automaticahy from the SPICE or Fast SPICE (Mentor Graphics) software packages.
  • a user with chip design skill drives the conversion from the SPICE or Fast SPICE circuit into Cadence, Mentor Graphics or Synopsis chip design.
  • IC fabrication module 258 that further includes Analog-to-Digital Conversion (ADC), Digital-to-Analog Conversion (DAC), or similar other interfaces 260, and/or fabricated ICs or models 262.
  • ADC Analog-to-Digital Conversion
  • DAC Digital-to-Analog Conversion
  • Example integrated circuits and/or related modules are described below in reference to Figures 31A- 31Q, according to some implementations.
  • Each of the above identified executable modules, applications, or sets of procedures may be stored in one or more of the previously mentioned memory devices, and corresponds to a set of instructions for performing a function described above.
  • the above identified modules or programs i.e., sets of instructions
  • the memory 214 stores a subset of the modules and data structures identified above.
  • the memory 214 stores additional modules or data structures not described above.
  • Figure 2A shows a computing device 200
  • Figure 2A is intended more as a functional description of the various features that may be present rather than as a structural schematic of the implementations described herein.
  • items shown separately could be combined and some items could be separated.
  • FIG. 3A shows an example process 300 for generating schematic models of analog networks corresponding to trained neural networks, according to some implementations.
  • a trained neural network 302 e.g., MobileNet
  • the target neural network (sometimes called a T- network) 304 is exported (324) to SPICE (as a SPICE model 306) using a single neuron model (SNM), which is exported (326) from SPICE to CADENCE and full on-chip designs using a CADENCE model 308.
  • the CADENCE model 308 is cross-validated (328) against the initial neural network for one or more validation inputs.
  • a math neuron is a mathematical function which receives one or more weighted inputs and produces a scalar output.
  • a math neuron can have memory (e.g., long short-term memory (LSTM), recurrent neuron).
  • a SNM is a schematic model with analog components (e.g., operational amplifiers, resistors Ri, ..., R n , and other components) representing a specific type of math neuron (for example, trivial neuron) in schematic form.
  • SNM output voltage is represented by a corresponding formula that depends on K input voltages and SNM component values V out g(yi n , —. V , R ... Rn).
  • SNM formula is equivalent to math neuron formula, with a desired weights set.
  • the weights set is fully determined by resistors used in a SNM.
  • a target (analog) neural network 304 (sometimes called a T-network) is a set of math neurons which have defined SNM representation, and weighted connections between them, forming a neural network.
  • T-network follows several restrictions, such as an inbound limit (a maximum limit of inbound connections for any neuron within the T- network), an outbound limit (a maximum limit of outbound connections for any neuron within the T-network), and a signal range (e.g., all signals should be inside pre-defmed signal range).
  • T-transformation (322) is a process of converting some desired neural network, such as MobileNet, to a corresponding T-network.
  • a SPICE model 306 is a SPICE Neural Network model of a T-network 304, where each math neuron is substituted with corresponding one or more SNMs.
  • Figure 3B shows an example manual prototyping process used for generating a target chip model 320 based on a SNM model on Cadence 314, according to some implementations.
  • Cadence alternate tools from Mentor Graphic design or Synopsys (e.g., Synopsys design kit) may be used in place of Cadence tools, according to some implementations.
  • the process includes selecting SNM limitations, including inbound and outbound limits and signal limitation, selecting analog components (e.g., resistors, including specific resistor array technology) for connections between neurons, and developing a Cadence SNM model 314.
  • a prototype SNM model 316 (e.g., a PCB prototype) is developed (330) based on the SNM model on Cadence 314.
  • the prototype SNM model 316 is compared with a SPICE model for equivalence.
  • a neural network is selected for an on-chip prototype, when the neural network satisfies equivalence requirements. Because the neural network is small in size, the T-transformation can be hand- verified for equivalence.
  • an on-chip SNM model 318 is generated (332) based on the SNM model prototype 316.
  • the on-chip SNM model is optimized as possible, according to some implementations.
  • an on-chip density for the SNM model is calculated prior to generating (334) a target chip model 320 based on the on-chip SNM model 318, after finalizing the SNM.
  • a practitioner may iterate selecting neural network task or application and specific neural network (e.g., a neural network having in the order of 0.1 to 1.1 million neurons), performing T-transformation, building a Cadence neural network model, designing interfaces and/or the target chip model.
  • specific neural network e.g., a neural network having in the order of 0.1 to 1.1 million neurons
  • Figures 4A, 4B, and 4C show examples of trained neural networks (e.g., the neural networks 220) that are input to the system 100 and transformed to mathematically equivalent analog networks, according to some implementations.
  • Figure 4A shows an example neural network (sometimes called an artificial neural network) that are composed of artificial neurons that receive input, combine the input using an activation function, and produce one or more outputs.
  • the input includes data, such as images, sensor data, and documents.
  • each neural network performs a specific task, such as object recognition.
  • the networks include connections between the neurons, each connection providing the output of a neuron as an input to another neuron. After training, each connection is assigned a corresponding weight.
  • the neurons are typically organized into multiple layers, with each layer of neurons connected only to the immediately preceding and following layer of neurons.
  • An input layer of neurons 402 receives external input (e.g., the input Xi, X2, ..., X n ).
  • the input layer 402 is followed by one or more hidden layers of neurons (e.g., the layers 404 and 406), that is followed by an output layer 408 that produces outputs 410.
  • Various types of connection patterns connect neurons of consecutive layers, such as a fully-connected pattern that connects every neuron in one layer to all the neurons of the next layer, or a pooling pattern that connect output of a group of neurons in one layer to a single neuron in the next layer.
  • the neural network shown in Figure 4B includes one or more connections from neurons in one layer to either other neurons in the same layer or neurons in a preceding layer.
  • the example shown in Figure 4B is an example of a recurrent neural network, and includes two input neurons 412 (that accepts an input XI) and 414 (that accepts an input X2) in an input layer followed by two hidden layers.
  • the first hidden layer includes neurons 416 and 418 that is fully connected with neurons in the input layer, and the neurons 420, 422, and 424 in the second hidden layer.
  • the output of the neuron 420 in the second hidden layer is connected to the neuron 416 in the first hidden layer, providing a feedback loop.
  • the hidden layer including the neurons 420, 422, and 424 are input to a neuron 426 in the output layer that produces an output y.
  • Figure 4C shows an example of a convolutional neural network (CNN), according to some implementations.
  • CNN convolutional neural network
  • the example shown in Figure 4C includes different types of neural network layers, that includes a first stage of layers for feature learning, and a second stage of layers for classification tasks, such as object recognition.
  • the feature learning stage includes a convolution and Rectified Linear Unit (ReLU) layer 430, followed by a pooling layer 432, that is followed by another convolution and ReLU layer 434, which is in turn followed by another pooling layer 436.
  • ReLU Rectified Linear Unit
  • the first layer 430 extracts features from an input 428 (e.g., an input image or portions thereof), and performs a convolution operation on its input, and one or more non-linear operations (e.g., ReLU, tanh, or sigmoid).
  • a pooling layer such as the layer 432, reduces the number of parameters when the inputs are large.
  • the output of the pooling layer 436 is flattened by the layer 438 and input to a fully connected neural network with one or more layers (e.g., the layers 440 and 442).
  • the output of the fully-connected neural network is input to a softmax layer 444 to classify the output of the layer 442 of the fully-connected network to produce one of many different output 446 (e.g., object class or type of the input image 428).
  • Some implementations store the layout or the organization of the input neural networks including number of neurons in each layer, total number of neurons, operations or activation functions of each neuron, and/or connections between the neurons, in the memory 214, as the neural network topology 224.
  • Figure 5 shows an example of a math model 500 for a neuron, according to some implementations.
  • the math model includes incoming signals 502 input multiplied by synaptic weights 504 and summed by a unit summation 506.
  • the result of the unit summation 506 is input to a nonlinear conversion unit 508 to produce an output signal 510, according to some implementations.
  • Figures 6A-6C illustrate an example process for analog hardware realization of a neural network for computing an XOR (classification of XOR results) of input values, according to some implementations.
  • Figure 6A shows a table 600 of possible input values Xi and X2 along x- and y-axis, respectively.
  • the expected result values are indicated by hollow circle (represents a value of 1) and a filled or dark circle (represents a value of 0) - this is a typical XOR problem with 2 input signals and 2 classes. Only if either, not both, of the values Xi and X2 are 1, the expected result is 1, and 0, otherwise.
  • Training set consists of 4 possible input signal combinations (binary values for the Xi and X2 inputs).
  • Figure 6B shows a ReLU-based neural network 602 to solve the XOR classification of Figure 6A, according to some implementations.
  • the neurons do not use any bias values, and use ReLU activation.
  • Inputs 604 and 606 (that correspond to Xi and X2, respectively) are input to a first ReLU neuron 608-2.
  • the inputs 604 and 606 are also input to a second ReLU neuron 608-4.
  • the results of the two ReLU neurons 608-2 and 608-4 are input to a third neuron 608- 6 that performs linear summation of the input values, to produce an output value 510 (the Out value).
  • the neural network 602 has the weights -1 and 1 (for the input values Xi and X2, respectively) for the ReLU neuron 608-2, the weights 1 and -1 (for the input values Xi and X2, respectively) for the ReLU neuron 608-4, and the weights 1 and 1 (for the output of the RelLu neurons 608-2 and 608-4, respectively).
  • the weights of trained neural networks are stored in memory 214, as the weights 222.
  • Figure 6C shows an example equivalent analog network for the network 602, according to some implementations.
  • the analog equivalent inputs 614 and 616 of the Xi and X2 inputs 604 and 606 are input to analog neurons N1 618 and N2 620 of a first layer.
  • the neurons N1 and N2 are densely connected with neurons N3 and N4 of a second layer.
  • the neurons of a second layer i.e. neuron N3 622 and neuron N4 624) are connected with an output neuron N5 626 that produces the output Out (equivalent to the output 610 of the network 602).
  • weights are stored in memory 214, as part of the weights 222.
  • data format is ‘Neuron [1 st link weight, 2 nd link weight, bias]’.
  • the input trained neural networks are transformed to pyramid- or trapezium-shaped analog networks.
  • Some of the advantages of pyramid or trapezium over cross bars include lower latency, simultaneous analog signal propagation, possibility for manufacture using standard integrated circuit (IC) design elements, including resistors and operational amplifiers, high parallelism of computation, high accuracy (e.g., accuracy increases with the number of layers, relative to conventional methods), tolerance towards error(s) in each weight and/or at each connection (e.g., pyramids balance the errors), low RC (low Resistance Capacitance delay related to propagation of signal through network), and/or ability to manipulate biases and functions of each neuron in each layer of the transformed network.
  • IC integrated circuit
  • pyramids are excellent computation block by itself, since it is a multi-level perceptron, which can model any neural network with one output. Networks with several outputs are implemented using different pyramids or trapezia geometry, according to some implementations.
  • a pyramid can be thought of as a multi-layer perceptron with one output and several layers (e.g., N layers), where each neuron has n inputs and 1 output.
  • a trapezium is a multilayer perceptron, where each neuron has n inputs and m outputs.
  • Each trapezium is a pyramid-like network, where each neuron has n inputs and m outputs, where n and m are limited by IC analog chip design limitations, according to some implementations.
  • pyramids and trapezia can be used as universal building blocks for transforming any neural networks.
  • An advantage of pyramid- or trapezia-based neural networks is the possibility to realize any neural network using standard IC analog elements (e.g., operational amplifiers, resistors, signal delay lines in case of recurrent neurons) using standard lithography techniques. It is also possible to restrict the weights of transformed networks to some interval. In other words, lossless transformation is performed with weights limited to some predefined range, according to some implementations.
  • Another advantage of using pyramids or trapezia is the high degree of parallelism in signal processing or the simultaneous propagation of analog signals that increases the speed of calculations, providing lower latency.
  • analog neuromorphic trapezia-like chips possess a number of properties, not typical for analog devices. For example, signal to noise ratio is not increasing with the number of cascades in analog chip, the external noise is suppressed, and influence of temperature is greatly reduced. Such properties make trapezia-like analog neuromorphic chips analogous to digital circuits. For example, individual neurons, based on operational amplifier, level the signal and are operated with the frequencies of 20,000-100,000 Hz, and are not influenced by noise or signals with frequency higher than the operational range, according to some implementations. Trapezia-like analog neuromorphic chip also perform filtration of output signal due to peculiarities in how operational amplifiers function. Such trapezia-like analog neuromorphic chip suppresses the synphase noise.
  • Trapezia-like analogous neuromorphic circuit is tolerant towards the errors and noise in input signals and is tolerant towards deviation of resistor values, corresponding to weight values in neural network. Trapezia-like analog neuromorphic networks are also tolerant towards any kind of systemic error, like error in resistor value settings, if such error is same for all resistors, due to the very nature of analog neuromorphic trapezia-like circuits, based on operational amplifiers.
  • the example transformations described herein are performed by the neural network transformation module 226 that transform trained neural networks 220, based on the mathematical formulations 230, the basic function blocks 232, the analog component models 234, and/or the analog design constraints 236, to obtain the transformed neural networks 228.
  • FIG. 7 shows an example perceptron 700, according to some implementations.
  • There is an output layer with 4 neurons 704-2, 704-8, in an output layer, that correspond to L 4 outputs.
  • the weights of the connections are represented by a weight matrix WP (element WPi , j corresponds to the weight of the connection between the i-th neuron in the input layer and the j-th neuron in the output layer).
  • each neuron performs an activation function F.
  • Figure 8 shows an example Pyramid-Neural Network (P-NN) 800, a type of Target-Neural Network (T-NN, or TNN), that is equivalent to the perceptron shown in Figure 7, according to some implementations.
  • P-NN Pyramid-Neural Network
  • T-NN Target-Neural Network
  • the set of neurons 804, including neurons 802-20, ..., 802-34, is a copy of the neurons 802-2, ..., 802-18, and the input is replicated.
  • the input to the neuron 802-2 is also input to the neuron 802-20
  • the input 20 the neuron 802-4 is also input to the neuron 802-22, and so on.
  • Each group of Ni neurons from the input layer LTI are fully connected to two neurons from the LTH1 layer.
  • Each neuron in the layer LTO is connected to distinct neurons from different groups in the layer LTH1.
  • the network shown in Figure 8 includes 40 connections. Some implementations perform weight matrix calculation for the P-NN in Figure 8, as follows. Weights for the hidden layer LTH1 (WTH1) are calculated from the weight matrix WP, and weights corresponding to the output layer LTO (WTO) form a sparse matrix with elements equal to 1.
  • FIG. 9 shows a Pyramid Single Neural Network (PSNN) 900 corresponding to an output neuron of Figure 8, according to some implementations.
  • the PSNN includes a layer (LPSI) of input neurons 902-02, ..., 902-16 (corresponding to the 8 input neurons in the network 700 of Figure 7).
  • An output layer LPSO consists of 1 neuron 906 with an activation function F, that is connected to both the neurons 904-02 and 904-04 of the hidden layer.
  • weight vector WPSH1 that is equal to the first row of WP, for the LPSH1 layer.
  • LPSOlayer some implementations compute a weight vector WPSO with 2 elements, each element equal to 1.
  • the process is repeated for the first, second, third, and fourth output neurons.
  • a P-NN such as the network shown in Figure 8, is a union of the PSNNs (for the 4 output neurons).
  • Input layer for every PSNN is a separate copy of P’s input layer.
  • Example Transformations with Target Neurons with N Inputs and 1 Output are performed by the neural network transformation module 226 that transform trained neural networks 220, based on the mathematical formulations 230, the basic function blocks 232, the analog component models 234, and/or analog design constraints 236, to obtain the transformed neural networks 228.
  • a single layer perceptron SLP(K,1) includes K inputs and one output neuron with activation function F.
  • U E R K is a vector of weights for SLP(K,1).
  • the following algorithm NeuronlTNNl constructs a T-neural network from T- neurons with N inputs and 1 output (referred to as TN(N,1)).
  • K 1
  • m 1 groups such that every group consists of no more than N inputs.
  • b Construct the first hidden layer LTHi of the T-NN from rr ⁇ neurons, each neuron performing an identity activation function.
  • c Connect input neurons from every group to corresponding neuron from the next layer. So every neuron from the LTHi has no more than N input connections.
  • d Set the weights for the new connections according the following equation:
  • Layer 1 (e.g., layer 1002): ui u 2 ... UN 0 0 ... 0 . 0
  • W 1 o 0 ... o UN+1 UN+2 ... U2N 0 ... 0
  • Output for the first layer is calculated as an output vector according to the following formula:
  • Every subsequent layer outputs a vector with components equal to linear combination of some sub-vector of x. [00152] Finally, the T-NN’s output is equal to:
  • Output of the PTNN is equal to the SLP(K, L)’s output for the same input vector because output of every pair SLPi(K, 1) and TNNi are equal.
  • MLP multilayer perceptron
  • Output of the MTNN is equal to the MLP(K, S, Li,... Ls)’s output for the same input vector because output of every pair SLPi(Li-i, Li) and PTNNi are equal.
  • the example transformations described herein are performed by the neural network transformation module 226 that transform trained neural networks 220, based on the mathematical formulations 230, the basic function blocks 232, the analog component models 234, and/or the analog design constraints 236, to obtain the transformed neural networks 228.
  • a single layer perceptron SLP(K, L) includes K inputs and L output neurons, each neuron performing an activation function F.
  • U E R LxK is a weight matrix for SLP(K,L).
  • the following algorithm constructs a T-neural network from neurons TN(Ni, No), according to some implementations.
  • PTNN Construct a PTNN from SLP(K,L) by using the algorithm Layer2TNNl (see description above).
  • PTNN has an input layer consisting of L groups of K inputs.
  • Each subset contains no more than No groups of input vector copies.
  • output of the PTNNX is calculated by means of the same formulas as for PTNN (described above), so the outputs are equal.
  • Figures 11 A-l 1C show an application 1100 of the above algorithm for a single layer neural network (NN) with 2 output neurons and TN(Ni, 2), according to some implementations.
  • Figure 11A shows an example source or input NN, according to some implementations.
  • K inputs are input to two neurons 1 and 2 belonging to a layer 1104.
  • Figure 1 IB shows a PTNN constructed after the first step of the algorithm, according to some implementations.
  • the PTNN consists of two parts implementing subnets corresponding to the output neuron 1 and neuron 2 of the NN shown in Figure 11 A.
  • the input 1102 is replicated and input to two sets of input neurons 1106-2 and 1106-4.
  • Each set of input neurons is connected to a subsequent layer of neurons with two sets of neurons 1108-2 and 1108-4, each set of neurons including mi neurons.
  • the input layer is followed by identity transform blocks 1110-2 and 1110-4, each block containing one or more layers with identity weight matrix.
  • the output of the identity transform block 1110-2 is connected to the output neuron 1112 (corresponding to the output neuron 1 in Figure 11 A), and the output of the identity transform block 1110-4 is connected to the output neuron 1114 (corresponding to the output neuron 1 in Figure 11 A).
  • Figure 11C shows application of the final steps of the algorithm, including replacing two copies of the input vector (1106-2 and 1106-4) with one vector 1116 (step 3), and rebuilding connections in the first layer 1118 by making two output links from every input neuron: one link connects to subnet related to output 1 and another link connects to subnet for the output 2.
  • MLP multilayer perceptron
  • K K, S, Li,... Ls
  • t/ j e R L i xL i-i is a weight matrix for i-th layer.
  • the following example algorithm constructs a T-neural network from neurons TN(Ni, No), according to some implementations.
  • output of the MTNNX is equal to the MLP(K, S, Li, ... Ls)’ s output for the same input vector, because output of every pair SLPi(Li- 1, Li) and PTNNXi are equal.
  • a Recurrent Neural Network contains backward connection allowing saving information.
  • Figure 12 shows an example RNN 1200, according to some implementations.
  • the example shows a block 1204 performing an activation function A, that accepts an input X t 1206 and performs an activation function A, and outputs a value h t 1202.
  • the backward arrow from the block 1204 to itself indicates a backward connection, according to some implementations.
  • An equivalent network is shown on the right up to the point in time when the activation block receives the input X t 1206.
  • the network accepts input X t 1208 and performs the activation function A 1204, and outputs a value h 0 1210; at time 1, the network accepts input Xi 1212 and the output of the network at time 0, and performs the activation function A 1204, and outputs a value hi 1214; at time 2, the network accepts input X2 1216 and the output of the network at time 1, and performs the activation function A 1204, and outputs a value hi 1218.
  • This process continues until time t, at which time the network accepts the input X t 1206 and the output of the network at time t-1, and performs the activation function A 1204, and outputs the value h t 1202, according to some implementations.
  • x t is a current input vector
  • h t -i is the RNN’ s output for the previous input vector x t -i.
  • This expression consists of the several operations: calculation of linear combination for two fully connected layers element-wise addition, and non-linear function calculation (f).
  • the first and third operations can be implemented by trapezium-based network (one fully connected layer is implemented by pyramid-based network, a special case of trapezium networks).
  • the second operation is a common operation that can be implemented in networks of any structure.
  • the RNN’s layer without recurrent connections is transformed by means of Layer2TNNX algorithm described above. After transformation is completed, recurrent links are added between related neurons. Some implementations use delay blocks described below in reference to Figure 13B.
  • a Long Short-Term Memory (LSTM) neural network is a special case of a RNN.
  • a LSTM network’s operations are represented by the following equations:
  • W ⁇ , W ⁇ , W D , and W 0 are trainable weight matrices
  • bf, b b D , and b 0 are trainable biases
  • x t is a current input vector
  • h t -i is an internal state of the LSTM calculated for the previous input vector x t -i
  • o is output for the current input vector.
  • the subscript t denotes a time instance /
  • the subscript t A denotes a time instance / - 1.
  • Figure 13A is a block diagram of a LSTM neuron 1300, according to some implementations.
  • a sigmoid (s) block 1318 processes the inputs h t-1 1330 and x t 1332, and produces the output f t 1336.
  • a second sigmoid (s) block 1320 processes the inputs h t-1 1330 and x t 1332, and produces the output i t 1338.
  • a hyperbolic tangent (tanh) block 1322 processes the inputs h t- 1330 and x t 1332, and produces the output D t 1340.
  • a third sigmoid (s) block 1328 processes the inputs h t- 1330 and x t 1332, and produces the output O t 1342.
  • a multiplier block 1304 processes f t 1336 and the output of a summing block 1306 (from a prior time instance) C t-1 1302 to produce an output that is in turn summed by the summing block 1306 along with the output of a second multiplier block 1314 that multiplies the outputs i t 1338 and D t 1340 to produce the output C t 1310.
  • the output C t 1310 is input to another tanh block 1312 that produces an output that is multiplied a third multiplier block 1316 with the output O t 1342 to produce the output h t 1334.
  • the layer in an LSTM layer without recurrent connections is transformed by using the Layer 2TNNX algorithm described above, according to some implementations. After transformation is completed, recurrent links are added between related neurons, according to some implementations.
  • Figure 13B shows delay blocks, according to some implementations.
  • some of the expressions in the equations for the LSTM operations depend on saving, restoring, and/or recalling an output from a previous time instance.
  • the multiplier block 1304 processes the output of the summing block 1306 (from a prior time instance) C t-1 1302.
  • Figure 13B shows two examples of delay blocks, according to some implementations.
  • the example 1350 includes a delay block 1354 on the left accepts input x t 1352 at time t, and outputs the input after a delay of dt indicated by the output x t-dt 1356.
  • the example 1360 on the right shows cascaded (or multiple) delay blocks 1364 and 1366 outputs the input x t 1362 after 2 units of time delays, indicated by the output x t -2 dt 1368, according to some implementations.
  • Figure 13C is a neuron schema for a LSTM neuron, according to some implementations.
  • the schema includes weighted summator nodes (sometimes called adder blocks) 1372, 1374, 1376, 1378, and 1396, multiplier blocks 1384, 1392, and 1394, and delay blocks 1380 and 1382.
  • the input x t 1332 is connected to the adder blocks 1372, 1374, 1376, and 1378.
  • the output h t-1 1330 for a prior input x t- is also input to the adder blocks 1372, 1374, 1376, and 1378.
  • the adder block 1372 produces an output that is input to a sigmoid block 1394-2 that produces the output f t 1336.
  • the adder block 1374 produces an output that is input to the sigmoid block 1386 that produces the output i t 1338.
  • the adder block 1376 produces an output that is input to a hyperbolic tangent block 1388 that produces the output D t 1340.
  • the adder block 1378 produces an output that is input to the sigmoid block 1390 that produces the output O t 1342.
  • the multiplier block 1392 uses the outputs i t 1338, f t 1336, and output of the adder block 1396 from a prior time instance C t-1 1302 to produce a first output.
  • the multiplier block 1394 uses the outputs i t 1338 and D t 1340 to produce a second output.
  • the adder block 1396 sums the first output and second output to produce the output C t 1310.
  • the output C t 1310 is input to a hyperbolic tangent block 1398 that produces an output that is input, along with the output of the sigmoid block 1390, O t 1342, to the multiplier block 1384 to produce the output h t 1334.
  • the delay block 1382 is used to recall (e.g., save and restore) the output of the adder block 1396 from a prior time instance.
  • the delay block 1380 is used to recall or save and restore the output of the multiplier block 1384 for a prior input x t- (e.g., from a prior time instance). Examples of delay blocks are described above in reference to Figure 13B, according to some implementations.
  • a Gated Recurrent Unit (GRU) neural network is a special case of RNN.
  • a RNN’s operations are represented by the following expressions:
  • x t is a current input vector
  • h t - is an output calculated for the previous input vector x t - .
  • Figure 14A is a block diagram of a GRU neuron, according to some implementations.
  • a sigmoid (s) block 1418 processes the inputs h t-1 1402 and x t 1422, and produces the output r t 1426.
  • a second sigmoid (s) block 1420 processes the inputs h t-1 1402 and x t 1422, and produces the output z t 1428.
  • a multiplier block 1412 multiplies the output r t 1426 and the input h t-1 1402 to produce and output that is input (along with the input x t 1422) to a hyperbolic tangent (tanh) block 1424 to produce the output j t 1430.
  • a second multiplier block 1414 multiplies the output j t 1430 and the output z t 1428 to produce a first output.
  • the block 1410 computes 1 - the output z t 1428 to produce an output that is input to a third multiplier block 1404 that multiplies the output and the input h t-t 1402 to produce a product that is input to an adder block 1406 along with the first output (from the multiplier block 1414) to produce the output h t 1408.
  • the input h t-1 1402 is the output of the GRU neuron from a prior time interval output t — 1.
  • Figure 14B is a neuron schema for a GRU neuron 1440, according to some implementations.
  • the schema includes weighted summator nodes (sometimes called adder blocks) 1404, 1406, 1410, 1406, and 1434, multiplier blocks 1404, 1412, and 1414, and delay block 1432.
  • the input x t 1422 is connected to the adder blocks 1404, 1410, and 1406.
  • the output h t-1 1402 for a prior input x t- is also input to the adder blocks 1404 and 1406, and the multiplier blocks 1404 and 1412.
  • the adder block 1404 produces an output that is input to a sigmoid block 1418 that produces the output Z t 1428.
  • the adder block 1406 produces an output that is input to the sigmoid block 1420 that produces the output r t 1426 that is input to the multiplier block 1412.
  • the output of the multiplier block 1412 is input to the adder block 1410 whose output is input to a hyperbolic tangent block 1424 that produces an output 1430.
  • the output 1430 as well as the output of the sigmoid block 1418 are input to the multiplier block 1414.
  • the output of the sigmoid block 1418 is input to the multiplier block 1404 that multiplies that output with the input from the delay block 1432 to produce a first output.
  • the mukltipler block produces a second output.
  • the adder block 1434 sums the first output and the second output to produce the output h t 1408.
  • the delay block 1432 is used to recall (e.g., save and restore) the output of the adder block 1434 from a prior time instance. Examples of delay blocks are described above in reference to Figure 13B, according to some implementations.
  • Operation types used in GRU are the same as the operation types for LSTM networks (described above), so GRU is transformed to trapezium-based networks following the principles described above for LSTM (e.g., using the Layer2TNNX algorithm), according to some implementations.
  • CNN Convolutional Neural Networks
  • CNN include several basic operations, such as convolution (a set of linear combinations of image’s (or internal map’s) fragments with a kernel), activation function, and pooling (e.g., max, mean, etc.). Every calculation neuron in a CNN follows the general processing scheme of a neuron in an MLP: linear combination of some inputs with subsequent calculation of activation function. So a CNN is transformed using the MLP2TNNX algorithm described above for multilayer perceptrons, according to some implementations.
  • ConvlD is a convolution performed over time coordinate.
  • Figures 15A and 15B are neuron schema of variants of a single ConvlD filter, according to some implementations.
  • a weighted summator node 1502 (sometimes called adder block, marked ‘+’) has 5 inputs, so it corresponds to lDconvolution with a kernel of 5.
  • the inputs are x t 1504 from time t, x t-t 1514 from time t — 1 (obtained by inputting the input to a delay block 1506), x t-2 1516 from time t — 2 (obtained by inputting the output of the delay block 1506 to another delay block 1508), x t-3 1518 from time t — 3 (obtained by inputting the output of the delay block 1508 to another delay block 1510), and x t-4 1520 from time t — 4 (obtained by inputting the output of the delay block 1510 to another delay block 1512.
  • Some implementations substitute several small delay blocks for one large delay block, as shown in Figure 15B.
  • the example uses a delay _3 block 1524 that produces x t-3 1518 from time t — 3, and another delay block 1526 that produces the x t-5 1522 from time t — 5.
  • the delay _3 1524 block is an example of multiple delay blocks, according to some implementations. This operation does not decrease total number of blocks, but it may decrease total number of consequent operations performed over the input signal and reduce accumulation of errors, according to some implementations.
  • convolutional layers are represented by trapezia like neurons and fully connected layer is represented by cross-bar of resistors. Some implementations use cross-bars, and calculate resistance matrix for the cross-bars.
  • the example transformations described herein are performed by the neural network transformation module 226 that transform trained neural networks 220, based on the mathematical formulations 230, the basic function blocks 232, the analog component models 234, and/or the analog design constraints 236, to obtain the transformed neural networks 228.
  • a single layer perceptron SLP(K, L) includes K inputs and L output neurons, each output neuron performing an activation function F.
  • U E R LxK is a weight matrix for SLP(K, L).
  • the algorithm applies Layer2TNNl algorithm (described above) at the first stage in order to decrease a number of neurons and connections, and subsequently applies Layer2TNNX to process the input of the decreased size.
  • the outputs of the resulted neural net are calculated using shared weights of the layers constructed by the Layer2TNNl algorithm. The number of these layers is determined by the value p, a parameter of the algorithm. If p is equal to 0 then Layer2TNNX algorithm is applied only and the transformation is equivalent. If p > 0, then p layers have shared weights and the transformation is approximate.
  • the net PNN has N p neurons in the output layer.
  • C is any constant not equal to zero
  • ki (i — 1 )jV j + 1, — —C, for all weights j of this neuron except k,.
  • All other weights of the PNN net are set to 1 represents a weight for the first layer (as denoted by the superscript (1)) for the connection between the neuron i and the neuron k t in the first layer.
  • Figure 16 shows an example architecture 1600 of the resulting neural net, according to some implementations.
  • the example includes a PNN 1602 connected to a TNN 1606.
  • the PNN 1602 includes a layer for K inputs and produce N p outputs, that is connected as input 1612 to the TNN 1606.
  • the TNN 1606 generates L outputs 1610, according to some implementations.
  • a multilayer perceptron includes K inputs, S layers and Li calculation neurons in i-th layer, represented as MLP( K, S, Li,...Ls).
  • MLP multilayer perceptron
  • t/ j e R L i xL i is a weight matrix for the i-th layer.
  • the following example algorithm constructs a T-neural network from neurons TN(Ni, No), according to some implementations.
  • Example Generation of Optimal Resistor Set the example computations described herein are performed by the weight matrix computation or weight quantization module 238 (e.g., using the resistance calculation module 240) that compute the weights 272 for connections of the transformed neural networks, and/or corresponding resistance values 242 for the weights 272.
  • This section describes an example of generating an optimal resistor set for a trained neural network, according to some implementations.
  • An example method is provided for converting connection weights to resistor nominals for implementing the neural network (sometimes called a NN model) on a microchip with possibly less resistor nominals and possibly higher allowed resistor variance.
  • test set ‘Test’ includes around 10,000 values of input vector (x and y coordinates) with both coordinates varying in the range [0; 1], with a step of 0.01.
  • Classification error is defined by the following equation:
  • Some implementations set the desired classification error as no more than 1%.
  • Figure 17A shows an example chart 1700 illustrating dependency between output error and classification error on the M network, according to some implementations.
  • the x-axis corresponds to classification margin 1704
  • the y-axis corresponds to total error 1702 (see description above).
  • the graph shows total error (difference between output of model M and real data) for different classification margins of output signal.
  • the optimal classification margin 1706 is 0.610.
  • Possible weight error is determined by analyzing dependency between weight/bias relative error over the whole network and output error.
  • the charts 1710 and 1720 shown in Figures 17B and 17C, respectively, are obtained by averaging 20 randomly modified networks over the ‘Test’ set, according to some implementations.
  • x-axis represents the absolute weight error 1712
  • y-axis represents the absolute output error 1714.
  • Maximum weight modulus (maximum of absolute value of weights among all wieights) for the neural network is 1.94.
  • a resistor set together with a (R+, R- ⁇ pair chosen from this set has a value function over the required weight range [-wlim; wlim] with some degree of resistor error r err.
  • value function of a resistor set is calculated as follows:
  • the value function is a composition of square mean or maximum of the distances array.
  • Some implementations iteratively search for an optimal resistor set by consecutively adjusting each resistor value in the resistor set on a learning rate value.
  • the learning rate changes over time.
  • an initial resistor set is chosen as uniform (e.g., [1;1;...;1]), with minimum and maximum resistor values chosen to be within two orders of magnitude range (e.g., [1 ; 100] or [0.1; 10]).
  • R+ R-.
  • the iterative process converges to a local minimum.
  • the process resulted in the following set: [0.17, 1.036, 0.238, 0.21, 0.362, 1.473, 0.858, 0.69, 5.138, 1.215, 2.083, 0.275]
  • Some implementations do not use the whole available range [rmin; rmax] for finding a good local optimum. Only part of the available range (e.g., in this case [0.17; 5.13]) is used.
  • the resistor set values are relative, not absolute. Is this case, relative value range of 30 is enough for the resistor set.
  • the following resistor set of length 20 is obtained for abovementioned parameters: [0.300, 0.461, 0.519, 0.566, 0.648, 0.655, 0.689, 0.996, 1.006, 1.048, 1.186, 1.222, 1.261, 1.435, 1.488, 1.524, 1.584, 1.763, 1.896, 2.02]
  • This set is subsequently used to produce weights for NN, producing corresponding model S.
  • the model S’s mean square output error was 11 mV given the relative resistor error is close to zero, so the set of 20 resistors is more than required.
  • Maximum error over a set of input data was calculated to be 33 mV.
  • S, DAC, and ADC converters with 256 levels were analyzed as a separate model, and the result showed 14 mV mean square output error and 49 mV max output error.
  • An output error of 45 mV on NN corresponds to a relative recognition error of 1%.
  • the 45 mV output error value also corresponds to 0.01 relative or 0.01 absolute weight error, which is acceptable.
  • Maximum weight modulus in NN is 1.94. In this way, the optimal (or near optimal) resistor set is determined using the iterative process, based on desired weight range [-wlim; wlim], resistors error (relative), and possible resistors range.
  • a very broad resistor set is not very beneficial (e.g., between 1-1/5 orders of magnitude is enough) unless different precision is required within different layers or weight spectrum parts. For example, suppose weights are in the range of [0, 1], but most of the weights are in the range of [0, 0.001], then better precision is needed within that range. In the example described above, given the relative resistor error is close to zero, the set of 20 resistors is more than sufficient for quantizing the NN network, with given precision.
  • the example computations described herein are performed by the weight matrix computation or weight quantization module 238 (e.g., using the resistance calculation module 240) that compute the weights 272 for connections of the transformed neural networks, and/or corresponding resistance values 242 for the weights 272.
  • This section describes an example process for quantizing resistor values corresponding to weights of a trained neural network, according to some implementations.
  • the example process substantially simplifies the process of manufacturing chips using analog hardware components for realizing neural networks.
  • some implementations use resistors to represent neural network weights and/or biases for operational amplifiers that represent analog neurons.
  • the example process described here specifically reduces the complexity in lithographically fabricating sets of resistors for the chip. With the procedure of quantizing the resistor values, only select values of resistances are needed for chip manufacture. In this way, the example process simplifies the overall process of chip manufacture and enables automatic resistor lithographic mask manufacturing on demand.
  • Figure 18 provides an example scheme of a neuron model 1800 used for resistors quantization, according to some implementations.
  • the circuit is based on an operational amplifier 1824 (e.g., AD824 series precision amplifier) that receives input signals from negative weight fixing resistors (Rl- 1804, R2- 1806,, Rb- bias 1816, Rn- 1818, and R- 1812), and positive weight fixing resistors (R1+ 1808, R2+ 1810, Rb+ bias 1820, Rn+ 1822), and R+ 1814).
  • the positive weight voltages are fed into direct input of the operational amplifier 1824 and negative weights voltages are fed into inverse input of the operational amplifier 1824.
  • the operational amplifier 1824 is used to allow weighted summation operation of weighted outputs from each resistor, where negative weights are substracted from positive weights.
  • the operational amplifier 1824 also amplifies signal to the extent necessary for the circuit operation.
  • the operational amplifier 1824 also accomplishes RELU transformation of output signal at it’s output cascade.
  • connection weights and biases (wl, ..., wn, b ⁇ ..
  • Rmin, Rmax ⁇ are determined based on the technology used for manufacturing. Some implementations use TaN or Tellurium high resistivity materials. In some implementations, the minimum value of resistor is determined by minimum square that can be formed lithographically. The maximum value is determined by length, allowable for resistors (e.g., resistors made from TaN or Tellurium) to fit to the desired area, which is in turn determined by the area of an operational amplifier square on lithographic mask. In some implementations, the area of arrays of resistors is smaller than the area of one operational amplifier, since the arrays of resistors are stacked (e.g., one in BEOL, another in FEOL).
  • resistors e.g., resistors made from TaN or Tellurium
  • the goal is to select a set of resistor values (Rl, ..., Rn ⁇ of given length N within the defined [Rmin; Rmax], based on (wl, ..., wn, b ⁇ values.
  • An example search algorithm is provided below to find sub-optimal (Rl, ..., Rn ⁇ set based on particular optimality criteria. 5.
  • Another algorithm chooses (Rn, Rp, Rni, Rpi ⁇ for a network given that ⁇ RL.Rn ⁇ is determined.
  • Expected error value for each weight option is estimated based on potential resistor relative error r err determined by IC manufacturing technology.
  • Weight options list is limited or restricted to [-wlim; wlim] range
  • Value function is calculated as a square mean of distance between two neighboring weight options. So, value function is minimal when weight options are distributed uniformly within [-wlim; wlim] range
  • rmin and rmax are minimum and maximum values for resistances, respectively.
  • the following resistor set of length 20 was obtained for abovementioned parameters: [0.300, 0.461, 0.519, 0.566, 0.648, 0.655, 0.689, 0.996, 1.006, 1.048, 1.186, 1.222, 1.261, 1.435, 1.488, 1.524, 1.584, 1.763, 1.896, 2.02] MW.
  • Some implementations subsequently use the (Rni; Rpi; Rn; Rp ⁇ values set to implement neural network schematics.
  • the schematics produced mean square output error (sometimes called S mean square output error, described above) of 11 mV and max error of 33 mV over a set of 10,000 uniformly distributed input data samples, according to some implementations.
  • S model was analyzed along with digital-to- analog converters (DAC), analog-to-digital converters (ADC), with 256 levels as a separate model.
  • the model produced 14 mV mean square output error and 49 mV max output error on the same data set, according to some implementations.
  • DAC and ADC have levels because they convert analog value to bit value and vice-versa. 8 bits of digital value is equal to 256 levels. Precision cannot be better than 1/256 for 8-bit ADC.
  • Some implementations calculate the resistance values for analog IC chips, when the weights of connections are known, based on Kirchhoff s circuit laws and basic principles of operational amplifiers (described below in reference to Figure 19A), using Mathcad or any other similar software.
  • operational amplifiers are used both for amplification of signal and for transformation according to the activation functions (e.g., ReLU, sigmoid, Tangent hyperbolic, or linear mathematical equations),
  • Some implementations manufacture resistors in a lithography layer where resistors are formed as cylindrical holes in the Si02 matrix and the resistance value is set by the diameter of hole.
  • Some implementations use amorphous TaN, TiN of CrN or Tellurium as the highly resistive material to make high density resistor arrays.
  • Some ratios of Ta to N Ti to N and Cr to N provide high resistance for making ultra-dense high resistivity elements arrays. For example, for TaN, Ta5N6, Ta3N5, the higher the N ratio to Ta, the higher is the resistivity.
  • Some implementations use Ti2N, TiN, CrN, or Cr5N, and determine the ratios accordingly.
  • TaN deposition is a standard procedure used in chip manufacturing and is available at all major Foundries.
  • Figure 19A shows a schematic diagram of an operational amplifier made on CMOS (CMOS OpAmp) 1900, according to some implementations.
  • In+ positive input or pos
  • In- negative input or neg
  • Vdd- positive supply voltage relative to GND
  • Contact Vss- negative supply voltage or GND
  • the circuit output is Out 1410 (contact output).
  • Parameters of CMOS transistors are determined by the ratio of geometric dimensions: L (the length of the gate channel) to W (the width of the gate channel), examples of which are shown in the Table shown in Figure 19B (described below).
  • the current mirror is made on NMOS transistors Mil 1944, Ml 2 1946, and resistor R1 1921 (with an example resistance value of 12 kO), and provides the offset current of the differential pair (Ml 1926 and M3 1930).
  • the differential amplifier stage (differential pair) is made on the NMOS transistors Ml 1926 and M3 1930.
  • Transistors Ml, M3 are amplifying, and PMOS transistors M2 1928 and M4 1932 play the role of active current load. From the M3 transistor, the signal is input to the gate of the output PMOS transistor M7 1936. From the transistor Ml, the signal is input to the PMOS transistor M5 (inverter) 1934 and the active load on the NMOS transistor M6 1934.
  • the current flowing through the transistor M5 1934 is the setting for the NMOS transistor M8 1938.
  • Transistors M7 1936 is included in the scheme with a common source for a positive half-wave signal.
  • the M8 transistors 1938 are enabled by a common source circuit for a negative half-wave signal.
  • the M7 1936 and M8 1938 outputs include an inverter on the M9 1940 and M10 1942 transistors.
  • Capacitors Cl 1912 and C2 1914 are blocking.
  • Figure 19B shows a table 1948 of description for the example circuit shown in Figure 19 A, according to some implementations.
  • the values for the parameters are provided as examples, and various other configurations are possible.
  • the transistors Ml, M3, M6, M8, M10, Mi l, and M12 are N-Channel MOSFET transistors with explicit substrate connection.
  • the other transistors M2, M4, M5, M7, and M9 are P-Channel MOSFET transistors with explicit substrate connection.
  • the Table shows example shutter ratio of length (L, column 1) and width (W, column 2) are provided for each of the transistors (column 3).
  • operational amplifiers such as the example described above are used as the basic element of integrated circuits for hardware realization of neural networks.
  • the operational amplifiers are of the size of 40 square microns and fabricated according to 45 nm node standard.
  • activation functions such as ReLU, Hyperbolic Tangent, and Sigmoid functions are represented by operational amplifiers with modified output cascade.
  • RELU ReLU
  • Sigmoid or Tangent function is realized as an output cascade of an operational amplifier (sometimes called OpAmp) using corresponding well- known analog schematics, according to some implementations.
  • the operational amplifiers are substituted by inverters, current mirrors, two-quadrant or four quadrant multipliers, and/or other analog functional blocks, that allow weighted summation operation.
  • Figures 20A-20E show a schematic diagram of a LSTM neuron 20000, according to some implementations.
  • the inputs of the neuron are Vinl 20002 and Vin2 20004 that are values in the range [-0.1,0.1]
  • the LSTM neuron also input the value of the result of calculating the neuron at time H(t-1) (previous value; see description above for LST neuron) 20006 and the state vector of the neuron at time C(t-l) (previous value) 20008.
  • Outputs of the neuron LSTM (shown in Figure 20B) include the result of calculating the neuron at the present time H(t) 20118 and the state vector of the neuron at the present time C(t) 20120.
  • the scheme includes:
  • a “neuron O” assembled on the operational amplifiers U1 20094 and U220100, shown in Figure 20A.
  • Resistors R_Wol 20018, R_Wo2 20016, R_Wo3 20012, R_Wo4 20010, R Uopl 20014, R_Uoml 20020, Rr 20068 and Rf2 20066 set the weights of connections of the single “neuron O”.
  • the “neuron O” uses a sigmoid (module XI 20078, Figure 20B) as a nonlinear function;
  • Neuroneuron C • a "neuron C“ assembled on the operational amplifiers U3 20098 (shown in Figure 20C) and U420100 (shown in Figure 20A).
  • the “neuron C” uses a hyperbolic tangent (module X2 22080, Figure 2B) as a nonlinear function;
  • a “neuron I” assembled on the operational amplifiers U5 20102 and U6 20104, shown in Figure 20C.
  • Resistors R_Wil 20042, R_Wi2 20040, R_Wi3 20036, and R_Wi4 20034, R Uipl 20038, R_Uiml 20044, Rr 20124, and Rf2 20126 set the weights of connections of the “neuron I”.
  • the “neuron I” uses a sigmoid (module X3 20082) as a nonlinear function; and
  • Resistors R_Wfl 20054, R_Wf220052, R W13 20048, R_Wf420046, R Ufpl 20050, R Ufml 20056, Rr 20128 and Rf2 20130 set the weights of connections of the “neuron f’.
  • the “neuron f’ uses a sigmoid (module X4 20084) as a nonlinear function.
  • the output C(t) 20120 (a current state vector of the LSTM neuron) is obtained with the buffer-inverter on the U11 20114 output signal.
  • the outputs of modules XI 20078 and X7 20090 is input to a multiplier (module X8 20092) whose output is input to a buffer divider by 10 on the U12 20116.
  • the result of calculating the LSTM neuron at the present time H(t) 20118 is obtained from the output signal of U12 20116.
  • Figure 20E shows example values for the different configurable parameters (e.g., voltages) for the circuit shown in Figures 20A-20D, according to some implementations.
  • Vdd 20058 is set to +1.5V
  • Vss 20064 is set to -1.5V
  • Vddl 20060 is set to +1.8V
  • Vssl 20062 is set to -1.0V
  • GND 20118 is set to GND, according to some implementations.
  • Figure 20F shows a table 20132 of description for the example circuit shown in Figure 20A-20D, according to some implementations.
  • the values for the parameters are provided as examples, and various other configurations are possible.
  • the transistors U1 - U12 are CMOS OpAmps (described above in reference to Figures 19A and 19B).
  • XI, X3, and X4 are modules that perform the Sigmoid function.
  • X2 and X7 are modules that perform the Hyperbolic Tangent function.
  • X5 and X8 are modules that perform the multiplication function.
  • Figures 21A-21I show a schematic diagram of a multiplier block 21000, according to some implementations.
  • the neuron 21000 is based on the principle of a four- quadrant multiplier, assembled using operational amplifiers U1 21040 and U221042 (shown in Figure 2 IB), U3 21044 (shown in Figure 21H), and U4 21046 and U5 21048 (shown in Figure 211), and CMOS transistors Ml 21052 through M68 21182.
  • the inputs of the multiplier include V_one 2102021006 and V_two 21008 (shown in Figure 21B), and contact Vdd (positive supply voltage, e.g., +1.5 V relative to GND) 21004 and contact Vss (negative supply voltage, e.g., -1.5 V relative to GND) 21002.
  • additional supply voltages are used: contact Input Vddl (positive supply voltage, e.g., +1.8 V relative to GND), contact Vssl (negative supply voltage, e.g., -1.0 V relative to GND).
  • the result of the circuit calculations are output at mult out (output pin) 21170 (shown in Figure 211).
  • input signal (V one) from V one 21006 is connected to the inverter with a single gain made on U1 21040, the output of which forms a signal negA 21006, which is equal in amplitude, but the opposite sign with the signal V one.
  • the signal (V two) from the input V two 21008 is connected to the inverter with a single gain made on U2 21042, the output of which forms a signal negB 21012 which is equal in amplitude, but the opposite sign with the signal V two. Pairwise combinations of signals from possible combinations (V one, V two, negA, negB) are output to the corresponding mixers on CMOS transistors.
  • V_two 21008 and negA 21010 are input to a multiplexer assembled on NMOS transistors Ml 9 21086, M20 21088, M21 21090, M22 21092, and PMOS transistors M23 21094 and M24 21096.
  • the output of this multiplexer is input to the NMOS transistor M6 21060 ( Figure 2 ID).
  • negB 21012 and V_one 21020 are input to a multiplexer assembled on NMOS transistors Ml 1 21070, M122072, M13 2074, M1421076, and PMOS transistors M15 2078 and M16 21080.
  • the output of this multiplexer is input to the M5 21058 NMOS transistor (shown in Figure 2 ID);
  • V one 21020 and negB 21012 are input to a multiplexer assembled on PMOS transistors M18 21084, M48 21144, M49 21146, and M50 21148, and NMOS transistors M17 21082, M47 21142.
  • the output of this multiplexer is input to the M9 PMOS transistor 21066 (shown in Figure 21D);
  • V_two 21008 and V_one 21020 are input to a multiplexer assembled on NMOS transistors M41 21130, M42 21132, M43 21134, and M44 21136, and PMOS transistors M45 21138, and M46 21140.
  • the output of this multiplexer is input to the M30 NMOS transistor 21108 (shown in Figure 21H);
  • V one 21020 and V two 21008 are input to a multiplexer assembled on PMOS transistors M58 21162, M60 21166, M61 21168, and M62 21170, and NMOS transistors M57 21160, and M59 21164.
  • the output of this multiplexer is input to the M34 PMOS transistor 21116 (shown in Figure 21H); and
  • negA 21010 and negB 21012 are input to a multiplexer assembled on PMOS transistors M64 21174, M66 21178, M67 21180, and M68 21182, and NMOS transistors M63 21172, and M65 21176.
  • the output of this multiplexer is input to the PMOS transistor M33 21114 (shown in Figure 21H).
  • the current mirror powers the portion of the four quadrant multiplier circuit shown on the left, made with transistors M5 21058, M6 21060, M7 21062, M8 21064, M9 21066, and M10 21068.
  • Current mirrors on transistors M25 21098, M2621100, M2721102, and M2821104) power supply of the right portion of the four-quadrant multiplier, made with transistors M2921106, M30 21108, M31 21110, M3221112, M33 21114, and M3421116.
  • the multiplication result is taken from the resistor Ro 21022 enabled in parallel to the transistor M3 21054 and the resistor Ro 21188 enabled in parallel to the transistor M28 21104, supplied to the adder on U3 21044.
  • the output of U3 21044 is supplied to an adder with a gain of 7,1, assembled on U5 21048, the second input of which is compensated by the reference voltage set by resistors R1 21024 and R221026 and the buffer U421046, as shown in Figure 211.
  • the multiplication result is output via the Mult Out output 21170 from the output of U5 21048.
  • FIG. 21 J shows a table 21198 of description for the schematic shown in Figures 21 A-21I, according to some implementations.
  • U1 - U5 are CMOS OpAmps.
  • Figure 22A shows a schematic diagram of a sigmoid block 2200, according to some implementations.
  • the sigmoid function (e.g., modules XI 20078, X3 20082, and X4 20084, described above in reference to Figures 20A-20F) is implemented using operational amplifiers U1 2250, U2 2252, U3 2254, U4 2256, U5 2258, U62260, U7, 2262, and U8 2264, and NMOS transistors Ml 2266, M2 2268, and M3 2270.
  • Contact sigm_in 2206 is module input, contact Input Vddl 2222 is positive supply voltage +1.8 V relative to GND 2208, and contact Vssl 2204 is negative supply voltage -1.0 V relative to GND.
  • U4 2256 has a reference voltage source of -0.2332 V, and the voltage is set by the divider R10 2230 and R11 2232.
  • the U5 2258 has a reference voltage source of 0.4 V, and the voltage is set by the divider R12 2234 and R13 2236.
  • the U6 2260 has a reference voltage source of 0.32687 V, the voltage is set by the divider R14 2238 and R15 2240.
  • the U7 2262 has a reference voltage source of -0.5 V, the voltage is set by the divider R16 2242 and R17 2244.
  • the U8 2264 has a reference voltage source of -0.33 V, the voltage is set by the divider R18 2246 and R19 2248.
  • the sigmoid function is formed by adding the corresponding reference voltages on a differential module assembled on the transistors Ml 2266 and M2 2268.
  • a current mirror for a differential stage is assembled with active regulation operational amplifier U3 2254, and the NMOS transistor M3 2270.
  • the signal from the differential stage is removed with the NMOS transistor M2 and resistor R5 2220 is input to the adder U22252.
  • the output signal sigm out 2210 is removed from the U2 adder 2252 output.
  • Figure 22B shows a table 2278 of description for the schematic diagram shown in Figure 22A, according to some implementations.
  • U1-U8 are CMOS OpAmps.
  • FIG 23 A shows a schematic diagram of a hyperbolic tangent function block 2300, according to some implementations.
  • the hyperbolic tangent function e.g., the modules X2 20080, and X7 20090 described above in reference to Figures 20A-20F
  • the hyperbolic tangent function is implemented using operational amplifiers (U1 2312, U22314, U3 2316, U42318, U5 2320, U6 2322, U72328, and U82330) and NMOS transistors (Ml 2332, M22334, and M3 2336).
  • contact tanh in 2306 is module input
  • contact Input Vddl 2304 is positive supply voltage +1.8 V relative to GND 2308
  • contact Vssl 2302 is negative supply voltage -1.0 V relative to GND.
  • U4 2318 has a reference voltage source of -0.1 V, the voltage set by the divider R10 2356 and R11 2358.
  • the U5 2320 has a reference voltage source of 1.2 V, the voltage set by the divider R12 2360 and R13 2362.
  • the U62322 has a reference voltage source of 0.32687 V, the voltage set by the divider R14 2364 and R15 2366.
  • the U7 2328 has a reference voltage source of -0.5 V, the voltage set by the divider R162368 and R172370.
  • the U8 2330 has a reference voltage source of -0.33 V, the voltage set by the divider R18 2372 and R19 2374.
  • the hyperbolic tangent function is formed by adding the corresponding reference voltages on a differential module made on transistors Ml 2332 and M2 2334.
  • a current mirror for a differential stage is obtained with active regulation operational amplifier U3 2316, and NMOS transistor M3 2336. With NMOS transistor M2 2334 and resistor R5 2346, the signal is removed from the differential stage and input to the adder U2 2314.
  • the output signal tanh out 2310 is removed from the U2 adder 2314 output.
  • Figure 23B shows a table 2382 of description for the schematic diagram shown in Figure 23 A, according to some implementations.
  • U1-U8 are CMOS OpAmps
  • FIGS 24A-24C show a schematic diagram of a single neuron OP1 CMOS OpAmp_2400, according to some implementations.
  • the example is a variant of a single neuron on an operational amplifier, made on CMOS according to an OP1 scheme described herein.
  • contacts VI 2410 and V22408 are inputs of a single neuron
  • contact bias 2406 is voltage +0.4 V relative to GND
  • contact Input Vdd 2402 is positive supply voltage +5.0 V relative to GND
  • contact Vss 2404 is GND
  • contact Out 2474 is output of a single neuron.
  • Parameters of CMOS transistors are determined by the ratio of geometric dimensions: L (the length of the gate channel), and W (the width of the gate channel).
  • This Op Amp has two current mirrors.
  • the current mirror on NMOS transistors M3 2420, M6 2426, and Ml 3 2440 provides the offset current of the differential pair on NMOS transistors M2 2418 and M5 2424.
  • the current mirror in the PMOS transistors M72428, M82430, and Ml 5 2444 provides the offset current of the differential pair on the PMOS transistors M9 2432 and M102434.
  • NMOS transistors M22418 and M5 2424 are amplifying, and PMOS transistors Ml 2416 and M42422 play the role of active current load. From the M5 2424 transistor, the signal is output to the PMOS gate of the transistor Ml 3 2440.
  • the signal is output to the right input of the second differential amplifier stage on PMOS transistors M92432 and M102434.
  • NMOS transistors Ml 1 2436 and M122438 play the role of active current load for the M92432 and M10 2434 transistors.
  • the Ml 72448 transistor is switched on according to the scheme with a common source for a positive half-wave of the signal.
  • the M182450 transistor is switched on according to the scheme with a common source for the negative half-wave of the signal.
  • an inverter on the Ml 72448 and Ml 8 2450 transistors is enabled at the output of the M13 2440 and M142442 transistors.
  • Figure 24D shows a table 2476 of description for the schematic diagram shown in Figure 24A-24C, according to some implementations.
  • Figures 25A-25D show a schematic diagram of a variant of a single neuron 25000 on operational amplifiers, made on CMOS according to an OP3 scheme, according to some implementations.
  • the single neuron consists of three simple operational amplifiers (OpAmps), according to some implementations.
  • Transistors Ml 25028 - M16 25058 are used for summation of negative connections of the neuron.
  • Transistors M1725060 - M32 25090 are used for adding the positive connections of the neuron.
  • the RELU activation function is performed on the transistors M33 25092 - M4625118.
  • contacts VI 25008 and V2 25010 are inputs of the single neuron
  • contact bias 25002 is voltage +0.4 V relative to GND
  • contact Input Vdd 25004 is positive supply voltage +2.5 V relative to GND
  • contact Vss 25006 is negative supply voltage -2.5 V
  • contact Out 25134 is output of the single neuron.
  • Parameters of CMOS transistors used in a single neuron are determined by the ratio of geometric dimensions: L (the length of the gate channel) and W (the width of the gate channel).
  • the current mirror on NMOS transistors M3 25032 (Ml 9 25064, M35 25096), M6 25038 (M22 25070, M38 25102) and M16 25058 (M32 25090, M48 25122) provides the offset current of the differential pair on NMOS transistors M2 25030 (M18 25062, M34 25094) and M5 25036 (M21 25068, M35 25096).
  • PMOS transistors M7 25040 M23 25072, M39 25104
  • M8 25042 M24 25074, M40 25106
  • M15 25056 M31 2588
  • NMOS transistors M2 25030 Ml 8 25062, M34 25094
  • M5 25036 M21 25068, M37 25100
  • PMOS transistors Ml 25028 M1725060, M33 25092
  • M425034 M20 25066, M3625098
  • the signal is input to the PMOS gate of the transistor M13 25052 (M29 25084, M45 25116).
  • the signal is input to the right input of the second differential amplifier stage on PMOS transistors M9 25044 (M25 25076, M41 25108) and M10 25046 (M26 25078, M42 25110).
  • NMOS transistors Ml 1 25048 (M2725080, M43 25112) and M1225048 (M28 25080, M4425114) play the role of active current load for transistors M9 25044 (M25 25076, M41 25108) and M10 25046 (M2625078, M42 25110).
  • Transistor M13 25052 (M29 25082, M45 25116) is included in the scheme with a common source for a positive half-wave signal.
  • the transistor M14 25054 (M30 25084, M46 25118) is switched on according to the scheme with a common source for the negative half-wave of the signal.
  • R feedback 100k - used only for calculating wl, w2, wbias.
  • the input of the negative link adder of the neuron (Ml - Ml 7) is received from the positive link adder of the neuron (Ml 7 - M32) through the Rcom resistor.
  • FIG. 25E shows a table 25136 of description for the schematic diagram shown in Figure 25A-25D, according to some implementations.
  • FIGS 27A-27J show a flowchart of a method 2700 for hardware realization (2702) of neural networks, according to some implementations.
  • the method is performed (2704) at the computing device 200 (e.g., using the neural network transformation module 226) having one or more processors 202, and memory 214 storing one or more programs configured for execution by the one or more processors 202.
  • the method includes obtaining (2706) a neural network topology (e.g., the topology 224) and weights (e.g., the weights 222) of a trained neural network (e.g., the networks 220).
  • the trained neural network is trained (2708) using software simulations to generate the weights.
  • the method also includes transforming (2710) the neural network topology to an equivalent analog network of analog components.
  • the neural network topology includes (2724) one or more layers of neurons. Each layer of neurons computing respective outputs based on a respective mathematical function.
  • transforming the neural network topology to the equivalent analog network of analog components includes, performing (2726) a sequence of steps for each layer of the one or more layers of neurons. The sequence of steps include identifying (2728) one or more function blocks, based on the respective mathematical function, for the respective layer. Each function block has a respective schematic implementation with block outputs that conform to outputs of a respective mathematical function.
  • identifying the one or more function blocks includes selecting (2730) the one or more function blocks based on a type of the respective layer. For example, a layer can consist of neurons, and the layer’s output is a linear superposition of its inputs. Selecting the one or more function blocks is based on this identification of a layer type, if a layer’s output is a linear superposition, or similar pattern identification. Some implementations determine if number of output > 1, then use either a trapezium or a pyramid transformation.
  • the one or more function blocks include one or more basic function blocks (e.g., the basic function blocks 232) selected (2734) from the group consisting of: (i) a weighted summation block (2736) with a block output V out + bias).
  • ReLU Rectified Linear Unit (ReLU) activation function or a similar activation function (e.g., ReLU with a threshold), V t represents an i-th input, W j represents a weight corresponding to the i-th input, and bias represents a bias value, and ⁇ is a summation operator;
  • a signal multiplier block (2738) with a block output V out coeff.
  • V t . V j . V t represents an i-th input and V j represents a j-th input, and coeff is a predetermined coefficient;
  • a hyperbolic tangent activation block (2742) with a block output V out A * tanh(B * V in ).
  • V in represents an input
  • t represents a current time-period
  • V(t — 1) represents an output of the signal delay block for a preceding time period t — 1
  • dt is a delay value.
  • the sequence of steps also includes generating (2732) a respective multilayer network of analog neurons based on arranging the one or more function blocks.
  • Each analog neuron implements a respective function of the one or more function blocks, and each analog neuron of a first layer of the multilayer network is connected to one or more analog neurons of a second layer of the multilayer network.
  • transforming (2710) the neural network topology to an equivalent analog network of analog components requires more complex processing, according to some implementations.
  • the neural network topology includes (2746) one or more layers of neurons.
  • each layer of neurons computes respective outputs based on a respective mathematical function.
  • transforming the neural network topology to the equivalent analog network of analog components includes: (i) decomposing (2748) a first layer of the neural network topology to a plurality of sub-layers, including decomposing a mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions.
  • Each sub-layer implements an intermediate mathematical function.
  • the mathematical function corresponding to the first layer includes one or more weights
  • decomposing the mathematical function includes adjusting (2750) the one or more weights such that combining the one or more intermediate functions results in the mathematical function.
  • performing (2752) a sequence of steps for each sub-layer of the first layer of the neural network topology.
  • the sequence of steps includes selecting (2754) one or more sub-fimction blocks, based on a respective intermediate mathematical function, for the respective sub-layer; and generating (2756) a respective multilayer analog sub-network of analog neurons based on arranging the one or more sub-function blocks.
  • Each analog neuron implements a respective function of the one or more sub-fimction blocks, and each analog neuron of a first layer of the multilayer analog sub-network is connected to one or more analog neurons of a second layer of the multilayer analog sub-network.
  • transforming the neural network topology includes generating (2770) one or more signal delay blocks for each recurrent connection of the one or more GRU or LSTM neurons.
  • an external cycle timer activates the one or more signal delay blocks with a constant time period (e.g., 1, 5, or 10 time steps). Some implementations use multiple delay blocks over one signal for producing additive time shift.
  • the activation frequency of the one or more signal delay blocks is/are synchronized to network input signal frequency.
  • the one or more signal delay blocks are activated (2772) at a frequency that matches a predetermined input signal frequency for the neural network topology.
  • this predetermined input signal frequency may be dependent on the application, such as Human Activity Recognition (HAR) or PPG.
  • HAR Human Activity Recognition
  • the predetermined input signal frequency is 30-60 Hz for video processing, around 100 Hz for HAR and PPG, 16 KHz for sound processing, and around 1-3 Hz for battery management.
  • Some implementations activate different signal delay blocks activate at different frequencies.
  • transforming the neural network topology includes applying (2776) one or more transformations selected from the group consisting of: replacing (2778) the unlimited activation functions with limited activation (e.g., replacing ReLU with a threshold ReLU); and adjusting (2780) connections or weights of the equivalent analog network such that, for predetermined one or more inputs, difference in output between the trained neural network and the equivalent analog network is minimized.
  • the method also includes computing (2712) a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent analog network.
  • the method also includes generating (2714) a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components.
  • generating the schematic model includes generating (2716) a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value.
  • the method includes regenerating just the resistance matrix for the resistors for a retrained network.
  • the method further includes obtaining (2718) new weights for the trained neural network, computing (2720) a new weight matrix for the equivalent analog network based on the new weights, and generating (2722) a new resistance matrix for the new weight matrix.
  • the method further includes generating (2782) one or more lithographic masks (e.g., generating the masks 250 and/or 252 using the mask generation module 248) for fabricating a circuit implementing the equivalent analog network of analog components based on the resistance matrix.
  • the method includes regenerating just the masks for resistors (e.g., the masks 250) for retrained networks.
  • the method further includes: (i) obtaining (2784) new weights for the trained neural network; (ii) computing (2786) a new weight matrix for the equivalent analog network based on the new weights; (iii) generating (2788) a new resistance matrix for the new weight matrix; and (iv) generating (2790) a new lithographic mask for fabricating the circuit implementing the equivalent analog network of analog components based on the new resistance matrix.
  • the analog components include (2762) a plurality of operational amplifiers and a plurality of resistors.
  • Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons.
  • Some implementations include other analog components, such as four-quadrant multipliers, sigmoid and hyperbolic tangent function circuits, delay lines, summers, and/or dividers.
  • selecting (2764) component values of the analog components includes performing (2766) a gradient descent method and/or other weight quantization methods to identify possible resistance values for the plurality of resistors.
  • the method further includes implementing certain activation functions (e.g., Softmax) in output layer in digital.
  • the method further includes generating (2758) equivalent digital network of digital components for one or more output layers of the neural network topology, and connecting (2760) output of one or more layers of the equivalent analog network to the equivalent digital network of digital components.
  • FIGS 28A-28S show a flowchart of a method 28000 for hardware realization (28002) of neural networks according to hardware design constraints, according to some implementations
  • the method is performed (28004) at the computing device 200 (e.g., using the neural network transformation module 226) having one or more processors 202, and memory 214 storing one or more programs configured for execution by the one or more processors 202.
  • the method includes obtaining (28006) a neural network topology (e.g., the topology 224) and weights (e.g., the weights 222) of a trained neural network (e.g., the networks 220).
  • the method also includes calculating (28008) one or more connection constraints based on analog integrated circuit (IC) design constraints (e.g., the constraints 236).
  • IC design constraints can set the current limit (e.g., 1A)
  • neuron schematics and operational amplifier (OpAmp) design can set the OpAmp output current in the range [0-10mA], so this limits output neuron connections to 100.
  • This means that the neuron has 100 outputs which allow the current to flow to the next layer through 100 connections, but current at the output of the operational amplifier is limited to 10 mA, so some implementations use a maximum of 100 outputs (0.1 mA times 100 10 mA).
  • the method also includes transforming (28010) the neural network topology (e.g., using the neural network transformation module 226) to an equivalent sparsely connected network of analog components satisfying the one or more connection constraints.
  • transforming the neural network topology includes deriving (28012) a possible input connection degree JV j and output connection degree JV 0 , according to the one or more connection constraints.
  • the neural network topology includes (28018) at least one densely connected layer with K inputs (neurons in previous layer) and L outputs (neurons in current layer) and a weight matrix U , and transforming (28020) the at least one densely connected layer includes constructing (28022) the equivalent sparsely connected network with K inputs, L outputs, and [ log N. K] + ⁇ log No L] — 1 layers, such that input connection degree does not exceed JV j , and output connection degree does not exceed JV 0 .
  • the neural network topology includes (28024) at least one densely connected layer with K inputs (neurons in previous layer) and L outputs (neurons in current layer) and a weight matrix U , and transforming (28026) the at least one densely connected layer includes: constructing (28028) the equivalent sparsely connected network with K inputs, L outputs, and M 3 max( log N. L ], ⁇ log No K ) layers.
  • Each layer m is represented by a corresponding weight matrix U m , where absent connections are represented with zeros, such that input connection degree does not exceed JV j , and output connection degree does not exceed JV 0 .
  • the predetermined precision is a reasonable precision value that statistically guarantees that altered networks output differs from referent network output by no more than allowed error value, and this error value is task-dependent (typically between 0.1% and 1%).
  • the neural network topology includes (28030) a single sparsely connected layer with K inputs and L outputs, a maximum input connection degree of P a maximum output connection degree of P 0 , and a weight matrix of U , where absent connections are represented with zeros.
  • transforming (28032) the single sparsely connected layer includes constructing (28034) the equivalent sparsely connected network with K inputs, L outputs, M 3 max( log N. P i ⁇ , ⁇ log No P 0 ]) layers.
  • the neural network topology includes (28036) a convolutional layer (e.g., a Depthwise convolutional layer, or a Separable convolutional layer) with K inputs (neurons in previous layer) and L outputs (neurons in current layer).
  • transforming (28038) the neural network topology to the equivalent sparsely connected network of analog components includes decomposing (28040) the convolutional layer into a single sparsely connected layer with K inputs, L outputs, a maximum input connection degree of P and a maximum output connection degree of P 0 , where P t £ N t and P 0 £ N 0.
  • the method also includes computing (28014) a weight matrix for the equivalent sparsely connected network based on the weights of the trained neural network.
  • Each element of the weight matrix represents a respective connection between analog components of the equivalent sparsely connected network.
  • the neural network topology includes (28042) a recurrent neural layer, and transforming (28044) the neural network topology to the equivalent sparsely connected network of analog components includes transforming (28046) the recurrent neural layer into one or more densely or sparsely connected layers with signal delay connections.
  • the neural network topology includes a recurrent neural layer (e.g., a long short-term memory (LSTM) layer or a gated recurrent unit (GRU) layer), and transforming the neural network topology to the equivalent sparsely connected network of analog components includes decomposing the recurrent neural layer into several layers, where at least one of the layers is equivalent to a densely or sparsely connected layer with K inputs (neurons in previous layer) and L outputs (neurons in current layer) and a weight matrix U , where absent connections are represented with zeros.
  • LSTM long short-term memory
  • GRU gated recurrent unit
  • the method includes performing a transformation of a single layer perceptron with one calculation neurons.
  • the neural network topology includes (28054) K inputs, a weight vector U E R K , and a single layer perceptron with a calculation neuron with an activation function F.
  • the equivalent sparsely connected network includes respective one or more analog neurons in each layer of the m layers.
  • computing (28064) the weight matrix for the equivalent sparsely connected network includes calculating (28066) a weight vector W for connections of the equivalent sparsely connected network by solving a system of equations based on the weight vector U.
  • the method includes performing a transformation of a single layer perceptron with L calculation neurons.
  • the neural network topology includes (28068) K inputs, a single layer perceptron with L calculation neurons, and a weight matrix V that includes a row of weights for each calculation neuron of the calculation neurons.
  • Each single layer perceptron network includes a respective calculation neuron of the L calculation neurons; (iv) for each single layer perceptron network (28078) of the L single layer perceptron networks, constructing (28080) a respective equivalent pyramid like sub-network for the respective single layer perceptron network with the K inputs, the m layers and the connection degree N.
  • the equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m- 1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron; and (v) constructing (28082) the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating an input of each equivalent pyramid-like sub-network for the L single layer perceptron networks to form an input vector with L*K inputs.
  • the method includes performing a transformation algorithm for multi-layer perceptron.
  • the neural network topology includes (28092) K inputs, a multi-layer perceptron with S layers, each layer i of the S layers includes a corresponding set of calculation neurons L, and corresponding weight matrices V that includes a row of weights for each calculation neuron of the L x calculation neurons.
  • Each single layer perceptron network includes a respective calculation neuron of the Q calculation neurons.
  • K is number of inputs for the respective calculation neuron in the multi-layer perceptron, and (b) constructing (28104) the respective equivalent pyramid-like sub-network for the respective single layer perceptron network with K, , j inputs, the m layers and the connection degree N.
  • the equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m- ⁇ layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron network; and (iv) constructing (28106) the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating input of each equivalent pyramid-like sub-network for the Q single layer perceptron networks to form an input vector with Q*K J inputs.
  • the neural network topology includes (28116) a Convolutional Neural Network (CNN) with K inputs, S layers, each layer i of the S layers includes a corresponding set of calculation neurons L, and corresponding weight matrices V that includes a row of weights for each calculation neuron of the L r calculation neurons.
  • Each single layer perceptron network includes a respective calculation neuron of the Q calculation neurons.
  • the equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m- ⁇ layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron network; and (iv) constructing (28130) the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating input of each equivalent pyramid-like sub-network for the Q single layer perceptron networks to form an input vector with Q*K l J inputs.
  • the system of equations includes K j equations p jTn — ⁇ with S variables, and S is computed using the equation S K Lj y Nm-i ⁇ N-1-) ) ⁇
  • the method includes transforming two layers to trapezium-based network.
  • the neural network topology includes (28140) K inputs, a layer L p with K neurons, a layer L n with L neurons, and a weight matrix W E R LxK , where R is the set of real numbers, each neuron of the layer L p is connected to each neuron of the layer L forum, and each neuron of the layer L n performs an activation function F, such that output of the layer L n is computed using the equation Y 0 F(W. x) for an input x.
  • computing (28148) the weight matrix for the equivalent sparsely connected network includes generating (2850) a sparse weight matrices W 0 and W h by solving a matrix equation W 0 .
  • W h W that includes K ⁇ L equations in K ⁇ N 0 + L ⁇ N j variables, so that the total output of the layer LA 0 is calculated using the equation Y 0 F(W 0 . W h .x).
  • the sparse weight matrix W 0 E R KxM represents connections between the layers LA P and IA / and the sparse weight matrix W h E R MxL represents connections between the layers LA / , and LA 0 ,.
  • performing the trapezium transformation further includes: in accordance with a determination that K ⁇ L 3 L N j + K ⁇ N 0 : (i) splitting (28154) the layer L p to obtain a sub-layer L pi with K’ neurons and a sub-layer L P 2 with (K - K') neurons such that K’ L 3 L N, + K’ N 0 (ii) for the sub layer pi with f’ neurons, performing (28156) the constructing, and generating steps; and (iii) for the sub-layer L P2 with K K’ neurons, recursively performing (28158) the splitting, constructing, and generating steps.
  • the method includes transforming multilayer perceptron to trapezium-based network.
  • the neural network topology includes (28160) a multilayer perceptron network, the method further includes, for each pair of consecutive layers of the multilayer perceptron network, iteratively performing (28162) the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network.
  • the method includes transforming recurrent neural network to trapezium-based network.
  • the neural network topology includes (28164) a recurrent neural network (RNN) that includes (i) a calculation of linear combination for two fully connected layers, (ii) element-wise addition, and (iii) a non-linear function calculation.
  • the method further includes performing (28166) the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network, for (i) the two fully connected layers, and (ii) the non-linear function calculation.
  • Element-wise addition is a common operation that can be implemented in networks of any structure, examples of which are provided above.
  • Non-linear function calculation is a neuron-wise operation that is independent of the No and Ni restrictions, and are usually calculated with ‘sigmoid’ or ‘tanh’ block on each neuron separately.
  • the neural network topology includes (28168) a long short-term memory (LSTM) network or a gated recurrent unit (GRU) network that includes (i) a calculation of linear combination for a plurality of fully connected layers, (ii) element-wise addition, (iii) a Hadamard product, and (iv) a plurality of non-linear function calculations (sigmoid and hyperbolic tangent operations).
  • the method further includes performing (28170) the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network, for (i) the plurality of fully connected layers, and (ii) the plurality of non-linear function calculations.
  • Element-wise addition and Hadamard products are common operations that can be implemented in networks of any structure described above.
  • the neural network topology includes (28172) a convolutional neural network (CNN) that includes (i) a plurality of partially connected layers (e.g., sequence of convolutional and pooling layers; each pooling layer is assumed to be a convolutional later with stride larger than 1) and (ii) one or more fully-connected layers (the sequence ends in the fully-connected layers).
  • CNN convolutional neural network
  • the method further includes (i) transforming (28174) the plurality of partially connected layers to equivalent fully- connected layers by inserting missing connections with zero weights; and for each pair of consecutive layers of the equivalent fully-connected layers and the one or more fully- connected layers, iteratively performing (28176) the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network.
  • the neural network topology includes (28178) K inputs, L output neurons, and a weight matrix U E R LxK , where R is the set of real numbers, each output neuron performs an activation function F.
  • Each neuron in the pyramid neural network performs identity function; and (iv) constructing (28188) a trapezium neural network with N p inputs and L outputs. Each neuron in the last layer of the trapezium neural network performs the activation function F and all other neurons perform identity function.
  • computing (28190) the weight matrix for the equivalent sparsely connected network includes: (i) generating (28192) weights for the pyramid neural network including (i) setting weights of every neuron i of the first layer of the pyramid neural network according to following rule: for all weights j of the neuron except k,; and (ii) setting all other weights of the pyramid neural network to 1; and (ii) generating(28194) weights for the trapezium neural network including (i) setting weights of each neuron i of the first layer of the trapezium neural network ⁇ ') i ifc .
  • transforming (28198) the neural network topology to the equivalent sparsely connected network of analog components includes: for each layer j (28200) of the S layers of the multilayer perceptron, constructing (28202) a respective pyramid-trapezium network PTNNX j by performing the approximation transformation to a respective single layer perceptron consisting of L j ⁇ inputs, L j output neurons, and a weight matrix U j ⁇ ; and (ii) constructing (28204) the equivalent sparsely connected network by stacking each pyramid trapezium network (e.g., output of a pyramid trapezium network PTNNXj-1 is set as an input for PTNNXj).
  • each pyramid trapezium network e.g., output of a pyramid trapezium network PTNNXj-1 is set as an input for PTNNXj.
  • the method further includes generating (28016) a schematic model for implementing the equivalent sparsely connected network utilizing the weight matrix.
  • Figures 29A-29F show a flowchart of a method 2900 for hardware realization (2902) of neural networks according to hardware design constraints, according to some implementations.
  • the method is performed (2904) at the computing device 200 (e.g., using the weight quantization module 238) having one or more processors 202, and memory 214 storing one or more programs configured for execution by the one or more processors 202.
  • the method includes obtaining (2906) a neural network topology (e.g., the topology 224) and weights (e.g., the weights 222) of a trained neural network (e.g., the networks 220).
  • weight quantization is performed during training.
  • the trained neural network is trained (2908) so that each layer of the neural network topology has quantized weights (e.g., a particular value from a list of discrete values; e.g., each layer has only 3 weight values of +1, 0, -1).
  • the method also includes transforming (2910) the neural network topology (e.g., using the neural network transformation module 226) to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors.
  • Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons.
  • the method also includes computing (2912) a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection.
  • the method also includes generating (2914) a resistance matrix for the weight matrix.
  • Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value.
  • generating the resistance matrix for the weight matrix includes a simplified gradient-descent based iterative method to find a resistor set.
  • generating the resistance matrix for the weight matrix includes: (i) obtaining (2916) a predetermined range of possible resistance values ⁇ R m in , Rmax ⁇ and selecting an initial base resistance value R baS e within the predetermined range.
  • the range and the base resistance are selected according to values of elements of the weight matrix; the values are determined by the manufacturing process; ranges - resistors that can be actually manufactured; large resistors are not preferred; quantization of what can be actually manufactured.
  • the predetermined range of possible resistance values includes (2918) resistances according to nominal series E24 in the range 100 KW to 1 MW; (ii) selecting (2920) a limited length set of resistance values, within the predetermined range, that provide most uniform distribution of possible weights nn ⁇ ; ⁇ within the range [ base> base ⁇ for all combinations of ⁇ R t , R j ⁇ within the limited length set of resistance values.
  • R + R is the closest resistor set value to R base * W, max .
  • R ⁇ are chosen (2924) independently for each layer of the equivalent analog network.
  • R + and R ⁇ are chosen (2926) independently for each analog neuron of the equivalent analog network; and (iv) for each element of the weight matrix, selecting (2928) a respective first resistance value R ⁇ and a respective second resistance value R 2 that
  • a first one or more weights of the weight matrix and a first one or more inputs represent (2930) one or more connections to a first operational amplifier of the equivalent analog network.
  • the method further includes: prior to generating (2932) the resistance matrix, (i) modifying (2934) the first one or more weights by a first value (e.g., dividing the first one or more weights by the first value to reduce weight range, or multiplying the first one or more weights by the first value to increase weight range); and (ii) configuring (2936) the first operational amplifier to multiply, by the first value, a linear combination of the first one or more weights and the first one or more inputs, before performing an activation function.
  • Some implementations perform the weight reduction so as to change multiplication factor of one or more operational amplifiers.
  • the resistor values set produce weights of some range, and in some parts of this range the error will be higher than in others.
  • these resistors can produce weights [-3; -0.75; 0; 0.75; 3]
  • the first layer of a neural network has weights of ⁇ 0, 9 ⁇ and the second layer has weights of ⁇ 0, 1 ⁇
  • some implementations divide the first layer’s weights by 3 and multiply the second layer’s weights by 3 to reduce overall error.
  • Some implementations consider restricting weight values during training, by adjusting loss function (e.g., using 11 or 12 regularizer), so that resulting network does not have weights too large for the resistor set.
  • the method further includes restricting weights to intervals.
  • the method further includes obtaining (2938) a predetermined range of weights, and updating (2940) the weight matrix according to the predetermined range of weights such that the equivalent analog network produces similar output as the trained neural network for same input.
  • the method further includes reducing weight sensitivity of network.
  • the method further includes retraining (2942) the trained neural network to reduce sensitivity to errors in the weights or the resistance values that cause the equivalent analog network to produce different output compared to the trained neural network.
  • some implementations include additional training for an already trained neural network in order to give it less sensitivity to small randomly distributed weight errors. Quantization and resistor manufacturing produce small weight errors.
  • Some implementations transform networks so that the resultant network is less sensitive to each particular weight value. In some implementations, this is performed by adding a small relative random value to each signal in at least some of the layers during training (e.g., similar to a dropout layer).
  • some implementations include reducing weight distribution range. Some implementations include retraining (2944) the trained neural network so as to minimize weight in any layer that are more than mean absolute weight for that layer by larger than a predetermined threshold. Some implementations perform this step via retraining.
  • Example penalty function include a sum over all layers (e.g., A * max(abs(w)) / mean(abs(w)), where max and mean are calculated over a layer. Another example include order of magnitude higher and above. In some implementations, this function impacts weight quantization and network weight sensitivity. For e.g., small relative changes of weights due to quantization might cause high output error.
  • Example techniques include introducing some penalty functions during training that penalize network when it has such weight outcasts.
  • Figures 31A-31Q show a flowchart of a method 3100 for fabricating an integrated circuit 3102 that includes an analog network of analog components, according to some implementations.
  • the method is performed at the computing device 200 (e.g., using the IC fabrication module 258) having one or more processors 202, and memory 214 storing one or more programs configured for execution by the one or more processors 202.
  • the method includes obtaining (3104) a neural network topology and weights of a trained neural network.
  • the method also includes transforming (3106) the neural network topology (e.g., using the neural network transformation module 226) to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors (for recurrent neural networks, also use signal delay lines, multipliers, Tanh analog block, Sigmoid Analog Block).
  • Each operational amplifier represents a respective analog neuron
  • each resistor represents a respective connection between a respective first analog neuron and a respective second analog neuron.
  • the method also includes computing (3108) a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection.
  • the method also includes generating (3110) a resistance matrix for the weight matrix ach element of the resistance matrix corresponds to a respective weight of the weight matrix.
  • the method also includes generating (3112) one or more lithographic masks (e.g., generating the masks 250 and/or 252 using the mask generation module 248) for fabricating a circuit implementing the equivalent analog network of analog components based on the resistance matrix, and fabricating (3114) the circuit (e.g., the ICs 262) based on the one or more lithographic masks using a lithographic process.
  • generating (3112) one or more lithographic masks e.g., generating the masks 250 and/or 252 using the mask generation module 248) for fabricating a circuit implementing the equivalent analog network of analog components based on the resistance matrix
  • fabricating (3114) the circuit e.g., the ICs 262
  • the integrated circuit further includes one or more digital to analog converters (3116) (e.g., the DAC converters 260) configured to generate analog input for the equivalent analog network of analog components based on one or more digital signals (e.g., signals from one or more CCD/CMOS image sensors).
  • one or more digital to analog converters (3116) e.g., the DAC converters 260
  • the integrated circuit further includes one or more digital to analog converters (3116) (e.g., the DAC converters 260) configured to generate analog input for the equivalent analog network of analog components based on one or more digital signals (e.g., signals from one or more CCD/CMOS image sensors).
  • the integrated circuit further includes an analog signal sampling module (3118) configured to process 1- dimensional or 2-dimensional analog inputs with a sampling frequency based on number of inferences of the integrated circuit (number of inferences for the IC is determined by product Spec - we know sampling rate from Neural Network operation and exact task the chip is intended to solve).
  • an analog signal sampling module (3118) configured to process 1- dimensional or 2-dimensional analog inputs with a sampling frequency based on number of inferences of the integrated circuit (number of inferences for the IC is determined by product Spec - we know sampling rate from Neural Network operation and exact task the chip is intended to solve).
  • the integrated circuit further includes a voltage converter module (3120) to scale down or scale up analog signals to match operational range of the plurality of operational amplifiers.
  • the integrated circuit further includes a tact signal processing module (3122) configured to process one or more frames obtained from a CCD camera.
  • a tact signal processing module (3122) configured to process one or more frames obtained from a CCD camera.
  • the trained neural network is a long short-term memory (LSTM) network
  • the integrated circuit further includes one or more clock modules to synchronize signal tacts and to allow time series processing.
  • LSTM long short-term memory
  • the integrated circuit further includes one or more analog to digital converters (3126) (e.g., the ADC converters 260) configured to generate digital signal based on output of the equivalent analog network of analog components.
  • analog to digital converters e.g., the ADC converters 260
  • the integrated circuit includes one or more signal processing modules (3128) configured to process 1-dimensional or 2-dimensional analog signals obtained from edge applications.
  • the trained neural network is trained (3130), using training datasets containing signals of arrays of gas sensors (e.g., 2 to 25 sensors) on different gas mixture, for selective sensing of different gases in a gas mixture containing predetermined amounts of gases to be detected (in other words, the operation of trained chip is used to determine each of known to neural network gases in the gas mixture individually, despite the presence of other gases in the mixture).
  • the neural network topology is a 1-Dimensional Deep Convolutional Neural network (1D-DCNN) designed for detecting 3 binary gas components based on measurements by 16 gas sensors, and includes (3132) 16 sensor-wise 1-D convolutional blocks, 3 shared or common 1-D convolutional blocks and 3 dense layers.
  • the equivalent analog network includes (3134): (i) a maximum of 100 input and output connections per analog neuron, (ii) delay blocks to produce delay by any number of time steps, (iii) a signal limit of 5, (iv) 15 layers, (v) approximately 100,000 analog neurons, and (vi) approximately 4,900,000 connections.
  • the trained neural network is trained (3136), using training datasets containing thermal aging time series data for different MOSFETs (e.g., NASA MOSFET dataset that contains thermal aging time series for 42 different MOSFETs; data is sampled every 400 ms and typically several hours of data for each device), for predicting remaining useful life (RUL) of a MOSFET device.
  • the neural network topology includes (3138) 4 LSTM layers with 64 neurons in each layer, followed by two dense layers with 64 neurons and 1 neuron, respectively.
  • the equivalent analog network includes (3140): (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 18 layers, (iv) between 3,000 and 3,200 analog neurons (e.g., 3137 analog neurons), and (v) between 123,000 and 124,000 connections (e.g., 123,200 connections).
  • the trained neural network is trained (3142), using training datasets containing time series data including discharge and temperature data during continuous usage of different commercially available Li-Ion batteries (e.g., NASA battery usage dataset; the dataset presents data of continuous usage of 6 commercially available Li-Ion batteries; network operation is based on analysis of discharge curve of battery ), for monitoring state of health (SOH) and state of charge (SOC) of Lithium Ion batteries to use in battery management systems (BMS).
  • the neural network topology includes (3144) an input layer, 2 LSTM layers with 64 neurons in each layer, followed by an output dense layer with 2 neurons for generating SOC and SOH values.
  • the equivalent analog network includes (3146): (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 9 layers, (iv) between 1,200 and 1,300 analog neurons (e.g., 1271 analog neurons), and (v) between 51,000 and 52,000 connections (e.g., 51,776 connections).
  • the trained neural network is trained (3148), using training datasets containing time series data including discharge and temperature data during continuous usage of different commercially available Li-Ion batteries (e.g., NASA battery usage dataset; the dataset presents data of continuous usage of 6 commercially available Li-Ion batteries; network operation is based on analysis of discharge curve of battery ), for monitoring state of health (SOH) of Lithium Ion batteries to use in battery management systems (BMS).
  • the neural network topology includes (3150) an input layer with 18 neurons, a simple recurrent layer with 100 neurons, and a dense layer with 1 neuron.
  • the equivalent analog network includes (3152): (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 4 layers, (iv) between 200 and 300 analog neurons (e.g., 201 analog neurons), and (v) between 2,200 and 2,400 connections (e.g., 2,300 connections).
  • the trained neural network is trained (3154), using training datasets containing speech commands (e.g., Google Speech Commands Dataset), for identifying voice commands (e.g., 10 short spoken keywords, including “yes”, “no”, “up”, “down”, “left”, “right”, “on”, “off’, “stop”, “go”).
  • the neural network topology is (3156) a Depthwise Separable Convolutional Neural Network (DS-CNN) layer with 1 neuron.
  • DS-CNN Depthwise Separable Convolutional Neural Network
  • the equivalent analog network includes (3158): (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 13 layers, (iv) approximately 72,000 analog neurons, and (v) approximately 2.6 million connections.
  • the trained neural network is trained (3160), using training datasets containing photoplethysmography (PPG) data, accelerometer data, temperature data, and electrodermal response signal data for different individuals performing various physical activities for a predetermined period of times and reference heart rate data obtained from ECG sensor (e.g., PPG data from the PPG-Dalia dataset (CHECK LICENSE). Data is collected for 15 individuals performing various physical activities during 1-4 hours each.
  • Wrist-based sensor data contains PPG, 3-axis accelerometer, temperature and electrodermal response signals sampled from 4 to 64 Hz, and a reference heartrate data obtained from ECG sensor with sampling around 2 Hz.
  • Original data was split into sequences of 1000 timesteps (around 15 seconds), with a shift of 500 timesteps, thus getting 16541 samples total.
  • Dataset was split into 13233 training samples and 3308 test samples), for determining pulse rate during physical exercises (e.g., jogging, fitness exercises, climbing stairs) based on PPG sensor data and 3-axis accelerometer data.
  • the neural network topology includes (3162) two ConvlD layers each with 16 filters and a kernel of 20, performing time series convolution, two LSTM layers each with 16 neurons, and two dense layers with 16 neurons and 1 neuron, respectively.
  • the equivalent analog network includes (3164): (i) delay blocks to produce any number of time steps, (ii) a maximum of 100 input and output connections per analog neuron, (iii) a signal limit of 5, (iv) 16 layers, (v) between 700 and 800 analog neurons (e.g., 713 analog neurons), and (vi) between 12,000 and 12,500 connections (e.g., 12,072 connections).
  • the trained neural network is trained (3166) to classify different objects (e.g., humans, cars, cyclists, scooters) based on pulsed Doppler radar signal (remove clutter and provide noise to Doppler radar signal), and the neural network topology includes (3168) multi-scale LSTM neural network.
  • the trained neural network is trained (3170) to perform human activity type recognition (e.g., walking, running, sitting, climbing stairs, exercising, activity tracking), based on inertial sensor data (e.g., 3-axes accelerometers, magnetometers, or gyroscope data, from fitness tracking devices, smart watches or mobile phones; 3-axis accelerometer data as input, sampled at up to 96Hz frequency.
  • inertial sensor data e.g., 3-axes accelerometers, magnetometers, or gyroscope data, from fitness tracking devices, smart watches or mobile phones; 3-axis accelerometer data as input, sampled at up to 96Hz frequency.
  • Network was trained on 3 different publicly available datasets, presenting such activities as “open then close the dishwasher”, “drink while standing”, “close left hand door”, “jogging”, “walking”, “ascending stairs” etc.).
  • the neural network topology includes (3172) three channel-wise convolutional networks each with a convolutional layer of 12 filters and a kernel dimension of 64, and each followed by a max pooling layer, and two common dense layers of 1024 neurons and N neurons, respectively, where N is a number of classes.
  • the equivalent analog network includes (3174): (i) delay blocks to produce any number of time steps, (ii) a maximum of 100 input and output connections per analog neuron, (iii) an output layer of 10 analog neurons, (iv) signal limit of 5, (v) 10 layers, (vi) between 1,200 and 1,300 analog neurons (e.g., 1296 analog neurons), and (vi) between 20,000 and 21,000 connections (e.g., 20,022 connections).
  • the trained neural network is further trained (3176) to detect abnormal patterns of human activity based on accelerometer data that is merged with heart rate data using a convolution operation (so as to detect pre-stroke or pre heart attack states or signal in case of sudden abnormal patterns, caused by injuries or malfunction due to medical reasons, like epilepsy, etc).
  • Some implementations include components that are not integrated into the chip (i.e., these are external elements, connected to the chip) selected from the group consisting of: voice recognition, video signal processing, image sensing, temperature sensing, pressure sensing, radar processing, LIDAR processing, battery management, MOSFET circuits current and voltage, accelerometers, gyroscopes, magnetic sensors, heart rate sensors, gas sensors, volume sensors, liquid level sensors, GPS satellite signal, human body conductance sensor, gas flow sensor, concentration sensor, pH meter, and IR vision sensors.
  • a neuromorphic IC is manufactured according to the processes described above.
  • the neuromorphic IC is based on a Deep Convolutional Neural Network trained for selective sensing of different gases in the gas mixture containing some amounts of gases to be detected.
  • the Deep Convolutional Neural Network is trained using training datasets, containing signals of arrays of gas sensors (e.g., 2 to 25 sensors) in response to different gas mixtures.
  • the integrated circuit (or the chip manufactured according to the techniques described herein) can be used to determine one or more known gases in the gas mixture, despite the presence of other gases in the mixture.
  • the trained neural network is a Multi-label 1D- DCNN network used for Mixture Gases Classification.
  • the network is designed for detecting 3 binary gas components based on measurements by 16 gas sensors.
  • the 1D-DCNN includes sensor-wise ID convolutional block (16 such blocks), 3 common ID convolutional blocks, and 3 Dense layers.
  • the 1D-DCNN network performance for this task is 96.3%.
  • the resulting T-network has the following properties: 15 layers, approximately 100,000 analog neurons, approximately 4,900,000 connections.
  • MOSFET on-resistance degradation due to thermal stress is a well-known serious problem in power electronics.
  • MOSFET device temperature changes over a short period of time. This temperature sweeps produce thermal degradation of a device, as a result of which the device might exhibit exponential. This effect is typically studied by power cycling that produces temperature gradients, which cause MOSFET degradation.
  • a neuromorphic IC is manufactured according to the processes described above.
  • the neuromorphic IC is based on a network discussed in the article titled “Real-time Deep Learning at the Edge for Scalable Reliability Modeling of SI- MO SFET Power Electronics Converters” for predicting remaining useful life (RUL) of a MOSFET device.
  • the neural network can be used to determine Remaining Useful Life (RUL) of a device, with an accuracy over 80%.
  • the network is trained on NASA MOSFET Dataset which contains thermal aging timeseries for 42 different MOSFETs. Data is sampled every 400 ms and typically includes several hours of data for each device.
  • the network contains 4 LSTM layers of 64 neurons each, followed by 2 Dense layers of 64 and 1 neurons.
  • the network is T-transformed with following parameters: maximum input and output connections per neuron is 100; signal limit of 5, and the resulting T-network had following properties: 18 layers, approximately 3,000 neurons (e.g., 137 neurons), and approximately 120,000 connections (e.g., 123200 connections).
  • a neuromorphic IC is manufactured according to the processes described above.
  • the neuromorphic IC can be used for predictive analytics of Lithium Ion batteries to use in Battery Management Systems (BMS).
  • BMS device typically presents such functions as overcharge and over-discharge protection, monitoring State of Health (SOH) and State of Charge (SOC), and load balancing for several cells.
  • SOH and SOC monitoring normally requires digital data processor, which adds to the cost of the device and consumes power.
  • the Integration Circuit is used to obtain precise SOC and SOH data without implementing digital data processor on the device.
  • the Integrated Circuit determines SOC with over 99% accuracy and determines SOH with over 98% accuracy.
  • network operation is based on analysis of the discharge curve of the battery, as well as temperature, and/or data is presented as a time series.
  • Some implementations use data from NASA Battery Usage dataset. The dataset presents data of continuous usage of 6 commercially available Li-Ion batteries.
  • the network includes an input layer, 2 LSTM layers of 64 neurons each, and an output dense layer of 2 neurons (SOC and SOH values).
  • the resulting T-network include the following properties: 9 layers, approximately 1,200 neurons (e.g., 1,271 neurons), and approximately 50,000 connections (e.g., 51,776 connections).
  • the network operation is based on analysis of the discharge curve of the battery, as well as temperature. The network is trained using Network IndRnn disclosed in the paper titled “State-of-Health Estimation of Li-ion Batteries inElectric Vehicle Using IndRNN under VariableLoad Condition” designed for processing data from NASA Battery Usage dataset.
  • the dataset presents data of continuous usage of 6 commercially available Li-Ion batteries.
  • the IndRnn network contains an input layer with 18 neurons, a simple recurrent layer of 100 neurons and a dense layer of 1 neuron.
  • the resulting T-network had following properties: 4 layers, approximately 200 neurons (e.g., 201 neurons), and approximately 2,000 connections (e.g., 2,300 connections).
  • Some implementations output only SOH with an estimation error of 1.3%.
  • the SOC is obtained similar to how the SOH is obtained.
  • a neuromorphic IC is manufactured according to the processes described above.
  • the neuromorphic IC can be used for keyword spotting.
  • the input network is a neural network with 2-D Convolutional and 2-D Depthwise Convolutional layers, with input audio mel-spectrogram of size 49 times 10.
  • the network includes 5 convolutional layers, 4 depthwise convolutional layers, an average pooling layer, and a final dense layer.
  • the networks are pre-trained to recognize 10 short spoken keywords (yes”, “no”, “up”, “down”, “left”, “right”, “on”, “off, “stop”, “go”) from Google Speech Commands Dataset, with a recognition accuracy of 94.4%.
  • the Integration Circuit is manufactured based on Depthwise Separable Convolutional Neural Network (DS-CNN) for the voice command identification.
  • DS-CNN Depthwise Separable Convolutional Neural Network
  • the resulting T-network had following properties: 13 layers, approximately 72,000 neurons, and approximately 2.6 million connections.
  • a keyword spotting network is transformed to a T-network, according to some implementations.
  • the network is a neural network of 2-D Convolutional and 2-D Depthwise Convolutional layers, with input audio spectrogram of size 49x10.
  • Network consists of 5 convolutional layers, 4 depthwise convolutional layers, average pooling layer and final dense layer.
  • Network is pre-trained to recognize 10 short spoken keywords (yes”, “no”, “up”, “down”, “left”, “right”, “on”, “off 1 , "stop”, “go”) from Google Speech Commands Dataset https://ai.googleblog.com/2017/08/launching-speech- commands-dataset.html.
  • There are 2 additional classes which correspond to ‘silence’ and ‘unknown’.
  • Network output is a softmax of length 12.
  • the trained neural network (input to the transformation) had a recognition accuracy of 94.4%, according to some implementations.
  • each convolutional layer is followed with BatchNorm layer and ReLU layer, and ReLU activations are unbounded, and included around 2.5 million multiply-add operations.
  • Resulting T-network had 12 layers including an Input layer, approximately 72,000 neurons and approximately 2.5 million connections.
  • Figures 26A-26K show example histograms 2600 for absolute weights for the layers 1 through 11, respectively, according to some implementations.
  • the weight distribution histogram (for absolute weights) was calculated for each layer.
  • the dashed lines in the charts correspond to a mean absolute weight value for the respective layer.
  • the average output absolute error (calculated over test set) of converted network vs original is calculated to be 4. le-9.
  • some implementations use a nominal set of 30 resistors [0.001, 0.003, 0.01, 0.03, 0.1, 0.324, 0.353, 0.436, 0.508, 0.542, 0.544, 0.596, 0.73, 0.767, 0.914, 0.985, 0.989, 1.043, 1.101, 1.149, 1.157, 1.253, 1.329, 1.432, 1.501, 1.597, 1.896, 2.233, 2.582, 2.844]
  • Some implementations select R- and R+ values (see description above) separately for each layer. For each layer, some implementations select a value which delivers most weight accuracy. In some implementations, subsequently all the weights (including bias) in the T-network are quantized (e.g., set to the closest value which can be achieved with the input or chosen resistors).
  • Output layer is a dense layer that does not have ReLU activation.
  • the layer has softmax activation which is not implemented in T-conversion and is left for digital part, according to some implementations. Some implementations perform no additional conversion.
  • PPG is an optically obtained plethysmogram that can be used to detect blood volume changes in the microvascular bed of tissue.
  • a PPG is often obtained by using a pulse oximeter which illuminates the skin and measures changes in light absorption.
  • PPG is often processed to determine heart rate in devices, such as fitness trackers.
  • Deriving heart rate (HR) from PPG signal is an essential task in edge devices computing.
  • PPG data obtained from device located on wrist usually allows to obtain reliable heartrate only when the device is stable. If a person is involved in physical exercise, obtaining heartrate from PPG data produces poor results unless combined with inertial sensor data.
  • an Integrated Circuit based on combination of Convolutional Neural Network and LSTM layers, can be used to precisely determine the pulse rate, basing on the data from photoplethysmography (PPG) sensor and 3-axis accelerometer.
  • the integrated circuit can be used to suppress motion artifacts of PPG data and to determine the pulse rate during physical exercise, such as jogging, fitness exercises, and climbing stairs, with an accuracy exceeding 90%
  • the input network is trained with PPG data from the PPG-Dalia dataset. Data is collected for 15 individuals performing various physical activities for a predetermined duration (e.g., 1-4 hours each).
  • the training data included wrist-based sensor data contains PPG, 3-axis accelerometer, temperature and electro-dermal response signals sampled from 4 to 64 Hz, and a reference heartrate data obtained from an ECG sensor with sampling around 2 Hz.
  • the original data was split into sequences of 1000 time steps (around 15 seconds), with a shift of 500 time steps, thus producing 16541 samples total.
  • the dataset was split into 13233 training samples and 3308 test samples.
  • the input network included 2 ConvlD layers with 16 filters each, performing time series convolution, 2 LSTM layers of 16 neurons each, and 2 dense layers of 16 and 1 neurons.
  • the network produces MSE error of less than 6 beats per minute over the test set.
  • the resulting T-network had following properties: 15 layers, approximately 700 neurons (e.g., 713 neurons), and approximately 12,000 connections (e.g., 12072 connections).
  • the delay block has an external cycle timer (e.g., a digital timer) which activates the delay block with a constant period of time dt.
  • This activation produces an output of x(t-dt) where x(t) is input signal of delay block.
  • Such activation frequency can, for instance, correspond to network input signal frequency (e.g., output frequency of analog sensors processed by a T-converted network).
  • all delay blocks are activated simultaneously with the same activation signal. Some blocks can be activated simultaneously on one frequency, and other blocks can be activated on another frequency. In some implementations, these frequencies have common multiplier, and signals are synchronized.
  • multiple delay blocks are used over one signal producing additive time shift. Examples of delay blocks are described above in reference to Figure 13B shows two examples of delay blocks, according to some implementations.
  • the network for processing PPG data uses one or more LSTM neurons, according to some implementations. Examples of LSTM neuron implementations are described above in reference to Figure 13 A, according to some implementations.
  • the network also uses ConvlD, a convolution performed over time coordinate. Examples of ConvlD implementations are described above in reference to Figures 15A and 15B, according to some implementations.
  • PPG is an optically obtained plethysmogram that can be used to detect blood volume changes in the microvascular bed of tissue.
  • a PPG is often obtained by using a pulse oximeter which illuminates the skin and measures changes in light absorption.
  • PPG is often processed to determine heart rate in devices such as fitness trackers.
  • Deriving heart rate (HR) from PPG signal is an essential task in edge devices computing.
  • Some implementations use PPG data from the Capnobase PPG dataset.
  • the data contains raw PPG signal for 42 individuals of 8 min duration each, sampling 300 samples per second, and a reference heartrate data obtained from ECG sensor with sampling around
  • the input trained neural network NN-based allows for 1-3% accuracy in obtaining heartrate (HR) from PPG data.
  • This section describes a relatively simple neural network in order to demonstrate how T-conversion and analog processing can deal with this task. This description is provided as an example, according to some implementations.
  • dataset is split into 4,670 training samples and 1,168 test samples.
  • the network included: 1 ConvlD layer with 16 filters and kernel of 20,
  • the input network was T-transformed with following parameters: delay block with periods of 1, 5 and 10 time steps, and the following properties: 17 layers, 15,448 connections, and 329 neurons (OP3 neurons and multiplier blocks, not counting delay blocks).
  • an Integration Circuit is manufactured, based on a multi-scale LSTM neural network, that can be used to classify the objects, based on pulse Doppler Radar signal.
  • the IC can be used to classify different objects, like humans, cars, cyclists, scooters, based on Doppler radar signal, removes clutter, and provides the noise to Doppler radar signal.
  • the accuracy of classification of object with multi-scale LSTM network exceeded 90%.
  • a neuromorphic Integrated Circuit is manufactured, and can be used for human activity type recognition based on multi-channel convolutional neural networks, which have input signals from 3-axes accelerometers and possibly magnetometers and/or gyroscopes of fitness tracking devices, smart watches or mobile phones.
  • the multi-channel convolutional neural network can be used to distinguish between different types of human activities, such as walking, running, sitting, climbing stairs, exercising and can be used for activity tracking.
  • the IC can be used for detection of abnormal patterns of human activity, based on accelerometer data, convolutionally merged with heart rate data. Such IC can detect pre-stroke or pre heart attack states or signal in case of sudden abnormal patterns, caused by injuries or malfunction due to medical reasons, like epilepsy and others, according to some implementations.
  • the IC is based on a channel-wise ID convolutional network discussed in the article “Convolutional Neural Networks for Human Activity Recognition using Mobile Sensors.”
  • this network accepts 3-axis accelerometer data as input, sampled at up to 96Hz frequency.
  • the network is trained on 3 different publicly available datasets, presenting such activities as “open then close the dishwasher”, “drink while standing”, “close left hand door”, “jogging”, “walking”, “ascending stairs,” etc.
  • the network included 3 channel-wise Conv networks with Conv layer of 12 filters and kernel of 64, followed by MaxPooling(4) layer each, and 2 common Dense layers of 1024 and N neurons respectively, where N is a number of classes.
  • the activity classification was performed with a low error rate (e.g., 3.12% error).
  • the resulting T-network had following properties: 10 layers, approximately 1,200 neurons (e.g., 1296 neurons), and approximately 20,000 connections (e.g., 20022 connections).
  • MobileNet v.l An example transformation of MobileNet v.l into an equivalent analog network is described herein, according to some implementations.
  • single analog neurons are generated, then converted into SPICE schematics with a transformation of weights from MobileNet into resistor values.
  • MobileNet vl architecture is depicted in the Table shown in Figure 34.
  • the first column 3402 corresponds to type of layer and stride
  • the second column 3404 corresponds to filter shape for the corresponding layer
  • the third column 3406 corresponds to input size for the corresponding layer.
  • the network consists of 27 convolutional layers, 1 dense layer, and has around 600 million multiply- accumulate operations for a 224x224x3 input image.
  • Output values are the result of softmax activation function which means the values are distributed in the range [0, 1] and the sum is 1.
  • the network is pre-trained for CIFAR-10 task (50,000 32x32x3 images divided into 10 non-intersecting classes). Batch normalization layers operate in ‘test’ mode to produce simple linear signal transformation, so the layers are interpreted as weight multiplier + some additional bias.
  • Convolutional, AveragePooling and Dense layers are transformed using the techniques described above, according to some implementations.
  • Softmax activation function is not implemented in transformed network but applied to output of the transformed network (or the equivalent analog network) separately.
  • the resulting transformed network included 30 layers including an input layer, approximately 104,000 analog neurons, and approximately 11 million connections.
  • the average output absolute error (calculated over 100 random samples) of transformed network versus MobileNet v.l was 4.9e-8.
  • output signal on each layer of transformed network is also limited by the value 6.
  • the weights are brought into accordance with a resistor nominal set. Under each nominal set, different weight values are possible. Some implementations use resistor nominal sets e24, e48 and e96, within the range of [0.1 - 1] Mega Ohm. Given that the weight ranges for each layer vary, and for most layers weight values do not exceed 1-2, in order to achieve more weight accuracy, some implementations decrease R- and R+ values. In some implementations, the R- and R+ values are chosen separately for each layer from the set [0.05, 0.1, 0.2, 0.5, 1] Mega Ohm.
  • a value which delivers most weight accuracy is chosen. Then all the weights (including bias) in the transformed network are ‘quantized’, i.e., set to the closest value which can be achieved with used resistors. In some implementations, this reduced transformed network accuracy versus original MobileNet according to the Table shown below. The Table shows mean square error of transformed network, when using different resistor sets, according to some implementations.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Neurology (AREA)
  • Feedback Control In General (AREA)

Abstract

L'invention concerne des systèmes et des procédés de réalisation matérielle analogique de réseaux neuronaux. Le procédé consiste à obtenir une topologie de réseau neuronal et des pondérations d'un réseau neuronal entraîné. Le procédé consiste également à transformer la topologie de réseau neuronal en un réseau analogique équivalent de composants analogiques. Le procédé consiste aussi à calculer une matrice de pondérations pour le réseau analogique équivalent sur la base des pondérations du réseau neuronal entraîné. Chaque élément de la matrice de pondérations représente une connexion respective entre des composants analogiques du réseau analogique équivalent. Le procédé consiste également à générer un modèle schématique pour la mise en œuvre du réseau analogique équivalent sur la base de la matrice de pondérations, y compris à sélectionner des valeurs de composants pour les composants analogiques.
PCT/EP2020/067800 2020-06-25 2020-06-25 Réalisation matérielle analogique de réseaux neuronaux WO2021259482A1 (fr)

Priority Applications (16)

Application Number Priority Date Filing Date Title
PCT/EP2020/067800 WO2021259482A1 (fr) 2020-06-25 2020-06-25 Réalisation matérielle analogique de réseaux neuronaux
US17/189,109 US20210406661A1 (en) 2020-06-25 2021-03-01 Analog Hardware Realization of Neural Networks
US17/196,960 US20210406662A1 (en) 2020-06-25 2021-03-09 Analog hardware realization of trained neural networks for voice clarity
US17/198,198 US20210406663A1 (en) 2020-06-25 2021-03-10 Analog Hardware Realization of Trained Neural Networks
US17/199,373 US20210406664A1 (en) 2020-06-25 2021-03-11 Optimizations for Analog Hardware Realization of Trained Neural Networks
US17/199,422 US20210406666A1 (en) 2020-06-25 2021-03-11 Integrated Circuits for Neural Networks
US17/199,407 US20210406665A1 (en) 2020-06-25 2021-03-11 Analog Hardware Realization of Trained Neural Networks
US17/200,707 US20210406667A1 (en) 2020-06-25 2021-03-12 Systems and Methods for Generating Libraries for Hardware Realization of Neural Networks
US17/200,723 US20220004861A1 (en) 2020-06-25 2021-03-12 Systems and Methods for Optimizing Energy Efficiency of Analog Neuromorphic Circuits
TW110122949A TWI773398B (zh) 2020-06-25 2021-06-23 神經網路的模擬硬體實現
TW111125566A TWI796257B (zh) 2020-06-25 2021-06-23 神經網路的模擬硬體實現
US17/733,932 US11885271B2 (en) 2020-06-25 2022-04-29 Systems and methods for detonation control in spark ignition engines using analog neuromorphic computing hardware
US17/744,565 US20220280072A1 (en) 2020-06-25 2022-05-13 Systems and Methods for Human Activity Recognition Using Analog Neuromorphic Computing Hardware
US17/902,757 US20230081715A1 (en) 2020-06-25 2022-09-02 Neuromorphic Analog Signal Processor for Predictive Maintenance of Machines
US18/093,315 US20230147781A1 (en) 2020-06-25 2023-01-04 Sound Signal Processing Using A Neuromorphic Analog Signal Processor
US18/196,412 US20230367998A1 (en) 2020-06-25 2023-05-11 Hybrid Fixed/Flexible Neural Network Architecture

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/EP2020/067800 WO2021259482A1 (fr) 2020-06-25 2020-06-25 Réalisation matérielle analogique de réseaux neuronaux

Related Child Applications (1)

Application Number Title Priority Date Filing Date
PCT/RU2020/000306 Continuation-In-Part WO2021262023A1 (fr) 2020-06-25 2020-06-25 Réalisation matérielle analogique de réseaux neuronaux

Publications (1)

Publication Number Publication Date
WO2021259482A1 true WO2021259482A1 (fr) 2021-12-30

Family

ID=71409377

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2020/067800 WO2021259482A1 (fr) 2020-06-25 2020-06-25 Réalisation matérielle analogique de réseaux neuronaux

Country Status (1)

Country Link
WO (1) WO2021259482A1 (fr)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220138540A1 (en) * 2020-10-30 2022-05-05 International Business Machines Corporation Integrated circuit with a configurable neuromorphic neuron apparatus for artificial neural networks
CN115019129A (zh) * 2022-06-07 2022-09-06 宁波力斗智能技术有限公司 基于时序成像与图像融合的双有源桥变换器故障诊断方法及系统
CN115099275A (zh) * 2022-06-29 2022-09-23 西南医科大学 一种基于人工神经网络的心律失常诊断模型的训练方法
CN116232781A (zh) * 2022-12-08 2023-06-06 中国联合网络通信集团有限公司 节能方法、装置及存储介质
CN116542203A (zh) * 2023-06-27 2023-08-04 贵州轻工职业技术学院 基于嵌入式与数字化实现穿戴设备的电路模拟方法及装置

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
CADENCE DESIGN SYSTEMS: "From TensorFlow to RTL in three months", YOUTUBE, 14 November 2018 (2018-11-14), pages 1 pp., XP054981600, Retrieved from the Internet <URL:XP054981600> [retrieved on 20210324] *
CADENCE: "Engineering Change Orders", 8 December 2019 (2019-12-08), XP055789511, Retrieved from the Internet <URL:https://web.archive.org/web/20191208162944if_/https://www.cadence.com/content/dam/cadence-www/global/en_US/documents/tools/digital-design-signoff/conformal-eco-designer-ds.pdf> *
CONVOLUTIONAL NEURAL NETWORKS FOR HUMAN ACTIVITY RECOGNITION USING MOBILE SENSORS
REAL-TIME DEEP LEARNING AT THE EDGE FOR SCALABLE RELIABILITY MODELING OF SI-MOSFET POWER ELECTRONICS CONVERTERS

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220138540A1 (en) * 2020-10-30 2022-05-05 International Business Machines Corporation Integrated circuit with a configurable neuromorphic neuron apparatus for artificial neural networks
CN115019129A (zh) * 2022-06-07 2022-09-06 宁波力斗智能技术有限公司 基于时序成像与图像融合的双有源桥变换器故障诊断方法及系统
CN115019129B (zh) * 2022-06-07 2024-03-29 宁波力斗智能技术有限公司 基于时序成像与图像融合的双有源桥变换器故障诊断方法及系统
CN115099275A (zh) * 2022-06-29 2022-09-23 西南医科大学 一种基于人工神经网络的心律失常诊断模型的训练方法
CN116232781A (zh) * 2022-12-08 2023-06-06 中国联合网络通信集团有限公司 节能方法、装置及存储介质
CN116232781B (zh) * 2022-12-08 2024-04-16 中国联合网络通信集团有限公司 节能方法、装置及存储介质
CN116542203A (zh) * 2023-06-27 2023-08-04 贵州轻工职业技术学院 基于嵌入式与数字化实现穿戴设备的电路模拟方法及装置
CN116542203B (zh) * 2023-06-27 2023-09-01 贵州轻工职业技术学院 基于嵌入式与数字化实现穿戴设备的电路模拟方法及装置

Similar Documents

Publication Publication Date Title
US20210406666A1 (en) Integrated Circuits for Neural Networks
WO2021259482A1 (fr) Réalisation matérielle analogique de réseaux neuronaux
EP3963514A1 (fr) Réalisation matérielle analogique de réseaux neuronaux
WO2022191879A1 (fr) Réalisation matérielle analogique de réseaux neuronaux entraînés pour clarté vocale
Jain et al. RxNN: A framework for evaluating deep neural networks on resistive crossbars
Gaikwad et al. Efficient FPGA implementation of multilayer perceptron for real-time human activity classification
US20220280072A1 (en) Systems and Methods for Human Activity Recognition Using Analog Neuromorphic Computing Hardware
US20230081715A1 (en) Neuromorphic Analog Signal Processor for Predictive Maintenance of Machines
WO2024049998A1 (fr) Processeur de signal analogique neuromorphique pour la maintenance prédictive de machines
WO2023167607A1 (fr) Systèmes et procédés de reconnaissance d&#39;activité humaine
US11885271B2 (en) Systems and methods for detonation control in spark ignition engines using analog neuromorphic computing hardware
US20210406662A1 (en) Analog hardware realization of trained neural networks for voice clarity
WO2023220437A1 (fr) Systèmes et procédés de reconnaissance d&#39;activité humaine à l&#39;aide d&#39;un matériel informatique neuromorphique analogique
RU2796649C2 (ru) Аналоговая аппаратная реализация нейронных сетей
US11823037B1 (en) Optocoupler-based flexible weights in neuromorphic analog signal processors
US20240005140A1 (en) Quantization Algorithms for Analog Hardware Realization of Neural Networks
US20230147781A1 (en) Sound Signal Processing Using A Neuromorphic Analog Signal Processor
US20240202513A1 (en) Compact CMOS Spiking Neuron Circuit that works with an Analog Memory-Based Synaptic Array
Abdel-Aty-Zohdy et al. A recurrent dynamic neural network approach and implementation, for noise-contaminated signal representation
Atibi et al. Parallel and mixed hardware implementation of artificial neuron network on the FPGA platform
Simpkins Design, modeling, and simulation of a compact optoelectronic neural coprocessor
Zhao Spike processing circuit design for neuromorphic computing
Thakur Stochastic Electronics for Neuromorphic Systems
Janke Overcoming Noise and Variations In Low-Precision Neural Networks.
Briscoe Interacting neural modules

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20735515

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20735515

Country of ref document: EP

Kind code of ref document: A1