US11562240B2 - Efficient tile mapping for row-by-row convolutional neural network mapping for analog artificial intelligence network inference - Google Patents

Efficient tile mapping for row-by-row convolutional neural network mapping for analog artificial intelligence network inference Download PDF

Info

Publication number
US11562240B2
US11562240B2 US16/884,128 US202016884128A US11562240B2 US 11562240 B2 US11562240 B2 US 11562240B2 US 202016884128 A US202016884128 A US 202016884128A US 11562240 B2 US11562240 B2 US 11562240B2
Authority
US
United States
Prior art keywords
crosspoint
cnn
array
input data
integrators
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US16/884,128
Other languages
English (en)
Other versions
US20210374514A1 (en
Inventor
Hsinyu Tsai
Geoffrey Burr
Pritish Narayanan
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Business Machines Corp
Original Assignee
International Business Machines Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by International Business Machines Corp filed Critical International Business Machines Corp
Assigned to INTERNATIONAL BUSINESS MACHINES CORPORATION reassignment INTERNATIONAL BUSINESS MACHINES CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: NARAYANAN, PRITISH, BURR, GEOFFREY, TSAI, Hsinyu
Priority to US16/884,128 priority Critical patent/US11562240B2/en
Priority to PCT/IB2021/054105 priority patent/WO2021240286A1/en
Priority to IL297331A priority patent/IL297331A/en
Priority to GB2218705.8A priority patent/GB2610774A/en
Priority to CA3178030A priority patent/CA3178030A1/en
Priority to KR1020227041560A priority patent/KR20230005309A/ko
Priority to CN202180036838.4A priority patent/CN115699028A/zh
Priority to DE112021002939.8T priority patent/DE112021002939T5/de
Priority to AU2021281628A priority patent/AU2021281628B2/en
Priority to JP2022568491A priority patent/JP2023526915A/ja
Publication of US20210374514A1 publication Critical patent/US20210374514A1/en
Priority to US18/061,074 priority patent/US11868893B2/en
Publication of US11562240B2 publication Critical patent/US11562240B2/en
Application granted granted Critical
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G06N3/065—Analogue means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G06N3/0445—

Definitions

  • the present invention relates in general to computing technology, and more particularly to artificial neural networks (ANN). More specifically, embodiments of the present invention relate to mapping a convolutional neural network (CNN) to crosspoint devices in crosspoint arrays, such as in analog-memory-based hardware for providing output from a trained CNN during a forward-inference phase.
  • CNN convolutional neural network
  • Machine learning is used to broadly describe a primary function of electronic systems that learn from data.
  • neural networks are a family of statistical learning models inspired by the biological neural networks of animals, and in particular, the brain. Neural networks can be used to estimate or approximate systems and functions that depend on a large number of inputs and are generally unknown. Neural networks use a class of algorithms based on a concept of interconnected “neurons.” In a typical neural network, neurons have a given activation function that operates on the inputs. By determining proper connection weights (a process also referred to as “training”), a neural network achieves efficient recognition of the desired patterns, such as images and characters.
  • these neurons are grouped into “layers” in order to make connections between groups more obvious and to organize the computation process. With these proper connection weights, other patterns of interest that have never been seen by the network during training can also be correctly recognized, a process known as “Forward-inference.”
  • a computer-implemented method for implementing a convolutional neural network (CNN) using a crosspoint array includes configuring the crosspoint array that implements a convolution layer in the CNN.
  • the configuring is performed by storing one or more convolution kernels of the convolution layer in one or more crosspoint devices of the crosspoint array.
  • the method further includes performing computations for the CNN via the crosspoint array by iterating, for a predetermined number of times, a set of operations.
  • the set of operations includes transmitting voltage pulses corresponding to a subpart of a vector of input data of the convolution layer to the crosspoint array.
  • the set of operations further includes outputting electric currents that are representative of performing multiplication operations at the one or more crosspoint device in the crosspoint array, the electric currents based on weight values stored by the crosspoint devices and the voltage pulses from the input data.
  • the set of operations also includes accumulating, by a set of integrators, an electric charge based on the output electric currents from the crosspoint devices.
  • the method further includes outputting, by the set of integrators, the accumulated charge after iterating for the predetermined number of times, the accumulated charge representing a multiply-add result of the vector of input data, and the one or more convolution kernels.
  • outputting the accumulated charge in the set of integrators includes pooling the accumulated charge.
  • the subpart of each vector of input data is associated with the set of integrators.
  • the crosspoint array includes several crosspoint arrays and wherein a first subpart of the vector of input data is transmitted to a first crosspoint array, and a second subpart of the vector of input data is transmitted to a second crosspoint array.
  • accumulating the electric charge by the set of integrators includes accumulating, by the set of integrators of the first crosspoint array, the electric charges that are accumulated by the set of integrators of the second crosspoint array.
  • the crosspoint devices are arranged to implement one or more columns of a convolution kernel of a given layer of the CNN, and wherein the vector of input data represents neuron excitations to the given layer of the CNN presented from the input data, one row at a time.
  • the electric charge accumulated by an integrator from the set of integrators represents an output excitation according to the given layer of the CNN, the output excitation being converted and transmitted only after all rows of said convolution kernel are integrated.
  • the crosspoint devices are arranged so as to implement one or more rows of a convolution kernel of a given layer of the CNN, and wherein the input data represent neuron excitations to said layer of the CNN presented one column at a time.
  • the electric charge accumulated by an integrator from the set of integrators represents an output excitation according to the given layer of the CNN, the output excitation being converted and transmitted only after all columns of said convolution kernel are integrated.
  • an electronic circuit for performing computations of a trained convolutional neural network (CNN).
  • the electronic circuit includes a crosspoint array, and an output circuit that in turn includes one or more integrators.
  • the method further includes providing the crosspoint array, and providing the output circuit.
  • the method further includes configuring the crosspoint array corresponding to a convolution layer in the CNN by storing one or more convolution kernels of the convolution layer in one or more crosspoint devices of the crosspoint array.
  • the method further includes iterating, for a predetermined number of times, a set of operations. Training of the CNN is performed using the method described above.
  • an electronic circuit that includes an array of resistive memory elements.
  • the array provides a vector of current outputs equal to an analog vector-matrix-product between (i) a vector of voltage inputs to the array encoding a vector of analog input values and (ii) a matrix of analog resistive weights within the array.
  • the electronic circuit further includes accumulation wires and circuits aggregating a current from a dedicated subset of the resistive memory elements.
  • the electronic circuit further includes integration capacitors, each of the integration capacitors being electrically switchable so as to aggregate current from one of a plurality of accumulation wires during a single integration step.
  • the electronic circuit further includes data-output circuitry to allow an integrated charge from a subset of the integration capacitors, accumulated over several integration steps, to be suitably converted and transmitted either as an analog duration or as a digital representation using binary digits, wherein the resistive memory elements are arranged so as to implement vectors of synaptic weight kernels of a given layer of a convolutional neural network.
  • a method uses the electronic circuit to perform an accumulation over several integration steps implements multiply-accumulate operations across multiple partial vectors of said weight kernels.
  • the accumulation includes performing computations by the resistive memory elements of the crosspoint array by iterating, for a predetermined number of times, a set of operations.
  • the set of operations includes partitioning, into the multiple partial vectors, each vector of analog input values.
  • the set of operations also includes accumulating, in analog memory, partial output excitations corresponding to each of the multiple partial vectors.
  • the set of operations also includes combining the partial output excitations by routing the partial output excitations to an integration capacitor that accumulates an integrated charge. Further, the accumulation further includes transmitting the integrated charge on the integration capacitors representing output excitations.
  • the integrated charge on the integration capacitors is pooled locally prior to transmitting the integrated charge.
  • the resistive memory elements are non-volatile memory devices.
  • the subset of the resistive memory elements corresponds to one or more column(s) of the array. In one or more embodiments of the present invention, the subset of the resistive memory elements corresponds to one or more row(s) of the array.
  • the crosspoint devices are arranged so as to implement one or more rows of the convolution kernels of a given layer of the convolutional neural network, and wherein the input data represent neuron excitations to said layer of the convolutional neural network presented one column at a time.
  • the crosspoint devices are arranged to implement one or more columns of the convolution kernels of a given layer of the convolutional neural network, and wherein the vector of input data represent neuron excitations to the given layer of the convolutional neural network presented from the input data, one row at a time.
  • FIG. 1 depicts a simplified diagram of input and output connections of a mathematical neuron
  • FIG. 2 depicts a simplified model of the mathematical neuron shown in FIG. 1 ;
  • FIG. 3 depicts a simplified model of an ANN incorporating the mathematical neuron model shown in FIG. 2 ;
  • FIG. 4 illustrates a simplified block diagram of a representative CNN, which is interpreting a sample input map
  • FIG. 5 illustrates an example convolutional layer in a CNN being trained using training data that include input maps and convolution kernels
  • FIG. 6 depicts a system for performing a matrix-matrix multiplication using a crosspoint array according to one or more embodiments of the present invention
  • FIG. 7 depicts a two-dimensional (2D) crossbar system that performs forward matrix multiplication, backward matrix multiplication and weight updates according to the present description
  • FIG. 8 depicts an expanded view of the crosspoint array according to one or more embodiments of the present invention.
  • FIG. 9 depicts a typical output circuitry in a crossbar system
  • FIG. 10 depicts existing operations to perform forward-inference operations using a crosspoint array
  • FIG. 11 depicts performing forward-inference operations using partial accumulation, where the partial accumulation is based on time-partitioning according to one or more embodiments of the present invention
  • FIG. 12 depicts performing forward-inference operations using partial accumulation across multiple crosspoint arrays according to one or more embodiments of the present invention.
  • FIG. 13 depicts performing forward-inference operations using partial accumulation, where the partial accumulation is based on space-partitioning according to one or more embodiments of the present invention.
  • Deep learning techniques are widely used in machine-based pattern recognition problems, such as image and speech recognition. Deep learning inherently leverages the availability of massive training datasets (that are enhanced with the use of Big Data) and compute power (that is expected to grow according to Moore's Law).
  • Embodiments of the present invention facilitate efficient workload mapping of convolutional neural networks (CNNs) into analog arrays when implementing an analog artificial intelligence system, such as an artificial neural network (ANN) using crosspoint arrays.
  • CNNs convolutional neural networks
  • ANN artificial neural network
  • Existing techniques describe a “row-by-row” mapping of weights for CNN inference workload so that activations through each layer of the CNN are efficiently used and streamlined to limit storage requirements.
  • a technical challenge exists with the existing techniques, in that the analog array area utilization with such a “row-by-row” mapping technique is low and impacts the scalability of such techniques. For example, mapping a large CNN (such as ResNet-50) can require a large number of analog arrays to implement the CNN, which can make the implementation to be inefficient, unwieldy, and cross prohibitive.
  • Embodiments of the present invention address such technical challenges during implementation of an ANN, particularly, a CNN, by providing a flexible inter-array routing scheme that facilitates a compact mapping of CNN layers for the row-by-row mapping technique.
  • One or more embodiments of the present invention benchmark the number of analog arrays (tiles) needed against existing row-by-row mapping techniques that use a generic mapping technique where activations are not streamlined or reused. Accordingly, embodiments of the present invention facilitate array utilization that is comparable for a wide range of CNNs while preserving the advantages of streamlined activation for row-by-row mapping.
  • ANNs are often embodied as so-called “neuromorphic” systems of interconnected processor elements that act as simulated “neurons” and exchange “messages” between each other in the form of electronic signals. Similar to the so-called “plasticity” of synaptic neurotransmitter connections that carry messages between biological neurons, the connections in ANNs that carry electronic messages between simulated neurons are provided with numeric weights that correspond to the strength or weakness of a given connection. The weights can be adjusted and tuned based on experience, making ANNs adaptive to inputs and capable of learning. For example, an ANN for handwriting recognition is defined by a set of input neurons which can be activated by the pixels of an input image.
  • the activations of these input neurons are then passed to other downstream neurons, which are often referred to as “hidden” neurons. This process is repeated until an output neuron is activated. The activated output neuron determines which character was read.
  • Crossbar arrays also known as crosspoint arrays, crosswire arrays, or resistive processing unit (RPU) arrays, are high density, low-cost circuit architectures used to form a variety of electronic circuits and devices, including ANN architectures, neuromorphic microchips, and ultra-high density nonvolatile memory.
  • a basic crosspoint array configuration includes a set of conductive row wires and a set of conductive column wires formed to intersect the set of conductive row wires. The intersections between the two sets of wires are separated by so-called crosspoint devices, which can be formed from thin-film material.
  • Crosspoint devices in effect, function as the ANN's weighted connections between neurons.
  • Nanoscale two-terminal devices for example, memristors having “ideal” conduction state switching characteristics, are often used as the crosspoint devices in order to emulate synaptic plasticity with high energy efficiency.
  • the conduction state (e.g., resistance) of the ideal memristor material can be altered by controlling the voltages applied between individual wires of the row and column wires.
  • Digital data can be stored by alteration of the memristor material's conduction state at the intersection to achieve a high conduction state or a low conduction state.
  • the memristor material can also be programmed to maintain two or more distinct conduction states by selectively setting the conduction state of the material.
  • the conduction state of the memristor material can be read by applying a voltage across the material and measuring the current that passes through the target crosspoint device.
  • the crosspoint devices of ANN chip architectures are often designed to utilize offline learning techniques, wherein the approximation of the target function does not change once the initial training phase has been resolved. Offline learning allows the crosspoint devices of crossbar-type ANN architectures to be simplified such that they draw very little power.
  • ANNs also known as neuromorphic or synaptronic systems, are computational systems that can estimate or approximate other functions or systems, including, for example, biological neural systems, the human brain, and brain-like functionality such as image recognition, speech recognition and the like.
  • ANNs incorporate knowledge from a variety of disciplines, including neurophysiology, cognitive science/psychology, physics (statistical mechanics), control theory, computer science, artificial intelligence, statistics/mathematics, pattern recognition, computer vision, parallel processing and hardware (e.g., digital/analog/VLSI/optical).
  • ANNs create connections between processing elements that are substantially the functional equivalent of the core system functionality that is being estimated or approximated.
  • a computer chip that is the central component of an electronic neuromorphic machine attempts to provide similar form, function, and architecture to the mammalian brain.
  • the computer chip uses the same basic transistor components as conventional computer chips, its transistors are configured to mimic the behavior of neurons and their synapse connections.
  • the computer chip processes information using a network of just over one million simulated “neurons,” which communicate with one another using electrical spikes similar to the synaptic communications between biological neurons.
  • the architecture of such a computer chip includes a configuration of processors (i.e., simulated “neurons”) that read a memory (i.e., a simulated “synapse”) and perform simple operations.
  • processors i.e., simulated “neurons”
  • memory i.e., a simulated “synapse”
  • the communications between these processors are performed by on-chip network routers.
  • FIG. 1 depicts a simplified diagram of a mathematical neuron 102 having pathways 104 , 106 , 108 , 110 that connect it to upstream inputs 112 , 114 , downstream outputs 116 and downstream “other” neurons 118 , configured and arranged as shown.
  • Each mathematical neuron 102 sends and receives electrical impulses through pathways 104 , 106 , 108 , 110 .
  • Mathematical neuron 102 is modeled in FIG. 2 as a node 202 having a mathematical function, f(x) depicted by the equation shown in FIG. 2 .
  • Node 202 takes electrical signals from inputs 212 , 214 , multiplies each input 212 , 214 by the strength of its respective connection pathway 204 , 206 , takes a sum of the inputs, passes the sum through a function, f(x), and generates a result 216 , which can be a final output or an input to another node, or both.
  • an asterisk (*) is used to represent a multiplication, which can be a matrix multiplication.
  • the matrix multiplication can be used to perform convolution operations between input data and one or more convolution kernels to generate output maps.
  • Weak input signals are multiplied by a very small connection strength number, so the impact of a weak input signal on the function is very low.
  • strong input signals are multiplied by a higher connection strength number, so the impact of a strong input signal on the function is larger.
  • the function f(x) is a design choice, and a variety of functions can be used.
  • a typical design choice for f(x) is the hyperbolic tangent function, which takes the function of the previous sum and outputs a number between minus one and plus one.
  • An alternative design choice of f(x) is a rectified linear unit (ReLU), a function in which the output matches the input for positive inputs and is zero otherwise.
  • ReLU rectified linear unit
  • FIG. 3 depicts a simplified ANN model 300 organized as a weighted directional graph, wherein the artificial neurons are nodes (e.g., 302 , 308 , 316 ), and wherein weighted directed edges (e.g., m1 to m20) connect the nodes.
  • ANN model 300 is organized such that nodes 302 , 304 , 306 are input-layer nodes, nodes 308 , 310 , 312 , 314 are hidden-layer nodes, and nodes 316 , 318 are output-layer nodes.
  • Each node is connected to every node in the adjacent layer by connection pathways, which are depicted in FIG. 3 as directional arrows having connection strengths m1 to m20.
  • Each hidden-layer node 308 , 310 , 312 , 314 receives its inputs from all input-layer nodes 302 , 304 , 306 , according to the connection strengths associated with the relevant connection pathways.
  • y4 f(m1*y1+m5*y2+m9*y3), wherein * represents a multiplication.
  • the multiplication can be a matrix multiplication used to perform a convolution operation.
  • a similar connection strength multiplication and node summation are performed for hidden-layer nodes 310 , 312 , 314 , and output-layer nodes 316 , 318 , as shown by the equations defining functions y5 to y9 depicted at the bottom of FIG. 3 .
  • ANN model 300 processes data records one at a time, and it “learns” by comparing an initially arbitrary classification of the record with the known actual classification of the record.
  • backpropagation i.e., “backward propagation of errors”
  • the errors from the initial classification of the first record are fed back into the network and used to modify the network's weighted connections the second time around, and this feedback process continues for several iterations.
  • the correct classification for each record is known, and the output nodes can, therefore, be assigned “correct” values, for example, a node value of “1” (or 0.9) for the node corresponding to the correct class, and a node value of “0” (or 0.1) for the others. It is thus possible to compare the network's calculated values for the output nodes to these “correct” values and to calculate an error term for each node (i.e., the “delta” rule). These error terms are then used to adjust the weights in the hidden-layers so that in the next iteration, the output values will be closer to the “correct” values.
  • ANN model 300 is a non-recurrent feed-forward network having inputs, outputs, and hidden-layers. The signals can only travel in one direction. Input data are passed onto a layer of processing elements that perform calculations. Each processing element makes its computation based upon a weighted sum of its inputs. The new calculated values then become the new input values that feed the next layer. This process continues until it has gone through all the layers and determined the output. A threshold transfer function is sometimes used to quantify the output of a neuron in the output-layer.
  • a feedback/recurrent network includes feedback paths, which means that the signals can travel in both directions using loops. All possible connections between nodes are allowed. Because loops are present in this type of network, under certain operations, it can become a non-linear dynamical system that continuously changes until it reaches a state of equilibrium. Feedback networks are often used in associative memories and optimization problems, wherein the network looks for the best arrangement of interconnected factors.
  • ANNs “learn” underlying rules (like input-output relationships) from the given collection of representative examples. Accordingly, a learning algorithm can be generally defined as the procedure by which learning rules are used to update and/or adjust the relevant weights.
  • the three main learning algorithm paradigms are supervised, unsupervised, and hybrid.
  • supervised learning or learning with a “teacher,” the network is provided with a correct answer (output) for every input pattern. Weights are determined to allow the network to produce answers as close as possible to the known correct answers.
  • Reinforcement learning is a variant of supervised learning in which the network is provided with only a critique on the correctness of network outputs, not the correct answers themselves.
  • unsupervised learning, or learning without a teacher does not require a correct answer associated with each input pattern in the training data set. It explores the underlying structure in the data, or correlations between patterns in the data, and organizes patterns into categories from these correlations.
  • Hybrid learning combines supervised and unsupervised learning.
  • the forward-inference of already trained networks includes applications, ranging from implementations of cloud-based services built on ANNs to a smartphone, Internet-Of-Things (IOT), and other battery-constrained applications which require extremely low power operation.
  • IOT Internet-Of-Things
  • the forward-inference is an application that calls for fast latency (so that any given new test example can be classified, recognized, or otherwise processed as rapidly as possible).
  • a CNN kernels convolute overlapping regions, such as those in a visual field, and accordingly emphasize the importance of spatial locality in feature detection.
  • Computing the convolutional layers of the CNN typically, encompasses more than 90% of computation time in neural network training and inference. Mapping of CNNs into analog arrays and ensuring efficient use of electrical power used while performing the mathematical operations of the convolutional layers, with minimum extraneous data movement or computation, is a technical challenge.
  • the technical challenge includes mapping the CNN for inference as well as maintaining the scalability of such a mapping so that even large CNNs, such as the ResNet-50, can be implemented.
  • one or more embodiments of the present invention facilitate a flexible inter-array routing of data that enable a compact mapping of CNN layers to crosspoint arrays for the row-by-row technique.
  • FIG. 4 illustrates a simplified block diagram of a CNN.
  • CNN is being used for interpreting a sample input map 400 , and in this particular example uses a handwritten letter “w” as an input map.
  • the input map 100 is used to create a set of values for the input-layer 410 , or “layer-1.”
  • layer-1 can be generated by direct mapping of a pixel of the sample input map 400 to a particular neuron in layer-1, such that the neuron shows a 1 or a 0 depending on whether the pixel exhibits a particular attribute.
  • each layer of the network can have differing numbers of neurons, and these may or may not be related to particular qualities of the input data.
  • neurons in layer-1 410 are connected to neurons in a next layer, layer-2 420 , as described earlier (see FIG. 3 ).
  • the neurons in FIG. 4 are as described with reference to FIG. 1 .
  • a neuron in layer-2 420 consequently, receives an input value from each of the neurons in layer-1 410 .
  • the input values are then summed, and this sum compared to a bias. If the value exceeds the bias for a particular neuron, that neuron then holds a value, which can be used as input to neurons in the next layer of neurons.
  • each value in the layer is assigned to a particular character.
  • the network is configured to end with the output-layer having only one large positive value in one neuron, which then demonstrates which character the network has computed to be the most likely handwritten input character.
  • the network may have been designed such that output neuron values may be used to estimate the probability (likelihood), confidence, or other metrics of interest.
  • each layer in the CNN is typically represented using matrices (or tensors in some examples), and computations are performed as matrix computations.
  • the indexes (and/or sizes) of the matrices vary from layer to layer and network to network, as illustrated in FIG. 4 . Different implementations orient the matrices or map the matrices to computer memory differently.
  • each level is a tensor of neuron values, as is illustrated by matrix dimensions for each layer of the neural network.
  • an example might be multiple input “planes,” each a two-dimensional image.
  • connections strengths which are in a transformation tensor known as a filter.
  • This matrix multiplication scales each value in the previous layer according to the connection strengths, with the aggregate total of these contributions then summed.
  • This fundamental operation is known as a multiply-accumulate operation.
  • a bias matrix may then added to the resulting product matrix to account for the threshold of each neuron in the next level.
  • an activation function is applied to each resultant value, and the resulting values are placed in the output tensor to be applied to the next layer.
  • the activation function can be rectified linear units, sigmoid, or tan h( ).
  • the connections between each layer, and thus an entire network can be represented as a series of matrices. Training the CNN includes finding proper values for these matrices.
  • FIG. 5 illustrates an example convolutional layer 500 in a CNN being trained using training data that includes input maps 510 and convolution kernels 520 .
  • the input maps 510 can include multiple input patterns, for example, D input maps.
  • Each input map is a matrix, such as a matrix of size N ⁇ M. Accordingly, a total number of input neurons, in this case, is N ⁇ M ⁇ D.
  • the input maps are convolved with F convolution kernels 520 of size k ⁇ k, as illustrated to produce corresponding output, maps 530 .
  • Each output map can have a dimension N′ ⁇ M′.
  • the output maps are of size n ⁇ k+1 ⁇ n ⁇ k+1.
  • Each convolution is a 3D convolution involving the D input maps.
  • a CNN can include multiple such layers, where the output maps 530 from a previous layer are used as input maps 510 for a subsequent layer.
  • the backpropagation algorithm can be used to learn the k ⁇ k ⁇ D ⁇ F weight values of the filters.
  • the input maps 510 are convolved with each filter bank to generate a corresponding output map.
  • the input maps 510 are combined with a filter bank that includes convolution kernels representing a vertical line.
  • the resulting output map identifies vertical lines that are present in the input maps 510 .
  • another filter bank can include convolution kernels representing a diagonal line, such as going up and to the right.
  • An output map resulting from a convolution of the input maps 510 with the second filter bank identifies samples of the training data that contain diagonal lines.
  • the two output maps show different information for the character while preserving pixel adjacency. This can result in more efficient character recognition.
  • FIG. 6 depicts a system 600 in which the crosspoint array 700 is controlled using a controller 610 for performing the matrix-matrix multiplication(s) among other operations according to one or more embodiments of the present invention.
  • the controller 610 sends the input data 510 to be multiplied by the crosspoint array 700 .
  • the controller 610 stores the weight values, such as from convolution kernels 520 in the crosspoint array 700 , and sends the input vectors.
  • the controller 610 and the crosspoint array 700 are coupled in a wired or a wireless manner, or a combination thereof.
  • the controller 610 further sends an instruction/command to the crosspoint array 700 to initiate the operations for one or more layers in the CNN.
  • the controller 610 further can read the output data 530 from the crosspoint array 700 after receiving a notification that the computations have been performed.
  • the controller 610 can be a processing unit, or a computing system, such as a server, a desktop computer, a tablet computer, a phone, and the like.
  • the controller 610 can include a memory device that has computer-executable instructions stored thereon, the instructions, when executed by the controller, cause the matrix-matrix computation.
  • one or more embodiments are directed to a crosspoint array having crosspoint devices at each intersection of the crossbar wires, the crosspoint array being used to implement the CNN.
  • An example of a crosspoint device is a two-terminal programmable resistive crosspoint component referred to herein as a resistive processing unit (RPU), which provides local data storage functionality and local data processing functionality.
  • RPU resistive processing unit
  • the weighted contribution represented by each crosspoint device is contributed to a massively-parallel multiply-accumulate operation that is performed at the stored location of data. This eliminates the need to move relevant data in and out of a processor and a separate storage element.
  • implementing a machine learning CNN architecture having the described crosspoint device enables the implementation of online machine learning capabilities that facilitate training the CNN, and subsequently, performing inference using the trained CNN models.
  • the described crosspoint device and resulting CNN architecture improve overall CNN performance and enable a broader range of practical CNN applications.
  • the described crosspoint device can be implemented as two-terminal resistive crosspoint devices.
  • the described crosspoint device can be implemented with resistive random access memory (RRAM), phase change memory (PCM), programmable metallization cell (PMC) memory, non-linear memristor systems, or any other device that offers a wide range to analog-tunable non-volatile resistive memory states that are sufficiently stable over time.
  • RRAM resistive random access memory
  • PCM phase change memory
  • PMC programmable metallization cell
  • non-linear memristor systems or any other device that offers a wide range to analog-tunable non-volatile resistive memory states that are sufficiently stable over time.
  • FIG. 7 depicts a two-dimensional (2D) crossbar system 700 that performs forward-inference according to the present description.
  • the crossbar system 700 can be used to implement simple matrix multiplication, backward matrix-multiplication, and even in-situ weight-update according to the backpropagation algorithm.
  • the crossbar system 700 includes a crosspoint array 705 , an input circuitry 710 , and output circuitry 720 , among other components.
  • the input circuitry 710 and the output circuitry 720 can be, together, referred to as peripheral circuitry.
  • the crossbar system 700 can be a computer chip in one or more examples.
  • FIG. 8 depicts an expanded view of the crosspoint array 705 according to one or more embodiments.
  • the crosspoint array 705 is formed from a set of conductive row wires 802 , 804 , 806 , and a set of conductive column wires 808 , 810 , 812 , 814 that intersect the set of conductive row wires 802 , 804 , 806 .
  • the intersections between the set of row wires and the set of column wires are separated by crosspoint devices, which are shown in FIG.
  • the conduction state (i.e., the stored weights) of the crosspoint device can be read by applying a voltage across the crosspoint device and measuring the current that passes through the crosspoint device.
  • Input voltages V 1 , V 2 , V 3 are applied to row wires 802 , 804 , 806 , respectively.
  • Each column wire 808 , 810 , 812 , 814 sums the currents I 1 , I 2 , I 3 , I 4 generated by each crosspoint device along the particular column wire using an integrator, such as a capacitor.
  • array 705 computes the forward matrix multiplication by multiplying the values stored in the crosspoint devices by the row wire inputs, which are defined by voltages V 1 , V 2 , V 3 .
  • the input circuitry 710 includes, in one or more examples, at least a support circuitry 712 , a shared circuitry 714 , and row circuitry 716 .
  • the row circuitry includes hardware components associated with each row wire 802 , 804 , and 806 .
  • the input circuitry 710 facilitates providing the input voltages to the crosspoint array 705 .
  • FIG. 9 depicts a typical output circuitry 720 .
  • the output circuitry includes integrators 908 , 910 , 912 , and 914 corresponding to the column wires 808 , 8120 , 812 , and 814 .
  • the integrators 908 , 910 , 912 , and 914 are capacitors.
  • the output currents along each column wire are accumulated in the integrators and passed on to the next layer of the CNN.
  • such an arrangement of the integrators makes the computations of the FC layers very efficient.
  • to use such an arrangement of the integrators incurs significant additional overhead in terms of data transport, storage, organization, and subsequent data transport. Such operations require additional resources such as time, power, and additional circuit-area, thus making the overall system inefficient.
  • FIG. 10 depicts existing operations to perform forward-inference operations using the crosspoint array.
  • one image-row ( 512 , 514 , and 516 ) of all input planes 510 is presented concurrently as a column of inputs to the array-rows ( 802 , 804 , and 806 ) of the crosspoint array 705 of the crossbar system 700 .
  • the crosspoint devices 820 at each crosspoint contains weight-elements from the filters 520 , each leading to a multiplication between the array-row excitation, x i , and the stored weight, w ij , by Ohm's law (voltage times conductance equals current).
  • the integration of all such read current contributions is summed along each array-column and stored in the corresponding integrators ( 908 , 910 , 912 , and 914 ) of the array-columns ( 808 , 810 , 812 , and 814 ).
  • the computation can be expressed as the current I 1 on column #1 ( 808 ) is stored on capacitor C 1 ( 908 ), I 2 is stored on capacitor C 2 , I 3 on C 3 , and so on.
  • the integrated charge on the capacitors ( 908 , 910 , 912 , and 914 ) is treated as the output of the multiply-accumulate and is either converted to a digital number or to pulse-duration for shipment to a next array 705 .
  • every output from convolutional layer i has to be combined with outputs from other convolutional layers as part of pooling.
  • the other convolutional layers from which the outputs that are to be pooled depend on the number of elements in the filter kernels 520 .
  • every output from layer i has to be positioned at different spots in the input planes 510 for the convolutional layer i+1.
  • Such organization of the output values for the purpose of pooling can also require additional computing resources, such as read-write access, power, and the like.
  • the system 700 integrates results into capacitors 908 , 910 , 912 , and 914 , but does not immediately send the result to the next layer. That is because the system 700 has to steer read current from several different columns onto the integration capacitor(s) 908 , 910 , 912 , and 914 . The system 700 performs such steering of the results from other columns at subsequent time-steps. In the same manner, the system 700 takes k time-steps to compute each kth output row. Accordingly, existing techniques using row-by-row mapping result in each output row taking k time-steps to generate.
  • FIG. 10 depicts the operations performed by the array 705 during forward-inference according to existing techniques.
  • the times-steps 1, 2, and 3 are depicted.
  • the input is mapped to a row in the crosspoint array 705 .
  • each of the integrators receives contributions from k*p multiply-accumulate terms, where p is the number of input planes 510 .
  • the total charge on an integrator contains all k*k*p terms and is ready to be output to the next convolutional layer.
  • every kth integrator from the output circuit 720 reaches this status, and accordingly, is ready to generate all the output pixels of one image-row ( 512 -A, 514 -A, and 516 -A) of the convolutional-layer output. All other jth integrators have a different phase in their respective integration phase, depending on the value of j.
  • the first rows of each input plane 512 -A, 514 -A, 516 -A are input to the convolutional layer.
  • the crosspoint devices 820 of the crosspoint array 705 are loaded with the filters 520 , as shown.
  • filter kernels 522 -A and 522 -B are loaded in the crosspoint devices 820 to perform a convolution with the first rows of the first input plane 516 -A.
  • filter kernels 524 -A and 524 -B from a second bank of filter kernels 520 are convolved with the first row of a second input plane 514 -A, and so on.
  • the results of the respective convolutions are forwarded to one or more of the integrators ( 908 , 910 , 912 , 914 ) from the output circuitry 720 by output controller 1110 .
  • the output controller 1110 can be part of the output circuitry 720 or an external controller that is coupled with the output circuitry 720 .
  • the output controller 1110 steers the output of the multiply-accumulate operations from each column in the array 705 to a particular integrator in the output circuitry 720 .
  • the output controller 1110 receives a mode signal that provides a selection of the integrators for each column at each time-step.
  • the output controller 1110 is provided a mode signal that indicates the selection of the integrator for each column until all convolutional layers are executed.
  • the mode signal in one or more examples, can be a bit pattern that is indicative of the selected integrators for each column.
  • the outputs from the columns 808 and 814 are stored in the integrators 908 and 912 , respectively, at time-step 1.
  • time-step #2 second rows 512 -B, 514 -B, and 516 -B from the input planes 510 are used as input to the crosspoint array 705 .
  • the crosspoint devices 820 are still loaded with the kernel filters 520 as in time-step #1 ( FIG. 10 ).
  • the output controller 1110 selects the same integrators 908 and 912 for the outputs of the columns 810 and 816 (different columns from time-step 1). Accordingly, the integrators 908 and 912 (and others), in this case, receive outputs from different columns in different time-steps.
  • time-step 3 in a similar manner to the first two time-steps, third rows 512 -C, 514 -C, and 516 -C from the input planes 510 are used as input to the crosspoint array 705 .
  • the output controller 1110 selects the same integrators 908 and 912 for the outputs of the columns 812 and 818 (different columns from time-step 1, 2). Accordingly, the integrators 908 and 912 (and others), in this case, receive outputs from different columns in different time-steps. In this manner, in general, after k time-steps, an entire row in the output planes 530 is computed.
  • the crosspoint array 705 can be accumulating for other output rows (C and D) at each time-step using the other integrators ( 910 , 914 , 916 , and 918 ), as shown in FIG. 10 .
  • Embodiments of the present invention facilitate row by row mapping with reducing the number of weight copies through CNN depth. Accordingly, embodiments of the present invention facilitate load balancing to account for a change in CNN weight reuse factor. Further, embodiments of the present invention facilitate row by row mapping with more compact weight mapping using flexible routing of data from the input circuit 710 , to and through the crosspoint array 705 , and to the output circuit 720 .
  • the technical solutions described herein address such technical challenges with the existing technical solutions by facilitating a row-by-row convolution with partial row input, where the input data are partitioned in time.
  • row-by-row convolution is facilitated with partial row input, where the input data is partitioned in space (crosspoint arrays).
  • FIG. 11 depicts a row-by-row convolution mapping with partial row input, where the input data are partitioned in time according to one or more embodiments of the present invention.
  • the partial sums from different input row segments are stored on separate sets of capacitors.
  • a first subset 1210 of input data from a first row are mapped to a first set of capacitors (or integrators) 1230 ; and a second subset 1220 of the input data from the first row are mapped to a second set of capacitors 1240 .
  • N is a number of copies of capacitors used to facilitate reusing the crosspoint array 705 for the computations for the forward-inference.
  • N can be predetermined based on the image width. For example, N can be increased as image size reduces to reduce the number of weight copies to keep the reuse factor the same.
  • CNN convolution neural network
  • weight kernels are convoluted over input images, i.e., the same weights are reused multiple times with different parts of the input images to generate the output. The number of times that the weights are reused is referred to as the reuse factor.
  • the two sets of capacitors 1230 and 1240 can reuse the weights that are stored in the crosspoint array 705 .
  • the input data are mapped to have a row-by-row convolution to be computed by the crosspoint devices 820 .
  • the computed L is the number of input data elements that are input to the crosspoint array 705 , with each sequential data element being from a sequential input plane.
  • the notation D1(1,1) refers to the element from the first row and first column in D1.
  • the crosspoint devices 820 perform in-memory computations to determined partial sums of the products of the stored weights and the data elements.
  • the computations are performed in-memory in an analog manner.
  • the resulting partial sums are stored in the capacitors in the sets 1230 , 1240 .
  • Number of capacitors in each of the sets of capacitors 2130 , 2140 increases if the number of copies of the weights is reduced.
  • the partial sums are sent to the destination capacitor (input side of the next layer crosspoint array 705 ). Even though the overlap results in redundant computation, the improvement in the efficiency of the crosspoint array 705 achieved by the reuse results in a reduced number of crosspoint arrays 705 required to implement the CNN.
  • FIG. 12 depicts a row-by-row convolution mapping with full or partial row input, where the input data are partitioned in time according to one or more embodiments of the present invention.
  • the depicted mapping scheme facilitates an even more compact mapping by using multiple reused copies weights that span across multiple crosspoint arrays 705 .
  • the number of output image channel is F
  • the number of input image channel is D.
  • each group of weights has a stride with an offset of (D*stride) to span across the (input image width*D) dimension. After every set of such (output image width) copies of weights, the next set is configured without any offset (D*stride).
  • a group of such copies of weights can span separate crosspoint arrays 705 A and 705 B.
  • a group 1280 has two copies of weights 1282 and 1284 that are stored in a first crosspoint array 705 A, and the third copy of weights 1286 that is stored on a second crosspoint array 705 B.
  • FIG. 13 depicts another row-by-row convolution mapping with partial row input, where the input data is partitioned in space according to one or more embodiments of the present invention.
  • a single set of capacitors 1320 is used to compute resulting partial sums based on the kernel weights stored in the crosspoint array 705 .
  • the input data elements are divided such that subsets of a single row are sent to separate crosspoint arrays 705 that are implementing a given CNN layer.
  • the charge that is accumulated on the capacitors 1320 which represents the partial sum, is sent to an input circuit 710 of a system 700 that is implementing a next layer of the CNN.
  • the input circuit 710 combines the partial sums and organizes such input data for the weights stored in the separate crosspoint arrays 705 for the next layer. For example, the input circuit 710 channels the outputs corresponding to the input data elements 1310 to the same kernel weights in the next layer as the outputs corresponding to the input data elements 1320 .
  • the flexible routing provided by the input circuit 710 as described above facilitates weights for different output channels to be mapped more compactly compared to existing solutions.
  • the cost for storing and re-ordering of the outputs from one layer during input to the next layer is lower than the row-by-row mapping techniques that are existing.
  • one or more embodiments of the present invention facilitate configuring a flexible signal routing scheme to improve the scalability of the existing row-by-row mapping techniques.
  • the CNN can be fine-tuned according to the particular operation detail of each network. For example, a CNN kernel size or a number of CNN kernels can be adjusted to further optimize mapping into analog crosspoint arrays.
  • the dimensions of the matrices shown in the figures herein are just examples, and in one or more examples, different dimensions can be used. Further, it should be noted that during the forward-inference operations, the CNN is already trained, and, that the embodiments of the present invention are applicable regardless of techniques used for training the CNN.
  • embodiments of the present invention facilitate row-by-row mapping for forward-inference operations of a trained CNN, where the mapping can be performed in a compact manner to reuse crosspoint arrays and supporting circuitry to facilitate the implementation of a CNN of any scale.
  • an electronic circuit that includes a crosspoint array of resistive memory elements.
  • the array provides a vector of current outputs equal to an analog vector-matrix-product between (i) a vector of voltage inputs to the array encoding a vector of analog input values and (ii) a matrix of analog resistive weights within the array.
  • the electronic circuit 700 further includes support circuitry 712 , 722 , and input circuitry 710 , and output circuitry 720 , that together include accumulation wires and circuits aggregating a current from a dedicated subset of the resistive memory elements.
  • the support circuitry 722 includes integration capacitors, each of the integration capacitors being electrically switchable so as to aggregate current from one of the accumulation wires during a single integration step.
  • the output circuitry 720 suitably converts and transmits, either as an analog duration or as a digital representation using binary digits, an integrated charge from a subset of the integration capacitors, accumulated over a predetermined number of integration steps.
  • the resistive memory elements are arranged so as to implement columns (rows) of synaptic weight kernels of a given layer of a convolutional neural network.
  • Accumulation over the predetermined number of integration steps implements multiply-accumulate operations across multiple partial rows (columns) of said weight kernels, as the input neuron excitations to the said layer of the convolutional neural network are presented one row (column) every iteration of the integration.
  • the input neuron excitations of the first layer are always presented one full row (column) at a time, while subsequent layers of input neuron excitations can be partitioned into multiple partial rows (columns) and partially stored in local analog memory (e.g., capacitors), to be processed in crosspoint arrays over multiple integration cycles.
  • the integrated charge representing a full and/or partial output excitation is suitably converted and transmitted only after all rows (columns) of said weight kernel are fully integrated. Partial sums from multiple crossbar arrays are flexibly routed to be combined in one of the integration capacitors, subsequently converted to the full output excitation, and then transmitted after all partial sums are fully integrated.
  • the integrated charge on the integration capacitors represents output excitations, which are suitably converted. Further, a suitably pooled result (e.g., the maximum, sum, or average of the said output excitations) is computed locally and then transmitted, only after all relevant weight kernels are fully integrated.
  • the present technical solutions may be a system, a method, and/or a computer program product at any possible technical detail level of integration
  • the computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to carry out aspects of the present technical solutions.
  • the computer-readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device.
  • the computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
  • a non-exhaustive list of more specific examples of the computer-readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing.
  • RAM random access memory
  • ROM read-only memory
  • EPROM or Flash memory erasable programmable read-only memory
  • SRAM static random access memory
  • CD-ROM compact disc read-only memory
  • DVD digital versatile disk
  • memory stick a floppy disk
  • a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon
  • a computer-readable storage medium is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
  • Computer-readable program instructions described herein can be downloaded to respective computing/processing devices from a computer-readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network.
  • the network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers.
  • a network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing/processing device.
  • Computer-readable program instructions for carrying out operations of the present technical solutions may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the “C” programming language or similar programming languages.
  • the computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
  • the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
  • electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present technical solutions.
  • These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
  • These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
  • the computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
  • each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s).
  • the functions noted in the blocks may occur out of the order noted in the Figures.
  • two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
  • a second action may be said to be “in response to” a first action independent of whether the second action results directly or indirectly from the first action.
  • the second action may occur at a substantially later time than the first action and still be in response to the first action.
  • the second action may be said to be in response to the first action even if intervening actions take place between the first action and the second action, and even if one or more of the intervening actions directly cause the second action to be performed.
  • a second action may be in response to a first action if the first action sets a flag and a third action later initiates the second action whenever the flag is set.
  • the phrases “at least one of ⁇ A>, ⁇ B>, . . . and ⁇ N>” or “at least one of ⁇ A>, ⁇ B>, ⁇ N>, or combinations thereof” or “ ⁇ A>, ⁇ B>, . . . and/or ⁇ N>” are to be construed in the broadest sense, superseding any other implied definitions hereinbefore or hereinafter unless expressly asserted to the contrary, to mean one or more elements selected from the group comprising A, B, . . . and N.
  • the phrases mean any combination of one or more of the elements A, B, . . . or N including any one element alone or the one element in combination with one or more of the other elements which may also include, in combination, additional elements not listed.
  • any module, unit, component, server, computer, terminal or device exemplified herein that executes instructions may include or otherwise have access to computer-readable media such as storage media, computer storage media, or data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape.
  • Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Such computer storage media may be part of the device or accessible or connectable thereto. Any application or module herein described may be implemented using computer-readable/executable instructions that may be stored or otherwise held by such computer-readable media.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Neurology (AREA)
  • Image Analysis (AREA)
  • Feedback Control In General (AREA)
  • Complex Calculations (AREA)
US16/884,128 2020-05-27 2020-05-27 Efficient tile mapping for row-by-row convolutional neural network mapping for analog artificial intelligence network inference Active 2041-05-22 US11562240B2 (en)

Priority Applications (11)

Application Number Priority Date Filing Date Title
US16/884,128 US11562240B2 (en) 2020-05-27 2020-05-27 Efficient tile mapping for row-by-row convolutional neural network mapping for analog artificial intelligence network inference
CN202180036838.4A CN115699028A (zh) 2020-05-27 2021-05-13 模拟人工智能网络推理的逐行卷积神经网络映射的高效瓦片映射
AU2021281628A AU2021281628B2 (en) 2020-05-27 2021-05-13 Efficient tile mapping for row-by-row convolutional neural network mapping for analog artificial intelligence network inference
GB2218705.8A GB2610774A (en) 2020-05-27 2021-05-13 Efficient tile mapping for row-by-row convolutional neural network mapping for analog artificial intelligence network inference
CA3178030A CA3178030A1 (en) 2020-05-27 2021-05-13 Efficient tile mapping for row-by-row convolutional neural network mapping for analog artificial intelligence network inference
KR1020227041560A KR20230005309A (ko) 2020-05-27 2021-05-13 아날로그 인공지능 네트워크 추론을 위한 행별 컨볼루션 신경망 매핑을 위한 효율적 타일 매핑
PCT/IB2021/054105 WO2021240286A1 (en) 2020-05-27 2021-05-13 Efficient tile mapping for row-by-row convolutional neural network mapping for analog artificial intelligence network inference
DE112021002939.8T DE112021002939T5 (de) 2020-05-27 2021-05-13 Effiziente kachel-zuordnung für zeilenweise zuordnung in neuronalen faltungsnetzen zur analogen inferenz in künstliche-intelligenz-netzen
IL297331A IL297331A (en) 2020-05-27 2021-05-13 Row-by-row mapping of an artificial convolutional neural network
JP2022568491A JP2023526915A (ja) 2020-05-27 2021-05-13 アナログ人工知能ネットワーク推論のための行単位畳み込みニューラル・ネットワーク・マッピングのための効率的なタイル・マッピング
US18/061,074 US11868893B2 (en) 2020-05-27 2022-12-02 Efficient tile mapping for row-by-row convolutional neural network mapping for analog artificial intelligence network inference

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US16/884,128 US11562240B2 (en) 2020-05-27 2020-05-27 Efficient tile mapping for row-by-row convolutional neural network mapping for analog artificial intelligence network inference

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/061,074 Continuation US11868893B2 (en) 2020-05-27 2022-12-02 Efficient tile mapping for row-by-row convolutional neural network mapping for analog artificial intelligence network inference

Publications (2)

Publication Number Publication Date
US20210374514A1 US20210374514A1 (en) 2021-12-02
US11562240B2 true US11562240B2 (en) 2023-01-24

Family

ID=78706397

Family Applications (2)

Application Number Title Priority Date Filing Date
US16/884,128 Active 2041-05-22 US11562240B2 (en) 2020-05-27 2020-05-27 Efficient tile mapping for row-by-row convolutional neural network mapping for analog artificial intelligence network inference
US18/061,074 Active 2040-07-29 US11868893B2 (en) 2020-05-27 2022-12-02 Efficient tile mapping for row-by-row convolutional neural network mapping for analog artificial intelligence network inference

Family Applications After (1)

Application Number Title Priority Date Filing Date
US18/061,074 Active 2040-07-29 US11868893B2 (en) 2020-05-27 2022-12-02 Efficient tile mapping for row-by-row convolutional neural network mapping for analog artificial intelligence network inference

Country Status (10)

Country Link
US (2) US11562240B2 (de)
JP (1) JP2023526915A (de)
KR (1) KR20230005309A (de)
CN (1) CN115699028A (de)
AU (1) AU2021281628B2 (de)
CA (1) CA3178030A1 (de)
DE (1) DE112021002939T5 (de)
GB (1) GB2610774A (de)
IL (1) IL297331A (de)
WO (1) WO2021240286A1 (de)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11562240B2 (en) 2020-05-27 2023-01-24 International Business Machines Corporation Efficient tile mapping for row-by-row convolutional neural network mapping for analog artificial intelligence network inference

Citations (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1345161A2 (de) 2002-03-15 2003-09-17 Microsoft Corporation System und Verfahren zur Unterstützung der Mustererkennung
US20170011290A1 (en) 2015-07-06 2017-01-12 University Of Dayton On-chip training of memristor crossbar neuromorphic processing systems
US9646243B1 (en) * 2016-09-12 2017-05-09 International Business Machines Corporation Convolutional neural networks using resistive processing unit array
US20170200078A1 (en) 2014-08-28 2017-07-13 Commissariat A L'energie Atomique Et Aux Energies Alternatives Convolutional neural network
CN107424647A (zh) 2017-08-03 2017-12-01 电子科技大学 一种基于忆阻器的语音存储与分类系统
US20180096226A1 (en) 2016-10-04 2018-04-05 Magic Leap, Inc. Efficient data layouts for convolutional neural networks
US20180253643A1 (en) * 2017-03-03 2018-09-06 Hewlett Packard Enterprise Development Lp Analog multiplier-accumulators
US20180275909A1 (en) 2017-03-27 2018-09-27 Oracle International Corporation Memory management for sparse matrix multiplication
US20190000122A1 (en) 2016-11-14 2019-01-03 Raphadeo Co.,Ltd Method for manufacturing processed ginseng containing honey component
US20190012296A1 (en) 2017-07-08 2019-01-10 British Cayman Islands Intelligo Technology Inc. Method for matrix by vector multiplication for use in artificial neural network
US20190080231A1 (en) 2017-09-08 2019-03-14 Analog Devices, Inc. Analog switched-capacitor neural network
CN109524039A (zh) 2018-11-21 2019-03-26 复旦大学 一种忆阻器阻态数目扩展的结构及相关方法
US10332004B2 (en) 2015-07-13 2019-06-25 Denso Corporation Memristive neuromorphic circuit and method for training the memristive neuromorphic circuit
US20190205741A1 (en) 2017-12-29 2019-07-04 Spero Devices, Inc. Digital Architecture Supporting Analog Co-Processor
US20190340508A1 (en) * 2017-01-13 2019-11-07 Huawei Technologies Co., Ltd. Computing Device and Computation Method for Neural Network Computation
US20200020393A1 (en) * 2018-07-11 2020-01-16 Sandisk Technologies Llc Neural network matrix multiplication in memory cells
US20200082252A1 (en) * 2018-09-07 2020-03-12 Tetramem Inc. Implementing a multi-layer neural network using crossbar array
US20200117986A1 (en) 2018-10-12 2020-04-16 International Business Machines Corporation Efficient processing of convolutional neural network layers using analog-memory-based hardware

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11475269B2 (en) 2015-12-15 2022-10-18 Analog Devices, Inc. Convolutional neural network
US10216703B2 (en) 2016-02-08 2019-02-26 Spero Devices, Inc. Analog co-processor
US10528864B2 (en) 2016-08-11 2020-01-07 Nvidia Corporation Sparse convolutional neural network accelerator
CN108182471B (zh) 2018-01-24 2022-02-15 上海岳芯电子科技有限公司 一种卷积神经网络推理加速器及方法
JP2020035502A (ja) 2018-08-28 2020-03-05 キオクシア株式会社 半導体集積回路
CN109460817B (zh) 2018-09-11 2021-08-03 华中科技大学 一种基于非易失存储器的卷积神经网络片上学习系统
US10642922B2 (en) 2018-09-28 2020-05-05 Intel Corporation Binary, ternary and bit serial compute-in-memory circuits
US10825509B2 (en) 2018-09-28 2020-11-03 Intel Corporation Full-rail digital read compute-in-memory circuit
US11562240B2 (en) 2020-05-27 2023-01-24 International Business Machines Corporation Efficient tile mapping for row-by-row convolutional neural network mapping for analog artificial intelligence network inference

Patent Citations (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1345161A2 (de) 2002-03-15 2003-09-17 Microsoft Corporation System und Verfahren zur Unterstützung der Mustererkennung
US20170200078A1 (en) 2014-08-28 2017-07-13 Commissariat A L'energie Atomique Et Aux Energies Alternatives Convolutional neural network
US20170011290A1 (en) 2015-07-06 2017-01-12 University Of Dayton On-chip training of memristor crossbar neuromorphic processing systems
US10332004B2 (en) 2015-07-13 2019-06-25 Denso Corporation Memristive neuromorphic circuit and method for training the memristive neuromorphic circuit
US9646243B1 (en) * 2016-09-12 2017-05-09 International Business Machines Corporation Convolutional neural networks using resistive processing unit array
US20180096226A1 (en) 2016-10-04 2018-04-05 Magic Leap, Inc. Efficient data layouts for convolutional neural networks
US20190000122A1 (en) 2016-11-14 2019-01-03 Raphadeo Co.,Ltd Method for manufacturing processed ginseng containing honey component
US20190340508A1 (en) * 2017-01-13 2019-11-07 Huawei Technologies Co., Ltd. Computing Device and Computation Method for Neural Network Computation
US20180253643A1 (en) * 2017-03-03 2018-09-06 Hewlett Packard Enterprise Development Lp Analog multiplier-accumulators
US20180275909A1 (en) 2017-03-27 2018-09-27 Oracle International Corporation Memory management for sparse matrix multiplication
US20190012296A1 (en) 2017-07-08 2019-01-10 British Cayman Islands Intelligo Technology Inc. Method for matrix by vector multiplication for use in artificial neural network
CN107424647A (zh) 2017-08-03 2017-12-01 电子科技大学 一种基于忆阻器的语音存储与分类系统
US20190080231A1 (en) 2017-09-08 2019-03-14 Analog Devices, Inc. Analog switched-capacitor neural network
US20190205741A1 (en) 2017-12-29 2019-07-04 Spero Devices, Inc. Digital Architecture Supporting Analog Co-Processor
US20200020393A1 (en) * 2018-07-11 2020-01-16 Sandisk Technologies Llc Neural network matrix multiplication in memory cells
US20200082252A1 (en) * 2018-09-07 2020-03-12 Tetramem Inc. Implementing a multi-layer neural network using crossbar array
US20200117986A1 (en) 2018-10-12 2020-04-16 International Business Machines Corporation Efficient processing of convolutional neural network layers using analog-memory-based hardware
CN109524039A (zh) 2018-11-21 2019-03-26 复旦大学 一种忆阻器阻态数目扩展的结构及相关方法

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
International Search Report and Written Opinion; dated Aug. 24, 2021; Application No. PCT/IB2021/054105; Filed May 13, 2021; 9 pages.
Valavi et al., "A Mixed-Signal Binarized Convolutional-Neural-Network Accelerator Integrating Dense Weight Storage and Multiplication for Reduced Data Movement," 2018 IEEE Symposium on VLSI Circuits, Honolulu, HI, 2018, pp. 141-142, doi: 10.1109/VLSIC.2018.8502421.

Also Published As

Publication number Publication date
US11868893B2 (en) 2024-01-09
WO2021240286A1 (en) 2021-12-02
GB202218705D0 (en) 2023-01-25
JP2023526915A (ja) 2023-06-26
US20230100139A1 (en) 2023-03-30
DE112021002939T5 (de) 2023-03-16
KR20230005309A (ko) 2023-01-09
GB2610774A (en) 2023-03-15
CA3178030A1 (en) 2021-12-02
AU2021281628B2 (en) 2023-04-27
CN115699028A (zh) 2023-02-03
US20210374514A1 (en) 2021-12-02
IL297331A (en) 2022-12-01
AU2021281628A1 (en) 2022-11-10

Similar Documents

Publication Publication Date Title
US10740671B2 (en) Convolutional neural networks using resistive processing unit array
US20200117986A1 (en) Efficient processing of convolutional neural network layers using analog-memory-based hardware
US10956815B2 (en) Killing asymmetric resistive processing units for neural network training
US10373051B2 (en) Resistive processing unit
US9779355B1 (en) Back propagation gates and storage capacitor for neural networks
US10708522B2 (en) Image sensor with analog sample and hold circuit control for analog neural networks
US11087204B2 (en) Resistive processing unit with multiple weight readers
US11501130B2 (en) Neural network hardware accelerator architectures and operating method thereof
US20210374546A1 (en) Row-by-row convolutional neural network mapping for analog artificial intelligence network training
US11868893B2 (en) Efficient tile mapping for row-by-row convolutional neural network mapping for analog artificial intelligence network inference
US11195089B2 (en) Multi-terminal cross-point synaptic device using nanocrystal dot structures
Wu et al. Bio-inspired computing with resistive memories—models, architectures and applications
Lu et al. NVMLearn: a simulation platform for non-volatile-memory-based deep learning hardware
US11250316B2 (en) Aggregate adjustments in a cross bar neural network
AU2021296187B2 (en) Suppressing undesired programming at half-selected devices in a crosspoint array of 3-terminal resistive memory
Yan et al. Neuromorphic Computing Systems with Emerging Nonvolatile Memories: A Circuits and Systems Perspective
US20230289563A1 (en) Multi-node neural network constructed from pre-trained small networks

Legal Events

Date Code Title Description
AS Assignment

Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW YORK

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:TSAI, HSINYU;BURR, GEOFFREY;NARAYANAN, PRITISH;SIGNING DATES FROM 20200506 TO 20200526;REEL/FRAME:052758/0906

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: EX PARTE QUAYLE ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE