JP2021532451A - Hierarchical parallel processing in the core network of distributed neural networks - Google Patents

Abstract

Hierarchical parallel processing is realized in a network of distributed neural cores. In various embodiments, a plurality of neural cores are provided. Each of the plurality of neural cores comprises a plurality of vector computing units configured to operate in parallel. Each of the plurality of neural cores is configured to compute the output activation in parallel by applying the plurality of vector calculation units to the input activation. A subset of the output activations of one layer of the neural network is assigned to each of the multiple neural cores to perform the calculation. Upon receiving a subset of the input activations of the layers of the neural network, each of the neural cores calculates a partial sum for each of its assigned output activations, and at least from the calculated partial sum, that allocation. Calculate the output activation that was done.

Description

The embodiments of the present disclosure relate to neural computations, and more particularly to hierarchical parallelism in a network of distributed neural cores.

According to embodiments of the present disclosure, a system for neural calculations is provided. In various embodiments, a plurality of neural cores are provided. Each of the plurality of neural cores comprises a plurality of vector computing units configured to operate in parallel. Each of the plurality of neural cores is configured to compute the output activation in parallel by applying the plurality of vector calculation units to the input activation. A subset of the output activations of one layer of the neural network is assigned to each of the multiple neural cores to perform the calculation. Upon receiving a subset of the input activations of the layers of the neural network, each of the neural cores calculates a partial sum for each of its assigned output activations, and at least from the calculated partial sum, that allocation. Calculate the output activation that was done.

According to the embodiments of the present disclosure, a method of neural calculation and a computer program product for neural calculation are provided. In various embodiments, each of the plurality of neural cores receives a subset of the input activations of the layers of the neural network. Each of the plurality of neural cores comprises a plurality of vector computing units configured to operate in parallel. Each of the plurality of neural cores is configured to compute the output activation in parallel by applying the plurality of vector calculation units to the input activation. A subset of the output activations of one layer of the neural network is assigned to each of the plurality of neural cores and calculated. Upon receiving a subset of the input activations of the layers of the neural network, each of the neural cores calculates a partial sum for each of its assigned output activations, and at least from the calculated partial sum, that allocation. Calculate the output activation that was done.

It is a figure which shows the neural core which concerns on embodiment of this disclosure. It is a figure which shows the exemplary neural core which shows the parallel processing between cores which concerns on embodiment of this disclosure. It is a figure which shows the exemplary neural core which shows the parallel processing in a core which concerns on embodiment of this disclosure. It is a figure which shows the method of the neural calculation which concerns on embodiment of this disclosure. It is a figure which drew the calculation node which concerns on embodiment of this disclosure.

An artificial neuron is a mathematical function whose output is a non-linear function of a linear combination of its inputs. If one output is an input to the other, the two neurons are connected. A weight is a scalar value that encodes the bond strength between the output of one neuron and the input of another neuron.

A neuron computes its output, called activation, by applying a non-linear activation function to the weighted sum of its inputs. The weighted sum is an intermediate result calculated by multiplying each input by the corresponding weight and accumulating the product. A partial sum is a weighted sum of a subset of inputs. By accumulating one or more partial sums, the weighted sum of all inputs can be calculated stepwise.

A neural network is a collection of one or more neurons. Neural networks are often divided into groups of neurons called layers. A layer is a collection of one or more neurons that all receive inputs from the same layer and all send outputs to the same layer, typically performing similar functions. The input layer is the layer that receives input from sources outside the neural network. The output layer is a layer that sends output to a target outside the neural network. All other layers are intermediate treatment layers. A multi-layer neural network is a neural network having two or more layers. A deep neural network is a multi-layer neural network having many layers.

A tensor is a multidimensional array of numbers. A tensor block is a continuous subarray of elements in a tensor.

Each neural network layer is associated with a parameter tensor V, a weighted tensor W, an input data tensor X, an output data tensor Y, and an intermediate data tensor Z. The parameter tensor includes all the parameters that control the neuron activation function σ in the layer. The weight tensor contains all of the weights that connect the inputs to the layer. The input data tensor contains all the data that the layer consumes as input. The output data tensor contains all the data that the layer calculates as output. The intermediate data tensor includes any data that the layer produces as intermediate calculated values, such as a subset.

The data tensors (inputs, outputs, and intermediates) for a layer may be three-dimensional, in which case the first two dimensions are interpreted as encoding spatial positions, and the third dimension is different. It can be interpreted as encoding a feature. For example, if the data tensor represents a color image, the first two dimensions encode the vertical and horizontal coordinates in the image, and the third dimension encodes the color at each position. Any element of the input data tensor X can be connected to any neuron by a separate weight, in which case the weighted tensor W is generally the three dimensions of the input data tensor (input row a, input column b, input features). It has 6 dimensions in which c) is concatenated with 3 dimensions of the output data tensor (output row i, output column j, output feature k). The intermediate data tensor Z has the same shape as the output data tensor Y. The parameter tensor V concatenates the three dimensions of the output data tensor with an additional dimension o that is the index of the parameters of the activation function σ.

The elements of the output data tensor Y of a certain layer can be calculated as in Equation 1, and in the equation, the neuroactivation function σ is composed of the vector V [i, j, k ,:] of the parameters of the activation function. , The weighted sum Z [i, j, k] can be calculated as in Equation 2.
Y [i, j, k] = σ (V [i, j, k ,:]; Z [i, j, k])
Equation 1

To simplify the notation, the weighted sum in Equation 2 is sometimes called the output, which is the linear activation function Y [i, j, k] = σ (Z [i, j, k]). It is understood that it is equivalent to using = Z [i, j, k] and the same explanation applies without loss of generality when different activation functions are used.

In various embodiments, the calculation of the output data tensor as described above is broken down into smaller problems. Each problem can be solved on one or more neural cores, or in parallel on one or more cores of a conventional multi-core system.

It will be clear from the above that neural networks are inherently parallel structures. Neurons in a given layer receive input X with _{element x i} from one or more layers or other inputs. Each neuron on the basis of the weight W having an input and an element w _i, to calculate the state Y∈Y. In various embodiments, the weighted sum of the inputs is adjusted by the bias b, and then the result is passed to the non-linearity F (.). For example, activation of a single neuron can be expressed as _{y = F (b + Σx i} w _i).

Since all neurons in a given layer receive inputs from the same layer and calculate their outputs independently, neuron activations can be calculated in parallel. This aspect of the overall neural network speeds up the overall computation by performing the computation in a parallel distributed core. Furthermore, vector operations can be calculated in parallel within each core. All neurons are also updated at the same time if there is a recursive input, for example if a layer is projected back to itself. In practice, the recursive connection is delayed to align with the next input to that layer.

Here, with reference to FIG. 1, a neural core according to an embodiment of the present disclosure is drawn. The neural core 100 is a tileable computational unit that computes one block of output tensors. The neural core 100 has M inputs and N outputs. In various embodiments, M = N. To calculate the output tensor block, the neural core multiplies the M × 1 input tensor block 101 by the M × N weighted tensor block 102 and accumulates the products to obtain a weighted sum. , This is stored in the 1 × N intermediate tensor block 103. The OxN parameter tensor block applies OxN parameter tensor blocks to each of the N neuron activation functions that are applied to the intermediate tensor block 103 to produce the 1xN output tensor block 105. Include.

Multiple neural cores can be tiled in an array of neural cores. In some embodiments, the array is two-dimensional.

A neural network model is a set of constants that collectively define the entire calculation performed by a neural network, including a graph of connections between neurons and parameters of weights and activation functions for every neuron. Training is the process of modifying a neural network model to perform the desired function. Inference is the process of applying a neural network to an input to produce an output without changing the neural network model.

An inference processing unit is a category of processors that perform neural network inference. The neural inference chip is a concrete physical example of an inference processing unit.

In embodiments that include multiple cores in an array, each core performs part of the overall neural network computation. Subscript _{i = 1:} the _{x i} with _{N INPUT} having _{N INPUT} inputs neurons value, subscript _{j = 1:} For a given layer having _{N NEURON} number of output neurons with _{N NEURON} The output value of that layer is given by Equation 3 for _{j = 1: N NEURON} and i = 1: N _INPUT.
Y _j = F (b _j + Σx _i w _ji )
Equation 3

In some embodiments, each core computes some number of output neurons. For N _CORE cores, each core calculates an average N N _EURON / N _CORE output neurons, each receiving all inputs. In this case, all cores require all inputs, but only some outputs are calculated. for example:
Core 1 calculates: j = 1: N N _EURON / N _CORE and i = 1: N _INPUT , Y _j = F (b _j + Σx _i w _ji )
Core 2 calculates: j = N N _EURON / N _CORE + 1: 2N N _EURON / N _CORE and i = 1: N _INPUT , Y _j = F (b _j + Σx _i w _ji )
Core k calculates the _{following: j = (k-1)} N NEURON / N CORE +1: kN NEURON / N CORE and _{i = 1:} against _{N INPUT, Y j = F (} b j + Σx i w ji)
The core _{N CORE} calculates the _{following: j = (N CORE -1)} N NEURON / N CORE +1: N NEURON and _{i = 1:} against _{N INPUT, Y j = F (} b j + Σx i w ji)

In some embodiments, each core computes all output neurons, but only for a subset of the inputs. For N _CORE cores, each core calculates all output neurons for _{an average N INPUT} / N _{CORE input neurons.} In this case, the core needs only some inputs, but the outputs need to be collected and added. for example:
Core 1 calculates: j = 1: N _NEURON and i = 1: N _INPUT / N _CORE , Y _{j_core1} = Σx _i w _ji
Core 2 calculates: j = 1: N _NEURON and i = N _INPUT / N _CORE + 1: 2N _INPUT / N _CORE , Y _{j_core2} = Σx _i w _ji
The core k calculates: j = 1: N _NEURON and i = (k-1) N _INPUT / N _CORE + 1: kN _NEURON / N _CORE , Y _{j_core} = Σx _i w _ji
The core N _CORE calculates: j = 1: N _NEURON and i = (N _CORE -1) N _INPUT / N _CORE +1: N _INPUT , Y _{j_core NCORE} = Σx _i w _ji

To reach a complete result, Y _j = F (b _j + _{ΣY i_corek} ) for _{j = 1: N NEURON} / N _CORE and k = 1: N _CORE. In various embodiments, complete results are calculated between or outside the core.

In some implementations, each core computes some number of output neurons, but not each core has access to all of the input neurons. In such an embodiment, each core calculates partial outputs and distributes them until each core has all of the partial outputs needed to fully calculate its set of neurons. In such an embodiment, the core only needs to pass non-zero information (partial sum), and the array can take advantage of the high-level structure of the neural network layer. For example, a convolutional neural network requires less input, output, and partial sum calculation and communication. for example:
Core 1 calculates: j = 1: N _NEURON and i = 1: N _INPUT / N _CORE , Y _{j_core1} = Σx _i w _ji
Core 2 calculates: j = 1: N _NEURON and i = N _INPUT / N _CORE + 1: 2N _INPUT / N _CORE , Y _{j_core2} = Σx _i w _ji
Core k calculates the _{following: j = 1: N NEURON} and _{i = (k-1) N} INPUT / N CORE +1: relative _{kN NEURON / N core, Y j_corek} = Σx i w ji
The core N _CORE calculates: j = 1: N _NEURON and i = (N _CORE -1) N _INPUT / N _CORE +1: N _INPUT , Y _{j_core NCORE} = Σx _i w _ji

The complete results are then calculated either continuously or overlapping. For j = 1: N _NEURON / N _CORE and k = 1: N _CORE , Y _j = F (b _j + _{ΣY i_core} ).
Core 1 calculates: j = 1: N N _EURON / N _CORE and k = 1: N _CORE , Y _j = F (b _j + _{ΣY j_corek} )
Core 2 calculates: j = N N _EURON / N _CORE + 1: 2N N _EURON / N _CORE and k = 1: N _CORE , Y _j = F (b _j + _{ΣY j_core} )
The core k calculates: j = (k-1) N N _EURON / N _CORE + 1: kN _{N EURON} / N _CORE and k = 1: N _CORE , Y _j = F (b _j + _{ΣY j_core} )
The core N _core calculates: j = (N _core -1) N N _EURON / N _CORE + 1: N N _EURON and k = 1: N _CORE , Y _j = F (b _j + _{ΣY j_core} )

を受け取り、これらにパラメータ

を乗算するものであり、ここで、Ｍ＝Ｎ_{ＩＮＰＵＴ}／Ｎ_ＣＯＲＥである。行列乗算の結果は並列和および非線形処理ユニット２０５に提供され、非線形処理ユニット２０５は、アレイ中の他のコアから部分和入力Ｙ_{ｉ＿ｃｏｒｅｋ}、２０６を受け取る。 Here, with reference to FIG. 2, an exemplary neural core showing parallel processing between cores is shown. The core 201 includes a crossbar 202 with a parallel vector calculation unit, the crossbar 202 having an input 203,

Receives these parameters

Is multiplied by, where M = N _INPUT / N _CORE . The result of the matrix multiplication is provided to the parallel sum and nonlinear processing unit 205, which receives the _{partial sum inputs Y i_core, 206 from the other cores in the array.}

As pointed out above, each core in the array contains parallel / simultaneous neural network compute elements. For example, a given core k calculates:
For j = 1: N _NEURON and i = (k-1) N _INPUT / N _CORE + 1: kN _NEURON / N _CORE , Y _{j_core} = Σx _i w _ji
Or calculate:
For j = (k-1) N N _EURON / N _CORE + 1: kN _{N EURON} / N _CORE and k = 1: N _CORE , Y _j = F (b _j + _{ΣY j_core} )
Or both are calculated.

The partial sum of N _{NEURON (Y j_core} ) can be calculated simultaneously in separate vector units, where each calculated vector multiplies and adds (Σx _i w _ji ) to a single neuron j. The sum and non-linear processing of N _{NEURON (Y j} ) can be calculated simultaneously in separate accumulation and non-linear units, each performing a calculation for a single neuron j.

Here, with reference to FIG. 3, an exemplary neural core showing in-core parallelism is shown. The core 301 includes a vector multiplication / addition unit 302 (corresponding to one of the units 202 in FIG. 2), where the vector multiplication / addition unit 302 has an input 303, [x ₁ , ..., X ₈ ]. _Is received, and these are multiplied by the parameters 304, [w _j1 , ..., w j8]. The result of the matrix multiplication is provided to the parallel sum and nonlinear processing unit 305, which receives the partial sum input 306 from the other cores in the array and provides the neuron output 307. In this way, a given vector unit acts on many inputs in unison to produce a single output or multiple outputs.

As mentioned above, each vector unit performs calculations in a parallel / simultaneous fashion. For example, the jth vector unit of the kth core calculates _{Y j_corek} = Σx _i w _ji. This operation can be performed in parallel / simultaneously. Vector units perform parallel addition, for example using an addition tree, which can be pipelined for further simultaneity. For example, if i = 1: 8 (as shown), the vector unit performs eight parallel multiplications, followed by four two-input additions, followed by two two-input additions, and so on. Perform two input additions at one time. These four parallel operations can be pipelined and executed simultaneously to obtain higher throughput.

Referring to Table 1, various core arrays for the size of the core array (columns 1x1 ... 64x64) and the size of the crossbars (rows 1x1 ... 1024x1024). The total value of parallel processing is shown. The sum of parallelism is the product of the size of the core array and the size of the crossbar.

Referring to FIG. 4, a method of neural calculation according to an embodiment of the present disclosure is shown. At 401, each of the plurality of neural cores receives a subset of the input activations of the layers of the neural network. Each of the plurality of neural cores comprises a plurality of vector computing units configured to operate in parallel. Each of the plurality of neural cores is configured to compute the output activation in parallel by applying the plurality of vector calculation units to the input activation. At 402, a subset of the output activations of one layer of the neural network is assigned to and calculated for each of the plurality of neural cores. Upon receiving a subset of the input activations of the layers of the neural network, each of the multiple neural cores calculated a partial sum for each of its assigned output activations at 403 and at least calculated at 405. Calculate the assigned output activation from the subset. In some embodiments, each of the plurality of cores receives, at 404, a partial sum for at least one of its assigned output activations from another one of the plurality of neural cores. At 405, the assigned output activation is calculated from the calculated partial sum and the received partial sum.

It will be appreciated that the neural networks provided herein can be used as classifiers or generators.

Here, with reference to FIG. 5, a schematic diagram of an example of a compute node is shown. Computational node 10 is merely an example of a suitable computing node and is not intended to suggest any limitation with respect to the use or scope of functionality of the embodiments described herein. In any case, the compute node 10 can be implemented, can perform either of the above-mentioned functionality herein, or both.

At the compute node 10, there is a computer system / server 12 that can operate with a number of general purpose or dedicated compute system environments or configurations. Examples of well-known computing systems, environments, or configurations or combinations thereof that may be suitable for use with the computer system / server 12 include personal computer systems, server computer systems, thin clients, and the like. Chic clients, portable or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframes. Computer systems and, but are not limited to, distributed cloud computing environments including, but not limited to, any of the above systems or devices.

The computer system / server 12 may be described in the general context of a computer system executable instruction executed by the computer system, such as a program module. In general, a program module can include routines, programs, objects, components, logic, data structures, etc. that perform a particular task or implement a particular abstract data type. The computer system / server 12 may be implemented in a distributed cloud computing environment in which tasks are performed by remote processing devices linked over a communication network. In a distributed cloud computing environment, program modules can be located within computer system storage media, including both local and remote memory storage devices.

As shown in FIG. 5, the computer system / server 12 in the compute node 10 is shown in the form of a general purpose computing device. The components of the computer system / server 12 may include one or more processors or processing units 16, system memory 28, and a bus 18 connecting various system components including system memory 28 to processor 16. However, it is not limited to these.

Bus 18 is of several types, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus that uses one of a variety of bus architectures. Represents one or more of the bus structures of. Examples include, but are not limited to, industry standard architecture (ISA) buses, microchannel architecture (MCA) buses, enhanced ISA (EISA) buses, and the American Video Electronics Standards Association. (Video Electronics Standards Association; VESA) Local Bus, Peripheral Component Interconnects (PCI) Bus, Peripheral Component Interconnect (Peripheral Component Interconnect) Bus, Peripheral Component Interconnect (Peripheral Component Interconnect) Microcontroller Bus Archive (AMBA) can be mentioned.

The computer system / server 12 typically includes various computer system readable media. Such media may be any available medium accessible to the computer system / server 12, including both volatile and non-volatile media, removable and non-removable media. included.

The system memory 28 may include a computer system readable medium in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer system / server 12 may further include other removable / non-removable volatile / non-volatile computer system storage media. As a mere example, a storage system 34 may be provided for reading and writing to a non-removable non-volatile magnetic medium (not shown, but typically referred to as a "hard drive"). Although not shown, magnetic disk drives for reading and writing to removable non-volatile magnetic disks (eg, "floppy (R) disks"), as well as CD-ROMs, DVD-ROMs, or others. An optical disk drive for reading and writing to a removable non-volatile optical disk such as an optical medium can be provided. In such an example, each may be connected to the bus 18 by one or more data medium interfaces. As further described and described below, the memory 28 has at least one program product having a set (eg, at least one) program module configured to perform the functions of the embodiments of the present disclosure. May include.

A program / utility 40 having a set (at least one) program module 42 may be stored in memory 28 as an example, but not limited to, an operating system, one or more application programs, and other programs. It can also be stored in modules and program data. The operating system, one or more application programs, other program modules, and each of the program data, or any combination thereof, may include an implementation of a networking environment. The program module 42 generally implements the functions or methodologies or combinations thereof of the embodiments described herein.

The computer system / server 12 may also be one or more external devices 14, such as a keyboard, pointing device, display 24, etc., one or more devices that allow interaction between the user and the computer system / server 12. Alternatively, it may communicate with any device (eg, network card, modem, etc.) that allows communication between the computer system / server 12 and one or more other computing devices, or a combination thereof. Such communication can be done via the input / output (I / O) interface 22. Furthermore, the computer system / server 12 may be a local area network (LAN), a general wide area network (WAN), or a public network (eg, the Internet), or a public network thereof (eg, the Internet), via a network adapter 20. It may communicate with one or more networks, such as a combination. As depicted, the network adapter 20 communicates with other components of the computer system / server 12 via bus 18. Although not shown, it should be understood that other hardware and / or software components may be used in combination with the computer system / server 12. Examples include, but are not limited to, microcodes, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems, etc. ..

The present disclosure may be embodied as a system, method, or computer program product or a combination thereof. The computer program product may include a computer-readable storage medium having computer-readable program instructions for causing the processor to perform aspects of the present disclosure.

The computer-readable storage medium can be a tangible device that can hold and store the instructions used by the instruction execution device. The computer-readable storage medium can be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof, but is not limited thereto. .. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erase. Possible Programmable Read-Only Memory (EPROM or Flash Memory), Static Random Access Memory (SRAM), Portable Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD), Memory Stick , Floppy (R) disks, mechanically encoded devices such as punch cards or raised structures in grooves on which instructions are recorded, and any suitable combination of the above. Computer-readable storage media as used herein are radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, optical pulses through optical fiber cables), or wiring. It should not be construed as a temporary signal itself, such as an electrical signal transmitted via.

The computer-readable program instructions described herein are from computer-readable storage media to their respective computing / processing devices or networks such as the Internet, local area networks, wide area networks, or wireless networks. , Or a combination thereof, may be downloaded to an external computer or external storage device. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and sends those computer-readable program instructions to the computer-readable storage medium within each computing / processing device. Transfer to be saved.

The computer-readable program instructions for performing the operations of the present disclosure are assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcodes, firmware instructions, state setting data, or Smalltalk (R). ), Object-oriented programming languages such as C ++, and any combination of one or more programming languages, including traditional procedural programming languages such as the "C" programming language or similar programming languages. It can be either code or object code. Computer-readable program instructions are exclusively on the user's computer, partly on the user's computer as a stand-alone software package, partly on the user's computer and partly on a remote computer, or exclusively. It can be run on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or externally. You may make a connection to your computer (eg, over the Internet using an Internet service provider). In some embodiments, electronic circuits including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) are computer readable to perform aspects of the present disclosure. By using the state information of a program instruction, a computer-readable program instruction can be executed to personalize an electronic circuit.

Aspects of the present disclosure are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present disclosure. It will be appreciated that each block of the flow chart and / or block diagram, and the combination of blocks in the flow chart and / or block diagram, can be performed by computer-readable program instructions.

These computer-readable program instructions are functions / operations in which an instruction executed through the processor of a computer or other programmable data processing device is specified in one or more blocks of a flowchart, a block diagram, or both. It may be provided to a general purpose computer, a dedicated computer, or the processor of another programmable data processing device to create a machine to create a means of implementation. These computer-readable program instructions also include instructions in which the computer-readable storage medium in which the instructions are stored performs the mode of function / operation specified in one or more blocks of the flowchart and / or block diagram. As provided with the product, it may be stored on a computer-readable storage medium and capable of instructing a computer, programmable data processing device, or other device, or a combination thereof, to function in a particular manner.

Computer-readable program instructions also perform a function / operation in which an instruction executed on a computer, other programmable device, or other device is specified in one or more blocks of a flowchart, a block diagram, or both. As such, it is loaded onto a computer, other programmable data processing device, or other device to create a process that is executed by the computer, and a series of operating steps on the computer, other programmable device, or other device. It may be something to be executed.

The flowcharts and block diagrams in the figure describe the architecture, functionality, and operation of possible implementations of the systems, methods, and computer program products according to the various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram can represent a module, segment, or part of an instruction that comprises one or more executable instructions for performing a given logical function. In some alternative implementations, the functions described within the block may be performed in a different order than that shown in the figure. For example, two blocks shown in succession can actually be executed substantially in parallel, or these blocks can sometimes be executed in reverse order, depending on the function involved. Also, each block of the block diagram and / or flow chart, and the combination of blocks in the block diagram and / or flow chart, performs the specified function or operation, or performs a combination of dedicated hardware and computer instructions. It will also be noted that it can be implemented by a dedicated hardware-based system.

In various embodiments, one or more inference processing units (not shown) are connected to the bus 18. In such an embodiment, the IPU receives data from memory 28 via bus 18 or writes data to memory 28. Similarly, the IPU can interact with other components via the bus 18 as described herein.

Descriptions of the various embodiments of the present disclosure have been presented for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and changes will be apparent to those of skill in the art without departing from the scope of the embodiments described. The terminology used herein is to best describe the principles of the embodiment, the actual application, or the technical improvements to the techniques found in the market, or disclosed herein by others. Selected so that the embodiments to be made can be understood.

Claims (25)

It's a neural inference chip
With multiple neural cores, each with multiple vector computing units configured to operate in parallel,
Each of the plurality of neural cores is configured to calculate the output activation in parallel by applying the plurality of vector calculation units to the input activation.
Each of the plurality of neural cores is assigned a subset of the output activation of a layer of the neural network to perform the calculation.
Neural inference chip. Upon receiving a subset of the input activations of said layer of the neural network, each of the plurality of neural cores
Calculate the partial sum for each of its assigned output activations and
The neural inference chip of claim 1, wherein at least the allocated output activation is calculated from the calculated partial sum. Upon receiving a subset of the input activations of said layer of the neural network, each of the plurality of neural cores
Receives a partial sum for at least one of its assigned output activations from another one of the plurality of neural cores.
The neural inference chip of claim 2, wherein the assigned output activation is calculated from the calculated partial sum and the received partial sum. The neural inference chip according to claim 1, wherein the vector calculation unit includes a multiplication and addition unit. The neural inference chip according to claim 1, wherein the vector calculation unit includes a cumulative unit. The neural inference chip according to claim 2, wherein the plurality of neural cores perform the calculation of the partial sum in parallel. The neural inference chip according to claim 2, wherein the plurality of neural cores perform the calculation of the output activation in parallel. The neural inference chip of claim 2, wherein calculating the partial sum comprises applying at least one of the plurality of vector calculation units to multiply the input activation by a synaptic weight. .. The neural inference chip of claim 2, wherein calculating the assigned output activation comprises applying a plurality of addition units. The neural inference chip of claim 2, wherein calculating the output activation comprises applying a non-linear function. The vector calculation unit is
Perform multiple multiplication operations in parallel
Perform multiple additions in parallel,
The neural inference chip according to claim 2, which is configured to accumulate the partial sums. The neural inference chip according to claim 2, wherein the plurality of vector calculation units are configured to calculate partial sums in parallel. The neural inference chip of claim 1, wherein the calculation by each of the plurality of neural cores is pipelined. 13. The neural inference chip of claim 13, wherein each of the plurality of neural cores is configured to simultaneously perform each stage of the calculation. The neural inference chip of claim 14, wherein the calculation maintains parallel processing. Each of the plurality of neural cores receives a subset of the input activations of a layer of the neural network, each of the plurality of neural cores being configured to operate in parallel. Each of the plurality of neural cores comprising a vector computing unit is configured to compute the output activation in parallel by applying the plurality of vector multipliers to the input activation, said receiving. When,
To perform the calculation by assigning a subset of the output activation of a layer of the neural network to each of the plurality of neural cores.
Upon receiving a subset of the input activations of the layer of the neural network, each of the plurality of neural cores
Calculate the partial sum for each of its assigned output activations and
At least to calculate the assigned output activation from the calculated partial sum,
Including, how. Upon receiving a subset of the input activations of said layer of the neural network, each of the plurality of neural cores
Receives a partial sum for at least one of its assigned output activations from another one of the plurality of neural cores.
16. The method of claim 16, wherein the assigned output activation is calculated from the calculated partial sum and the received partial sum. 16. The method of claim 16, wherein the vector calculation unit comprises a multiplication and addition unit. 16. The method of claim 16, wherein the vector calculation unit comprises a cumulative unit. 16. The method of claim 16, wherein the plurality of neural cores perform the partial sum calculation in parallel. 16. The method of claim 16, wherein the plurality of neural cores perform the calculation of output activation in parallel. 16. The method of claim 16, wherein calculating the partial sum comprises applying at least one of the plurality of vector calculation units to multiply the input activation by a synaptic weight. 16. The method of claim 16, wherein calculating the assigned output activation comprises applying a plurality of accrual units. 16. The method of claim 16, wherein calculating output activation comprises applying a non-linear function. Performing multiple multiplication operations in parallel and
Performing multiple additions in parallel and
Accumulating the partial sums and
16. The method of claim 16.

Images