JP7171622B2 - Neural network accelerator with on-chip resident parameters - Google Patents

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is related to U.S. Patent Application Serial No. 62/544, entitled "Neural Network Accelerator with Parameters Resident on Chip," filed Aug. 11, 2017. , 171, the entire disclosure of which is hereby incorporated by reference.

BACKGROUND This specification generally relates to neural network (NN) computational tiles for computation of deep neural network (“DNN”) layers.

Overview In general, one innovative aspect of the subject matter described in this specification can be implemented in storing neural network parameters on an accelerator. Neural networks differ from normal computational workloads in that their working set, ie the total amount of storage required for the entire computation, is really limited. Roughly, this working set typically corresponds to a number of parameters ranging from hundreds of thousands to billions. This amount of storage is consistent with existing hardware storage technology.

Despite these facts, current accelerators include local storage for parameters through which parameters pass. For example, parameters do not permanently reside on the chip. Rather, parameters flow from external memory for each new inference.

External memory bandwidth is therefore a key limitation of all neural network (NN) accelerators. The embodiments described herein replace temporary local storage for parameters with on-chip storage for parameters. That is, the embodiment keeps all parameters of the NN resident in the accelerator and not flushed from external memory.

The advantage of storing the parameters on-chip is that the performance limitations of NN accelerators can be overcome and, as performance limitations are overcome, it dramatically facilitates increasing the number of multiply-accumulate (“MAC”) operators. and providing a low power neural network accelerator, since external memory accesses typically require at least an order of magnitude more energy than local memory accesses.

In certain embodiments, the accelerator comprises a computing unit. The computation unit comprises a first memory bank for storing input or output activations and a second memory bank for storing neural network parameters used in performing the computation, the second memory bank is configured to store a sufficient amount of neural network parameters on the computational unit to allow delay below a certain level and throughput above a certain level for a given NN model and architecture; The calculation unit further includes at least one cell including at least one MAC operator for receiving parameters from the second memory bank and performing calculations, and a first traversal unit in data communication with the at least first memory bank. and the first traversal unit is configured to provide control signals to the first memory bank to provide input activation to the data bus accessible by the MAC operator. The accelerator performs one or more computations associated with at least one element of the data array, the one or more computations being performed by the MAC operator and in part based on input activations received from the data bus. and a parameter received from the second memory bank. If the parameter storage is sufficient to hold all neural network parameters, the performance of the accelerator is not determined by memory bandwidth. In that case, it is possible to supply all MACs with parameters every cycle.

Another innovative aspect of the subject matter described in this specification can be implemented in a computer-implemented method for accelerating tensor computation. The computer-implemented method includes sending a first input activation by the first memory bank in response to the first memory bank receiving a control signal, the first input activation being the data sent by the bus, the method further comprising receiving one or more parameters from a second memory bank for storing neural network parameters used in performing the computation by the at least one MAC operator. , a second memory bank to store a sufficient amount of neural network parameters on the computational unit to allow latency below a certain threshold and throughput above a certain threshold for a given NN model and architecture. and the method further includes performing, by the MAC operator, one or more computations associated with at least one element of the data array, the one or more computations being performed, in part, from the data bus. A multiplication operation of at least a first input activation accessed and at least one parameter received from a second memory bank.

Another innovative aspect of the subject matter described in this specification can be implemented in a method for accelerating computation. The method comprises the steps of loading neural network weight parameters into a neural network accelerator prior to execution, and processing inputs to the accelerator at runtime substantially without accessing the neural network weight parameters external to the accelerator. include. The method comprises the steps of: loading the neural network weight parameters onto multiple tightly coupled accelerator dies if the number of neural network weight parameters is too large to fit on one accelerator die; and processing the input to the accelerator die without substantial access to external neural network weight parameters.

The subject matter described in this specification can be implemented in particular embodiments to achieve one or more of the following advantages. Using registers to keep track of memory address values allows a program to iterate through deeply nested loops in a single instruction. Tensors accessible from narrow memory units and wide memory units in a single computational tile are traversed based on memory address values retrieved from registers. Memory address values correspond to elements of the tensor. Tensor computation occurs in individual computational tiles based on deep loop nesting execution. Computation can be distributed across multiple tiles. Computational efficiency is improved and accelerated based on distributing the multi-layer neural network's tensor computations over several computational tiles. You can traverse tensors and perform tensor computations with fewer instructions.

The embodiments described herein affect the operation and design of Neural Network (NN) accelerators. Embodiments address one of the significant limitations of neural network accelerator design: low delay combined with high throughput. By having the parameters reside on-chip, the delay is dramatically reduced given the high throughput and specific NN models and architectures. Accelerators do not require high memory bandwidth and are energy efficient.

Tiling as described in this specification provides compiled-wise local referentiality. For example, placing a fully coupled model next to SRAM results in greater internal bandwidth compared to a cached model. The embodiments described herein operate faster than convolutional neural network accelerators. Certain embodiments have even more operators. To feed the operators, the accelerator needs more internal bandwidth. To deal with this, the architecture needs to distribute the memory and collect the parameters on the accelerator.

The subject matter described in this specification may also be implemented in particular embodiments to achieve other advantages. For example, by employing a memory hierarchy that combines narrow low-bandwidth memories with high-bandwidth wide memories, high utilization of MAC operators for very different dimensions of DNN layers and local referentiality can be achieved. can be achieved. A narrow, low-bandwidth memory allows addressing flexibility for traversing a multi-dimensional array in any order.

Other implementations of this and other aspects include corresponding systems, apparatus and computer programs encoded on a computer storage device and configured to perform the actions of the method. A system of one or more computers can be so configured by software, firmware, hardware, or a combination thereof that is installed on the system and that in operation causes the system to perform actions. One or more computer programs may be so configured by having instructions that, when executed by a data processing apparatus, cause the apparatus to perform actions.

The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects and advantages of the subject matter will become apparent from the description, drawings and claims.

1 is a block diagram of an exemplary computing system; FIG. 4 shows an exemplary neural network computational tile; 2 shows an exemplary tensor traversal unit (TTU) structure. 1 illustrates an exemplary architecture including narrow memory units providing input activations for one or more multiply-accumulate (MAC) operators. 5 illustrates an exemplary architecture including an output bus providing output activation to the narrow memory units of FIGS. 2 and 4; FIG. FIG. 3 is an exemplary flow chart of a process for performing tensor computations using the neural network computational tile of FIG. 2; FIG. 4 is another embodiment of a computing system; FIG. 11 is another embodiment of a neural network computational tile; FIG. FIG. 4 is an exemplary flowchart of a process for accelerating computation by loading neural network weight parameters into a neural network accelerator; FIG.

Like reference numbers and designations in the various drawings indicate like elements.
DETAILED DESCRIPTION The subject matter described herein relates to methods for accelerating computation. The method comprises the steps of loading neural network weight parameters into a neural network accelerator prior to execution, and processing inputs to the accelerator at runtime substantially without accessing the neural network weight parameters external to the accelerator. include. The method comprises the steps of: loading the neural network weight parameters onto multiple tightly coupled accelerator dies if the number of neural network weight parameters is too large to fit on one accelerator die; and processing the input to the accelerator die without substantial access to external neural network weight parameters.

The subject matter described herein also relates to a hardware computing system including a plurality of computing units configured to accelerate machine learning inference workloads of neural network layers. Each computational unit of the hardware computing system is self-contained and can independently perform the computations required by a given layer of the multilayer neural network. Overall, this document describes deep computation using on-chip resident parameters to enable delay below a certain level and throughput above a certain level for a given NN model and architecture. A neural network (NN) computational tile for computation of neural network (“Deep Neural Network”) layers.

Inference can be computed using a neural network with multiple layers. For example, given an input, a neural network can compute an inference for that input. A neural network computes this inference by processing the input through each layer of the neural network. Specifically, each layer of the neural network has its own set of weights. Each layer receives an input and processes the input according to the set of weights for that layer to produce an output.

Thus, to compute inferences from the received inputs, a neural network receives an input, processes it through each neural network layer to produce inferences, and outputs from one neural network layer to the next neural network layer. given as an input to A data input or output for a neural network layer, eg, an input to the neural network or an output of a layer below that layer in the sequence, can be referred to as an activation of that layer.

In some implementations, the layers of the neural network are arranged in sequence. In other implementations, the layers are arranged in a directed graph. That is, any particular layer can receive multiple inputs, multiple outputs, or both. Layers of a neural network can also be configured so that the output of one layer can be sent back as an input to the previous layer.

The hardware computational system described herein can perform neural network layer computations by distributing tensor computations across multiple computational tiles. A computational process performed within a neural network layer may involve multiplication of an input tensor containing input activations and a parameter tensor containing weights. The computation involves multiplying the input activations by the weights in one or more cycles and performing product accumulation over many cycles.

Tensors are multidimensional geometric objects, and exemplary multidimensional geometric objects include matrices and data arrays. In general, processing is performed by computation tiles to perform tensor computations by processing nested loops to traverse an N-dimensional tensor. In one exemplary computational process, each loop may be responsible for traversing a particular dimension of an N-dimensional tensor. For a given tensor construct, a computational tile may need access to the elements of that tensor in order to perform multiple dot product computations associated with a particular tensor. A computation is performed when the input activations provided by the narrow memory structures are multiplied by the parameters or weights provided by the wide memory structures. Since tensors are stored in memory, a set of tensor indices may require translation into a set of memory addresses. In general, the tensor traversal unit of a computational tile performs control operations that give the index of each dimension associated with the tensor and the order in which index elements are traversed and computations are performed. The tensor computation ends when the multiplication result is written to the output bus and stored in memory.

FIG. 1 shows a block diagram of an exemplary computational system 100 for accelerating tensor computations associated with deep neural networks (DNNs). System 100 generally includes a controller 102, a host interface 108, an input/output (I/O) link 110, a plurality of tiles including a first tileset 112 and a second tileset 114, a classifier portion 116, and a bus map. It includes a data bus identified at 118 (shown for clarity but not included in system 100). Controller 102 generally includes data memory 104, instruction memory 106, and at least one processor configured to execute one or more instructions encoded on a computer-readable storage medium. Instruction memory 106 may store one or more machine-readable instructions executable by one or more processors of controller 102 . Data memory 104 may be any of a variety of data storage media for storing and subsequently accessing various data related to computations occurring within system 100 .

Controller 102 is configured to execute one or more instructions related to tensor computation within system 100 , including instructions stored in instruction memory 106 . In some implementations, data memory 104 and instruction memory 106 are volatile memory unit(s). In some other implementations, data memory 104 and instruction memory 106 are non-volatile memory unit(s). Data memory 104 and instruction memory 106 may also include floppy disk drives, hard disk drives, optical disk drives, or tape drives, flash memory or other similar solid-state memory devices, or devices in storage area networks or other configurations. It may also be another form of computer readable medium, such as an array of devices comprising. In various implementations, controller 102 may be referred to or called as core manager 102 .

As shown, host interface 108 is coupled to I/O link 110 , controller 102 , and classifier portion 116 . Host interface 108 receives instructions and data parameters from I/O link 110 and provides instructions and parameters to controller 102 . In general, instructions may be provided to one or more devices within system 100 via command bus 124 (described below), and parameters may be provided to one or more devices within system 100 via ring bus 128 (described below). can be given to In some implementations, instructions are initially received by controller 102 from host interface 118 and stored in instruction memory 106 for subsequent execution by controller 102 .

Classifier portion 116 is similarly coupled to controller 102 and tile 7 of second tileset 114 . In some implementations, classifier portion 116 is implemented as a separate tile within system 100 . In alternative implementations, classifier portion 116 is located or located within controller 102 as a sub-circuit or sub-device of controller 102 . Classifier portion 116 is generally configured to perform one or more functions on the accumulated pre-activation values received as the output of the fully connected layer. A fully connected layer may be split across the tiles in tilesets 112 and 114 . Thus, each tile is configured to generate a subset of pre-activation values (ie, linear outputs) that can be stored in the tile's memory unit. Classification result bus 120 provides a data path from classifier portion 116 to controller 102 . Data, including post-function values (ie, results), are provided from classifier portion 116 to controller 102 via classification result bus 120 .

Bus map 118 indicates data buses that provide one or more interconnected data communication paths between tiles of first tileset 112 and second tileset 114 . Bus map 118 provides a description of the codes used to identify classification result bus 120, CSR/master bus 122, command bus 124, mesh bus 126, and ring bus 128, as shown in FIG. In general, tiles are a core component within the accelerator architecture of system 100 and are the focus of tensor computations occurring within the system. Each tile is an individual computational unit that correlates with other tiles in the system and can accelerate computation across one or more layers of a multilayer neural network. Although tiles within tilesets 112, 114 may share execution of tensor computations associated with a given instruction, individual computational units are independent with respect to other corresponding tiles within tilesets 112, 114. It is a self-contained computational component configured to perform a subset of tensor computations as

CSR bus 122 is a single-master, multiple-slave bus that allows controller 102 to send one or more instructions to set program configurations and read status registers associated with one or more tiles. The CSR bus 122 may be connected in a single daisy chain configuration with one master bus segment and multiple slave bus segments. As shown in FIG. 1, CSR bus 122 provides communication coupling the tiles of tilesets 112, 114 and controller 102 via a bus data path that connects to host interface 110 in a ring. In some implementations, host interface 110 is the single master of the CSR bus ring and the entire CSR bus address space is memory mapped into memory space within host interface 110 .

CSR bus 122 may, for example, program a memory buffer pointer within controller 102 to allow controller 102 to begin fetching instructions from instruction memory 106, be quiet during one or more computations. updating/programming various tile settings (e.g., coefficient table for polynomial approximation calculations) that remain static and/or loading/reloading firmware for the classifier portion 116 or may be used by the host interface 110 to perform operations. In one example, a firmware reload may include a new function to be applied to the linear output (ie pre-activation value). Thus, every slave that has access to the CSR bus 122 will have a distinct node identifier (node ID) attached to it to identify it. The node ID is part of the instruction address and is used, examined, or used by the CSR slave (i.e., controller 102, tiles 112, 114 and classifier 116) to determine if the CSR packet is addressed to the slave. can be examined in other ways.

In some implementations, one or more instructions may be sent by host interface 102 through controller 102 . Instructions may be, for example, 32 bits wide, with the first 7 bits containing header information indicating the instruction address/destination where the instruction is to be received and executed. The first 7 bits of the header may contain a data parameter representing a particular node ID. Thus, a slave (e.g. each tile) on the CSR bus ring can inspect the header of the instruction to determine if a request by the master (host interface 110) is addressed to the tile that inspects the header. . If the node ID in the header does not indicate that the destination is a test tile, the test tile copies the incoming CSR command packet to the CSR bus input connected to the next tile for inspection by the next tile.

Command bus 124 provides communication originating from controller 102 and, like CSR bus 122, coupling the tiles in tilesets 112, 114 in a ring and back to controller 102 via a bus data path. In one implementation, controller 102 broadcasts one or more instructions over instruction bus 124 . The instructions broadcast by controller 102 may differ from the instructions provided via CSR bus 122 . However, the manner in which tiles receive and/or consume or execute instructions received via bus 124 may be similar to the process for executing instructions received via CSR bus 122 .

In one example, an instruction's header (ie, bitmap) indicates to a receiving tile that the receiving tile should consume a particular instruction based on the bitmap associated with that instruction. A bitmap may have a specific width defined in terms of bits. Instructions are typically transferred from one tile to the next based on the parameters of the instruction. In one implementation, the width of instruction bus 124 may be configured to be smaller than the size/width of an instruction. Thus, in such a configuration, instruction transmission occurs over several cycles, and a bus stop on instruction bus 124 directs the decoder to place the instruction received on that tile into the appropriate target instruction buffer associated with that tile. have.

Tiles in tilesets 112, 114 are generally configured to support two broad categories of instructions, as described further below. The two broad categories are also called instruction types. Instruction types include tensor operation (TensorOp) instructions and direct memory access (DMAOp) instructions. In some implementations, a DMAOp instruction has one or more specializations that are allowed to be concurrent. One or more specializations are sometimes referred to as DMAOp instruction subtypes or opcodes. In some cases, all unique and/or valid DMAOp instruction type/subtype tuples will have separate instruction buffers within a particular tile.

In a particular tile of tiles 112, 114, the bus stop associated with instruction bus 124 examines the header bitmap to determine the instruction type/subtype. Instructions may be received by a tile and subsequently written to the tile's instruction buffer prior to execution of the instruction by the tile. The instruction buffer of the tile to which the instruction is written may be determined by the type and subtype indicators/fields of the instruction. The instruction buffer may include a first-in first-out (FIFO) control scheme that prioritizes consumption of one or more related instructions. Therefore, under this FIFO control scheme, instructions of the same type/subtype will always be executed in the order in which they arrived on the instruction bus.

The different instruction buffers within a tile are the TensorOp instruction buffer and the DMAOp instruction buffer. As noted above, instruction types include TensorOp instructions and DMAOp instructions. For DMAOp instructions, the instruction subtypes (indicating the "write to" buffer location) include: 1) Mesh Inbound Instruction Buffer; 2) Mesh Outbound Instruction Buffer; 3) Narrow-Wide DMA Instruction Buffer; Wide-Narrow DMA Instruction Buffer; and 5) Ring Bus DMA Instruction Buffer. These buffer locations are described in more detail below with reference to FIG. The wide and narrow designations are used throughout this specification and generally refer to the approximate width size (bits/bytes) of one or more memory units. As used herein, "narrow" may refer to one or more memory units each having a size or width of less than 16 bits; may refer to one or more memory units having a size or width in between.

Mesh bus 126 provides a different data communication path than CSR bus 122, command bus 124, and ring bus 128 (discussed below). As shown in FIG. 1, mesh bus 126 provides communication paths coupling or connecting each tile to its corresponding neighboring tiles in both the X and Y dimensions. In various implementations, mesh bus 126 may be used to transport input activation quantities between one or more narrow memory units in adjacent tiles. As shown, mesh bus 126 does not allow input activation data to be transferred directly to non-adjacent tiles.

In various implementations, the mesh bus 126 and the various tiles connected via the mesh bus 126 may have the following configurations. The four corner tiles of the mesh have two outbound ports and two inbound ports. The four edge tiles of the mesh have 3 inbound ports and 3 outbound ports. All non-edge, non-corner tiles have 4 inbound ports and 4 outbound ports. Generally, in the example N×N tile layout, edge tiles are tiles with no more than 3 neighbors, and corner tiles are tiles with 2 neighbors. Regarding the dataflow methodology over the mesh bus 126, in general, all input activations arriving over the mesh bus 126 for a particular tile must be committed to one or more narrow memory units for that tile. . Additionally, for tile configurations with less than four inbound ports, a DMAOp instruction may write a zero value to a location in the tile's narrow memory instead of waiting for data on an input port that does not exist. Similarly, for tile configurations with less than four outbound ports, DMAOp instructions do not perform narrow memory reads and port writes associated with transfers to nonexistent ports.

In some implementations, the location or address of the narrow memory unit(s) to which a particular input activation is to be written or read is based on inbound/outbound DMAOps provided over mesh bus 126. is generated by a tensor traversal unit (hereinafter “TTU”). Inbound DMAOps and outbound DMAOps may run concurrently, and the necessary synchronization will be governed by a synchronization flag control scheme managed by controller 102 . TTUs are described in further detail below with reference to FIGS.

Ring bus 128 provides communication originating from controller 102 and coupling tiles 112 , 114 in a ring and back to controller 102 via a bus data path, similar to CSR bus 122 and instruction bus 124 . In various implementations, ring bus 128 generally connects or couples all wide memory units in all tiles 112, 114 (described in more detail below with reference to FIG. 2). Thus, the payload width of ring bus 128 corresponds to the width of the wide memory units located within each tile of tilesets 112,114. As noted above, ring bus 128 also includes a bitmap header that indicates which tiles need to consume payload data, including instructions or parameters communicated over ring bus 128 .

With respect to data (i.e. payload) received at a particular tile via ring bus 128, in response to receiving information, each tile zeroes (i.e. clear) and then transfer that data to another tile. Therefore, the transfer of payload to another tile will stop when the header bitmap has no remaining bitset data indicating the particular tile that is to receive the payload. Payload data generally refers to activations and weights used by one or more tiles during tensor computations performed based on execution of deeply nested loops.

In some implementations, controller 102 may be described as part of ring bus 128 . In one example, for a DMAOp instruction executing within a particular tile, controller 102 is used to pop data/payload from a ring bus stop and transfer the payload to a ring bus stop within the next tile in the ring. good too. Controller 102 can also cause payload data to be committed to one or more wide memory units of a tile if required by instructions in the bitmap header. The address of one or more wide memory units to which data needs to be written may be generated by a DMAOp instruction within a particular tile.

In various implementations, each tile in the tilesets 112, 114 can be either a producer of payload data or a consumer of payload data. If a tile is a producer of payload data, the tile reads data from one or more of its wide memory units and sends the data to ring bus 128 for consumption by one or more other tiles. multicast via If a tile is a consumer of payload data, the tile receives the data, writes it to one or more wide memory units within that tile, and forwards the payload data for consumption by one or more other tiles. . With respect to moving payload data over ring bus 128, there is typically only one producer/master of data on ring bus 128 at any given time. The DMAOp instruction execution order (eg, FIFO control scheme) in all tiles will ensure that there is only one producer/master of data on ring bus 128 at any given time.

In some implementations, controller 102 uses a synchronization flag control architecture to ensure that there is only one producer/master of payload data on ring bus 128 at any given time. In one example, each write by a tile to the ring output would trigger a corresponding increment of the sync flag count. Controller 102 can examine the payload data to determine the number of data chunks or segments that contain the payload. Controller 102 then monitors execution by the tile to ensure that the expected number of data segments are transferred and/or consumed by that tile before other tiles execute in master mode.

If there is a local multicast group connected via the ring bus 128 with no overlapping regions on the ring bus 128, there is only one producer/master of data on the ring bus 128 at any given time. There are exceptions to guaranteeing For example, tile 0 (master) multicasts (i.e. generates data) to a tile in the tiles 0-3 grouping, tile 4 (master) multicasts to a tile in the tiles 4-7 grouping. You can do the same for An important requirement of this dual-master multicast method is to prevent different multicast groups from seeing each other's data packets, as packet duplication may occur and cause one or more data computation errors. be.

As shown in FIG. 1, controller 102 provides communication data paths coupling or connecting tiles in tilesets 112, 114 to I/O 110 and includes several core functions. The core functionality of controller 102 is generally to supply one or more I/O input activations to tiles in tilesets 112, 114, one or more input activations and parameters received from I/O 110, to the tiles, one or more instructions received from I/O 110 to the tiles, sending I/O output activations to host interface 108, and to CSR bus 122 and ring bus 128. functioning as a ring stop. As described in more detail below, the first tileset 112 and the second tileset 114 each perform one or more tensor computations performed based on a deep loop nest consisting of inner and outer loops. Contains multiple tiles that are used to

System 100 generally operates as follows. Host interface 108 provides controller 102 with one or more instructions that define a direct memory access operation (DMAOp) to occur for a given computation. The descriptors associated with the instructions supplied to the controller 102 will contain the information needed by the controller to facilitate large-scale dot product computations involving multi-dimensional data arrays (tensors). In general, controller 102 receives input activations, tiling instructions, and model parameters (ie, weights) from host interface 108 to perform tensor computations for a given layer of the neural network. The controller 102 can then multicast the instruction(s) to the tiles 112, 114 in a dataflow manner defined by the instruction. As described above, the tile consuming the instruction can then begin broadcasting new/subsequent instructions to other tiles based on the bitmap data in the instruction header.

In terms of data flow, input activations and parameters are sent to the tiles of tilesets 112 , 114 via ring bus 128 . Each of the tiles 112, 114 will store the subset of input activations needed to compute the subset of output activations assigned to that particular tile. A DMAOp instruction for a tile moves input activations from wide memory to narrow memory. Computation within a tile starts when the necessary input activations, parameters/weights, and computation instructions (TTU operations, memory addresses, etc.) are available in the tile. Computations that occur within the tile are such that the MAC operator within the tile (described below) completes all inner product operations defined by the instruction set, and the pre-activation function is applied to the result of the multiplication operation (i.e. output activation). and exit.

The result of one or more tensor computations includes writing the output activations of the computational layer to the narrow memory unit(s) of the tile that performs the computation. In some tensor computations, there will be forwarding of output edge activations to neighboring tiles via the mesh bus 126 . Transferring output edge activations to neighboring tiles is required to compute output activations for subsequent layers when the computation spans multiple layers. When computations for all layers are complete, DMAOp moves the final activation to classifier tile 116 via ring bus 128 . Controller 102 then reads the final activation from classifier tile 116 and performs a DMAOp to move the final activation to host interface 108 . In some implementations, the classifier portion 116 performs computations for the output layer (ie, last layer) of the NN. In other implementations, the output layer of the NN is a classifier layer, a regression layer, or one of another layer type commonly associated with neural networks.

FIG. 2 shows an exemplary neural network (NN) computational tile 200. As shown in FIG. In general, exemplary tile 200 may correspond to any tile in first tileset 112 and second tileset 114 described above with reference to FIG. In various implementations, computation tile 200 may also be referred to or called computation unit 200 . Each computational tile 200 is a self-contained computational unit configured to independently execute instructions on other corresponding tiles in the tilesets 112,114. As briefly described above, each compute tile 200 executes two types of instructions, TensorOp instructions and DMAOp instructions. Generally, each instruction type includes computational operations associated with deep loop nesting, and thus each instruction type will generally be executed over multiple time epochs to ensure completion of all loop iterations.

As discussed in more detail below, different instruction types are executed by independent control units within the compute tile 200 that synchronize on data via synchronization flag controls managed within the compute tile 200 . Synchronization flag control manages concurrency between executions of different instruction types within compute tile 200 . Each computational operation associated with each instruction type is executed in strict issue order (ie, first-in-first-out). With respect to the two instruction types, TensorOP and DMAOp, there are no ordering guarantees between these different instruction types and each type is treated by compute tile 200 as a separate thread of control.

With respect to dataflow configuration, compute tile 200 generally includes datapath 202 and datapath 205 that each provide a communication path for dataflow into and out of compute tile 200 . As mentioned above, system 100 includes three different data bus structures laid out in a ring configuration: CSR bus 122, command bus 124, and ring bus 128. FIG. Referring to FIG. 2, data path 205 corresponds to command bus 124 and data path 202 generally corresponds to one of CSR bus 122 and ring bus 128 . As shown, data path 202 includes ring output 203 that provides an output path for data exiting compute tile 200 and ring input 204 that provides an input path for data entering compute tile 200 .

Compute tile 200 further includes TensorOp controls 206 including TensorOp tensor traversal units (TTUs) 226 and DMAOp controls 208 including DMAOp TTUs 228 . TensorOp control 206 generally manages writing to and reading from TensorOpTTU registers 232 and manages traversal operations for execution by TensorOpTTU 226 . Similarly, DMAOp control 208 generally manages writing to and reading from DMAOpTTU registers 234 and traverse operations for execution by DMAOpTTU 228 . TTU register 232 contains an instruction buffer for storing one or more instructions containing operations to be performed by TensorOp TTU 226 in executing instructions by TensorOp control 206 . Similarly, TTU register 234 includes an instruction buffer for storing one or more instructions containing operations to be performed by TTU 208 upon execution of instructions by DMAOp control 208 . As further described below, TTUs are used by computational tile 200 to traverse one or more tensor array elements that typically reside in narrow memory 210 and wide memory 212 .

In some implementations, certain instructions for execution by compute tile 200 arrive at the tile via data path 205 (ie, part of instruction bus 124). Compute tile 200 examines the header bitmap to determine the instruction type (TensorOp or DMAOp) and instruction subtype (read operation or write operation). Instructions received by the compute tile 200 are subsequently written to specific instruction buffers depending on the instruction type. Generally, instructions are received and stored (ie, written to a buffer) prior to execution of the instructions by components of compute tile 200 . As shown in FIG. 2, the instruction buffers (i.e., TensorOpTTU register 232 and DMAOpTTU register 234) may each include a first-in-first-out (FIFO) control scheme that prioritizes consumption (execution) of one or more associated instructions. .

As briefly mentioned above, tensors are multidimensional geometric objects, and exemplary multidimensional geometric objects include matrices and data arrays. Algorithms that include deeply nested loops may be executed by computational tile 200 to perform tensor computations by iterating through one or more nested loops to traverse an N-dimensional tensor. In one exemplary computational process, each loop of a loop nest may be responsible for traversing a particular dimension of an N-dimensional tensor. As described herein, the TensorOp control 206 generally implements a sequence that traverses and accesses the dimensional elements of a particular tensor construct to complete a computation defined by deeply nested loops. Manage one or more tensor operations to drive.

Compute tile 200 further includes narrow memory 210 and wide memory 212 . The narrow and wide designations generally refer to the width size (bits/bytes) of the memory units of narrow memory 210 and wide memory 212 . In some implementations, narrow memory 210 includes memory units each having a size or width of less than 16 bits, and wide memory 212 includes memory units each having a size or width of less than 32 bits. In general, compute tile 200 receives input activations via datapath 205 and DMA control 208 performs operations to write the input activations to narrow memory 210 . Similarly, compute tile 200 receives parameters (weights) via datapath 202 and DMA control 208 performs operations to write parameters to wide memory 212 . In some implementations, narrow memory 210 determines which controller (eg, TensorOp control 206 or DMAOp control 208) is allowed to access the shared memory unit of narrow memory 210 for each memory cycle. It may include a memory arbiter commonly used in shared memory systems to determine whether

Compute tile 200 further includes input activation bus 216 and MAC array 214 that includes a plurality of cells each containing MAC operator 215 and sum register 220 . In general, MAC array 214 performs tensor computations, including arithmetic operations related to dot product computations, using MAC operators 215 and sum registers 220 over multiple cells. Input activation bus 216 provides a data path through which input activations are provided by narrow memory 210 , one for each access by each MAC operator 215 of MAC array 214 . Thus, based on one-by-one broadcasting of input activations, each single MAC operator 215 in a particular cell will receive an input activation. The arithmetic operations performed by the MAC operators of MAC array 214 generally multiply input activations provided by narrow memory 210 with parameters accessed from wide memory 212 to produce a single output activation value. Including.

During arithmetic operations, partial sums are accumulated and stored in corresponding, e.g., sum registers 220 or written to wide memory 212 and re-accessed by specific cells of MAC array 214 to complete subsequent multiplication operations. may A tensor computation can be described as having a first part and a second part. The first part is completed when the multiplication operation produces the output activation, eg by completing the multiplication of the input activation with the parameter to produce the output activation. The second part involves applying a non-linear function to the output activations, and the second part is completed when the output activations are written to narrow memory 210 after applying the function.

The compute tile 200 further includes an output activation bus 218 , a non-linear unit (NLU) 222 that includes an output activation pipeline 224 , an NLU control 238 , and a reference map 230 that indicates core attributes of components within the compute tile 200 . Reference map 230 is shown for clarity, but is not included in calculation tile 200 . Core attributes include whether a particular component is a unit, storage device, operator, controller, or data path. Generally, when the first part of the tensor computation is complete, output activations are provided from MAC array 214 to NLU 222 via output activation bus 218 . After arriving at NLU 222 , data specifying an activation function received through activation pipeline 224 is applied to the output activation, which is then written to narrow memory 210 . In some implementations, the output activation bus 218 includes at least one pipelined shift register 236, and completing the second part of the tensor computation activates the shift registers 236 of the activation bus 218. using to shift the output activations towards the narrow memory 210 .

For example, for inner product calculations of two multi-dimensional data arrays for a single computational tile 200, MAC array 214 provides robust single instruction multiple data (SIMD) functionality. SIMD generally requires that all parallel units (multiple MAC operators 215) share the same instructions (based on deep loop nesting), but that each MAC operator 215 executes instructions on different data elements. means. One basic example is to add the arrays [1,2,3,4] and [5,6,7,8] element by element to get the array [6,8,10,12] in one cycle. Acquisition typically requires four arithmetic units to perform operations on each element. By using SIMD, four units can share the same instruction (eg, "add") and perform computations in parallel. Thus, system 100 and compute tile 200 provide improved acceleration and parallelism in tensor computation over conventional methods.

In one example, and as will be described in more detail below, a single instruction may cause multiple compute tiles 200 (tilesets 112, 114 in FIG. 1) to be processed by controller 102 for consumption by multiple MAC arrays 214. ) can be given. In general, a neural network layer can include multiple output neurons, which are partitioned such that tensor computations associated with subsets of the output neurons can be assigned to specific tiles of the tilesets 112, 114. can Each tile in the tilesets 112, 114 can then perform associated tensor computations on different groups of neurons for a given layer. Thus, the computational tile 200 can provide at least two forms of parallelism: 1) one form is the output activation (corresponding to a subset of output neurons) between multiple tiles of the tilesets 112, 114; 2) Another form involves simultaneous computation (using a single instruction) of multiple subsets of output neurons based on partitioning between tiles of tilesets 112,114.

FIG. 3 shows an exemplary tensor traversal unit (TTU) structure 300 containing four tensors to track, each having a depth of eight. TTU 300 generally includes counter tensor 302 , stride tensor 304 , initial tensor 306 , and limit tensor 308 . TTU 300 further includes adder bank 310 and tensor address index 312 . As mentioned above, a tensor is a multi-dimensional geometric object, and in order to access the elements of the tensor, each dimension must be indexed. Since tensors are stored in narrow memory 210 and wide memory 212, the set of tensor indices must be translated into a set of memory addresses. In some implementations, the conversion of indices to memory addresses is done by making the memory addresses a linear combination of the indices and reflecting the addresses through tensor address indices 312 .

There is a TTU for each control thread, and in compute tile 200 there is a control thread for each instruction type (TensorOP and DMAOp). Thus, as noted above, there are two sets of TTUs in compute tile 200: 1) TensorOpTTU 226; and 2) DMAOpTTU 228. In various implementations, TensorOp control 206 causes TTU 300 to load TensorOp TTU counter 302, limit 308, stride value 304 at the start of a particular tensor operation, and does not change register values before the instruction is retired. Each of the two TTUs will need to generate addresses for the following memory address ports in compute tile 200: 1) wide memory 212 address port, and 2) 4 presented as four address ports. Narrow memory 210 with four independent arbitrated banks.

As noted above, in some implementations, narrow memory 210 may be configured such that for each memory cycle, which controller (eg, TensorOp control 206 or DMAOp control 208) accesses the shared memory resources of narrow memory 210. It may include a memory arbiter commonly used in shared memory systems to determine if it is allowed to do so. In one example, different instruction types (TensorOp and DMAOp) are independent threads of control requesting memory accesses that require arbitration. When a particular control thread commits a tensor element to memory, that control thread increments a counter 302 of tensor references committed to memory.

In one example, when TensorOp control 206 executes an instruction to access a particular element of the tensor, TTU 300 can determine the address of the particular element of the tensor, and control 206 accesses storage, such as narrow memory 210. to retrieve data representing the activation value of a particular element. In some implementations, the program may include nested loops, and control 206 may set two-dimensional array variables within the nested loops according to current index variable values associated with the nested loops. An instruction can be executed to access an element of

TTU 300 may maintain traversal state for up to X number of TTU rows simultaneously for a given tensor(s). Each tensor co-resident in the TTU 300 occupies a dedicated hardware tensor control descriptor. The hardware control descriptor may consist of X number of TTU counters 302 per row position, stride 304, and limit registers 308 that support tensors with up to X number of TTU counters per row dimension. In some implementations, the number of rows and the number of counters per row may differ.

For a given location register, the final memory address is calculated from an addition operation that involves adding the location registers together. The base address is incorporated into counter 302 . One or more adders are shared for tensor references that reside in the same memory. In one implementation, there can only be a single load/store on any given port in a cycle, so multiple tensor references residing in the same narrow or wide memory can set their counters to any It is the function of loop nest control to ensure that it is not incremented in a given cycle. The use of registers to calculate memory access address values, including determination of offset values, is disclosed in patent application Ser. , 265, which is expressly incorporated herein by reference in its entirety.

The following gives the template parameters that may be used to instantiate the specialized TTU 300. 2) X number of TTU counters per row; 3) X number of TTU adder units; 4) per TTU row indicating a shared adder reference; and 5) per counter. shows the X counter size [TTU] [row] [depth]. All TTU registers are architecturally visible. The address of the particular tensor element that needs to be accessed for computation (ie tensor address 312) is the result of the addition of the counter. When an increment signal is issued from the control thread to a row of the TTU, the TTU 300 performs a single cycle operation, incrementing the innermost dimension by the stride 304 of that dimension and propagating the rollover through all depths. do.

In general, the TTU 300 determines states associated with one or more tensors. The state can include loop boundary values, current loop index variable values, dimension multipliers for calculating memory address values, and/or program counter values for processing branch loop boundaries. TTU 300 may include one or more tensor state elements and an arithmetic logic unit. Each of the tensor state elements may be a storage element, such as a register or any other suitable storage circuit. In some implementations, tensor state elements may be organized into physically or logically distinct groups.

FIG. 4 shows an exemplary architecture including narrow memory 210 that broadcasts activations 404 to one or more multiply-accumulate (MAC) operators via input bus 216 . Shift register 404 provides a shifting function in which activations 404 are sent one at a time onto input bus 216 for receipt by one or more MAC operators 215 within MAC cell 410 . In general, MAC cell 410 containing MAC operator 215 can be defined as a computational cell that computes partial sums and, in some implementations, is configured to write partial sum data to output bus 218 . As shown, cell 410 may consist of one or more MAC operators. In one implementation, the number of MAC operators 215 in MAC cell 410 is referred to as the issue width of the cell. As an example, a double issue cell computes the multiplication of two activation values (from narrow memory 210) by two parameters (from wide memory 212), the result of the two multipliers and the current partial sum. Refers to a cell that has two MAC operators that can perform addition between

As noted above, input bus 216 is a broadcast bus that provides input activations to MAC operator 215 of the linear unit (ie, MAC array 214). In some implementations, the same input is shared among all MAC operators 215 . The width of input bus 216 must be wide enough to supply the broadcast input to a corresponding number of cells for a given MAC array 214 . To illustrate the structure of input bus 216 consider the following example. When the number of cells in the linear unit equals 4 and the activation width equals 8 bits, input bus 216 can be configured to provide up to 4 input activations per cycle. In this example, all cells in MAC array 214 will access only one of the four broadcasted activations.

Based on the TensorOp field setting of the instruction received by computation tile 200, the cells of MAC array 214 may need to perform computations with the same input activations. This may be referred to as Zout partitioning within cells of MAC array 214 . Similarly, Zin partitioning within a cell occurs when cells of MAC array 214 require different activations to perform computations. In the former case, a single input activation is replicated four times, and four activations read from narrow memory 210 are broadcast over four cycles. In the latter case, a read of narrow memory 210 is required every cycle. In the example above, TensorOp control 206 orchestrates this broadcasting method based on the execution of instructions received from controller 102 .

FIG. 5 shows an exemplary architecture including an output bus 218 for providing output activation to narrow memory unit 210 of FIGS. In general, each MAC cell 215 of MAC array 214 within computational tile 200 computes a different output activation. However, with respect to the output feature array, if the output feature depth is less than the number of MAC cells 215 in computational tile 200, the cells may be grouped to form one or more cell groups. All MAC cells 215 within a cell group compute the same output (ie, for the output feature map), but each cell only computes a subset of the output corresponding to a subset of the Zin dimension. As a result, the output of MAC cell 215 is now a partial sum rather than a final linear output. In some implementations, NLU 222 aggregates these partial sums into a final linear output based on control signals provided to NLU 222 by NLU control 238 .

As mentioned above, output bus 218 is a pipelined shift register. In various implementations, when the first part of the tensor computation is finished and the TensorOp control 206 indicates (by executing an instruction) that the partial sums should be written out, the partial sums provided on the output bus 218 are There will be parallel loads. The number of parallel loads will correspond to the number of MAC cells in the compute tile 200 . TensorOp control 206 then shifts out the partial sum quantity and sends it through the non-linear pipeline. In some implementations, there may be situations where not all MAC cells within a tile are actually utilized to perform computations. In such a situation, not all partial sums shifted onto the output bus will be valid. In this example, TensorOp control 206 may provide control signals to MAC array 214 to indicate the number of valid cells to be shifted out. The amount of parallel load loaded onto output bus 218 still corresponds to the number of MAC cells in the computational tile, but only valid values will be shifted out and committed to narrow memory 210 .

FIG. 6 is an exemplary flowchart of a process 600 for performing tensor computations using a neural network (NN) computational tile, such as computational tile 200 of FIG. Process 600 begins at block 602, where sufficient parameters are stored in a second memory on-chip to allow delay below a specified level and throughput above a specified level for a given NN model and architecture. to load. Throughput is the maximum performance achieved in the presence of a large number of requests/inferences. Delay is the minimum amount of time it takes to compute one request. Process 600 continues at block 604 where narrow memory 210 of compute tile 200 sends (ie, broadcasts) activations to input activation data bus 216 one by one. The activation values are stored in narrow memory 210 . Narrow memory 210 may be a collection of static random access memory (SRAM) banks that allow addressing of specific memory locations to access input quantities. Activations read from memory 210 are broadcast via input activation bus 216 to the linear cells of MAC array 214 (ie, a linear unit) that includes a plurality of MAC operators 215 and a sum register 220 . At block 606 of process 600, MAC operators 215 of compute tile 200 each receive two inputs, one input (activation) from input activation bus 216; received from. Activation therefore supplies one of the inputs for each MAC operator 215 , and each MAC operator 215 within a cell of MAC array 214 gets its second multiplier input from wide memory 212 .

At block 608 of process 600, MAC array 214 of compute tile 200 performs tensor computations, including dot product computations, based on the elements of the data array structure accessed from memory. Wide memory 212 may have a width in bits equal to the width of a linear unit (eg, 32 bits). Thus, a linear unit (LU) is a SIMD vector arithmetic logic unit (ALU) unit that receives data from vector memory (ie, wide memory 212). In some implementations, MAC operator 215 may also obtain accumulator inputs (partial sums) from wide memory 212 . In some implementations, there is time sharing to wide memory 212 ports for reads and/or writes for two different operands (parameters and partial sums). In general, wide memory 212 may have a limited number of ports to optimize area. As a result, if an operand (eg, parameter) needs to be read from wide memory 212 and simultaneously written to wide memory 212 (eg, a partial sum), the pipeline associated with the particular operand may stall.

At block 610, a computational cell (having MAC operator 215 and sum register 220) of computational tile 200 generates at least one output activation based on the multiplication operation performed by the MAC/computational cell. The results of MAC cell operations include either partial sums written back to memory (during partial sum arithmetic operations) or output activations sent to output bus 218 . NLU 222 of compute tile 200 may apply a non-linear activation function to the output activations and write the activations to narrow memory 210 . In some implementations, output bus 218 is a shift register that accumulates parallel loads of results/output activations from MAC operator 215, applies non-linear functions and write operations to narrow memory 210 in the same tile. , they can be shifted out one at a time.

The embodiments described in this specification make use of the following two points of view. 1) The bottleneck of most existing Neural Network (NN) accelerators is the memory bandwidth required to load the NN weights (also known as parameters). 2) Even if the number of parameters in the generative model is large, i.e. several KB to several GB, whereas most models are several MB to hundreds of MB, these numbers are It is within the realm of hardware implementation including on-chip memory using memory distributed in .

Adding a mass memory, eg, cache or scratchpad, on the die to contain all the parameters is not sufficient. The goal of overcoming memory bandwidth limitations is to scale out the performance of the architecture. That means increasing the number of operators, usually MACs. However, to achieve high performance, we must be able to supply these operators along with their parameters every cycle. Also, it is important to understand "performance" not only as throughput, but also as delay. This is true for applications that face many users.

In other words, in Neural Network (NN) accelerators, loading parameters into one layer at a time is very costly. If the parameters can be preloaded onto the chip, they only need to be loaded at runtime activation. Thus, embodiments include extensive on-chip memory.

The embodiments described herein affect the operation and design of Neural Network (NN) accelerators. Embodiments address one of the significant limitations of neural network accelerator design: low delay combined with high throughput. Assume that the NN accelerator represents the entire neural network. The input is some data, eg a small image or sound. The NN accelerator executes layers one by one. It is costly in performance and energy to load the layer parameters one by one. The NN accelerator loads the parameters of one layer, performs the calculations, holds the output of the layer, and then loads the parameters of the next layer. This is where most of the memory bandwidth is consumed in this process.

By having the parameters reside on-chip, the delay is dramatically reduced given the high throughput and specific NN models and architectures. The accelerator only needs to load a few bytes of audio, after which the accelerator can be very fast. Accelerators do not require high memory bandwidth and are energy efficient.

The von Neumann model, where memory is loaded from memory to the CPU, is a common architecture. Architectures like this conventional von Neumann model, where memory resides at one end of the die and computational operators reside at the other end of the die, are impractical and, if not impossible, use a large number of operators. In the case, it means a huge number of wires for transferring data from memory (or memory bank) to the operator. Instead, embodiments of the present invention take advantage of the memory-locality property of NN computations to architecture into tile configurations that distribute memory among the tiles (as shown in FIGS. 2 and 8). configure.

Because NNs are large, but not gigantic, we can come close to effectively fitting all the parameters of one or a few NNs onto a single chip. NN accelerators are moving towards self-contained architectures. For tiled architectures, memory can be partitioned within the chip. Instead of having one large SRAM in one corner of the chip, embodiments allocate an appropriate amount of SRAM per tile to avoid on-chip bandwidth issues as well. In a particular embodiment, the wide memory in each tile contains parameters, and the wide ring (of approximately similar width as the wide memory) feeds the wide memory with high bandwidth. The embodiments described herein contemplate architectural variations. Depending on the characteristics of the NN layer, embodiments can have at least the following two NN architectures.

Neural networks, which consist mostly of fully connected layers, do not often reuse parameters between layers. A neuron contained in a fully connected layer has full connections to all activations contained in the previous layer. Consider a fully-connected neural network where parameters are not reused (without batching, eg, consider a real-time application). If all parameters are not contained in wide memory, they must be fetched from external memory via the ring bus. In that case, the performance of the overall design is limited by the external memory bandwidth. If all parameters reside in wide memory, there is no need to access external memory and all operators can be supplied with parameters every cycle, maximizing performance. Instead of utilizing external memory only to fetch parameters, the embodiments described herein keep parameters resident in wide memory.

As an example, consider a model with 50M parameters (or 50MB for clarity) in a fully connected layer. Consider an accelerator with 16384 MACs running at 1 GHz. Consider that the input to the model is 16KB. All values are reasonable for the current application. Maximum performance corresponds to running the model in 50×10̂6/(16384×10̂9)=3.05×10̂-6 seconds. This corresponds to a memory bandwidth of (50×10̂6+16,384)/(3.05×10̂-6)=16.40 TB/s. As a point of comparison, a typical DRAM chip is roughly around 10 GB/s, while state-of-the-art High Bandwidth Memory (HBM) is around 256 GB/s.

Convolutional neural networks pass parameters from one tile to another. In neural networks composed mostly of convolutional layers, where parameters are reused between neurons (also known as activations), the memory bandwidth requirements are not very high, but generally high. The ring bandwidth may be sufficient to load the parameters into the tiles, but connected to a large on-die memory as wide as the ring. In other words, every inference requires the tile to access/load all parameters of the model. This applies to any model. The only difference for fully connected layers in neural networks is that each parameter is used only once during an inference. For convolutional layers, the parameters are used multiple times within the layer.

As an example, consider a model with 50M parameters in a convolutional layer. Some of these layers may be very small and others large, resulting in variable reuse of parameters. A reasonable average between models is a maximum of 100 reuses per parameter. So, using the same theory as above, the bandwidth requirement drops to 16.40 TB/s/100-164 GB/s. However, the bandwidth requirements for cost-effective DRAMs remain high. However, with the above architecture, a wide ring of 164×8=1312 bits connected to a large memory of the same width can feed the tiles at reasonable speed.

Consider a first embodiment in which the memory is sufficient to contain all the parameters of the layers of the fully connected model. If all tiles operate simultaneously on a layer, the parameters need to be distributed among the tiles. Embodiments divide the output neurons/activations of each layer between tiles. At runtime, each tile processes a subset of layers and computes the corresponding partial sums to pass to its neighbor. That is, the partial sums go around the ring and after all around the tiles produce the final sum.

A second embodiment involves preloading/caching the same subset of (currently used) parameters for all tiles, since the tiles utilize the same parameters at the same time. At runtime, the parameters (subsets) that are not partial activation sums go around the ring.

The number of tiles is the scale factor. Robust scaling can be achieved by improving latency and throughput without increasing memory requirements by using the embodiments described herein. However, processing tiles together to increase computational power increases memory requirements and the number of activations required. Batch scaling is difficult without increasing memory bandwidth over traditional memory options. Batch processing often involves real-time applications and involves latency and throughput requirements.

It is worth noting that having parameters in cache is different than having parameters in wide memory as part of tiling. Laying out the tiles as described in this specification provides compiled-style local referencing. For example, placing the fully coupled model next to the SRAM results in higher internal bandwidth compared to the cache model.

The embodiments described herein operate faster than conventional neural network accelerators. Certain embodiments have more operators and the accelerator needs more internal bandwidth to supply the operators. To deal with this, the architecture needs to distribute the memory and collect the parameters on the accelerator.

The largest chip today is about 650 square millimeters. Therefore, there is a limit to the amount of SRAM that can be mounted on a chip. Embodiments include utilizing high density memory, including utilizing 3D stacking, given spatial constraints.

The embodiments described herein apply to inference (post-training) mode and training mode.

An additional consideration is another hierarchy level. There is a memory hierarchy and usually there is also a register file hierarchy. Parameters are loaded into registers and registers can be reused to expand memory bandwidth. There is memory bandwidth in register files and memory. This means further cost savings, ie less wiring from memory to register files and from register files to computation. The embodiments described herein can reduce wiring costs associated with register files. Addressing is done by consuming parameters directly from SRAM. That is, the memory directly feeds the ALU.

FIG. 7 shows a block diagram of another embodiment of a computation system 700 for accelerating tensor computations associated with deep neural networks (DNN). System 700 generally includes a controller/UNCORE 702 , a memory interface 708 , and multiple tiles including first tileset 112 and second tileset 714 . Controller 702 generally includes data memory 704, instruction memory 706, and at least one processor configured to execute one or more instructions encoded on a computer-readable storage medium. Instruction memory 706 may store one or more machine-readable instructions executable by one or more processors of controller 702 . Data memory 704 may be any of a variety of data storage media for storing and subsequently accessing various data related to computations occurring within system 700 .

Controller 702 is configured to execute one or more instructions related to tensor computation within system 700 , including instructions stored in instruction memory 706 . In some implementations, data memory 704 and instruction memory 706 are volatile memory unit(s). In some other implementations, data memory 704 and instruction memory 706 are non-volatile memory unit(s). Data memory 704 and instruction memory 706 may also include floppy disk drives, hard disk drives, optical disk drives, or tape drives, flash memory or other similar solid state memory devices, or devices in storage area networks or other configurations. It may also be another form of computer readable medium, such as an array of devices comprising. In various implementations, controller 702 may be referred to or called as core manager 702 .

Memory interface 708 receives instructions and data parameters from the I/O link and provides instructions and parameters to controller 702 . Generally, instructions can be provided to one or more devices in system 100 via a command bus (the command bus between the controller and the tiles is not shown), and parameters can be sent to system 700 via ring bus 728. can be provided to one or more devices within In some implementations, instructions are initially received by controller 702 from host interface 708 and stored in instruction memory 706 for subsequent execution by controller 702 .

Ring bus 728 provides communication originating from controller 702 and coupling tiles 712 , 714 in a ring through a bus data path connecting back to controller 702 . In various implementations, a ring bus 728 generally connects or couples all wide memory units in all tiles 712,714. Thus, the payload width of ring bus 728 corresponds to the width of the wide memory units located within each tile of tilesets 712,714. As noted above, ring bus 728 also includes a bitmap header that indicates which tiles need to consume payload data, including instructions or parameters communicated over ring bus 728 .

With respect to data (i.e. payload) received at a particular tile via ring bus 728, in response to receiving information, each tile zeroes (i.e. clear) and then transfer that data to another tile. Therefore, the transfer of payload to another tile will stop when the header bitmap has no remaining bitset data indicating the particular tile that is to receive the payload. Payload data generally refers to activations and weights used by one or more tiles during tensor computations performed based on execution of deeply nested loops.

In some implementations, controller 702 may be described as part of ring bus 728 . In one example, for a DMAOp instruction executing within a particular tile, controller 702 is used to pop data/payload from a ring bus stop and transfer the payload to a ring bus stop within the next tile in the ring. good too. The controller 702 can also cause payload data to be committed to one or more wide memory units of the tile if required by instructions in the bitmap header. The address of one or more wide memory units to which data needs to be written may be generated by a DMAOp instruction within a particular tile.

In various implementations, each tile in the tilesets 712, 714 can be either a producer of payload data or a consumer of payload data. If a tile is a producer of payload data, the tile reads data from one or more of its wide memory units and sends the data to ring bus 728 for consumption by one or more other tiles. multicast via If a tile is a consumer of payload data, the tile receives the data, writes it to one or more wide memory units within that tile, and forwards the payload data for consumption by one or more other tiles. . With respect to moving payload data over ring bus 728, there is typically only one producer/master of data on ring bus 728 at any given time. The DMAOp instruction execution order (eg, FIFO control scheme) in all tiles will ensure that there is only one producer/master of data on ring bus 728 at any given time.

In some implementations, controller 702 uses a synchronization flag control architecture to ensure that there is only one producer/master of payload data on ring bus 728 at any given time. In one example, each write by a tile to the ring output would trigger a corresponding increment of the sync flag count. Controller 702 can examine the payload data to determine the number of data chunks or segments that contain the payload. Controller 702 then monitors execution by the tile to ensure that the expected number of data segments are transferred and/or consumed by that tile before other tiles execute in master mode.

If there is a local multicast group connected via ring bus 728 that has no overlapping regions on ring bus 728, there is only one producer/master of data on ring bus 728 at any given time. There are exceptions to guaranteeing An important requirement of this dual-master multicast method is to prevent different multicast groups from seeing each other's data packets, as packet duplication may occur and cause one or more data computation errors. be.

In contrast to FIG. 1, the tiles in FIG. 7, while themselves forming a rectangular pattern, are connected in a zigzag or serpentine fashion by ring buses. In the illustrated embodiment, the accelerators consist of 8 and 16 tiles, respectively. In another embodiment, the accelerator may contain more tiles.

FIG. 8 is a simplified diagram of the calculation tile of FIG.
FIG. 9 is an exemplary flowchart of a process 900 for performing tensor computations using a neural network (NN) computational tile, such as computational tile 200 of FIG. Process 900 begins at block 902 by loading NN weight parameters into the NN accelerator, eg, prior to execution. Process 900 continues at block 904 to process the input to the accelerator without substantially accessing neural network weight parameters outside the accelerator. At block 906, the process generates at least one output activation based on processing the input to the accelerator.

The subject matter and functional operational embodiments described herein are computer software or firmware tangibly embodied in digital electronic circuitry, including the structures disclosed herein and their structural equivalents. , in computer hardware, or in one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., tangible, non-transitory programs for execution by a data processing apparatus or for controlling operation of a data processing apparatus. can be implemented as one or more modules of computer program instructions encoded on a program carrier. Alternatively or additionally, the program instructions are generated to encode information for transmission to a receiving device suitable for execution by a data processing device, e.g., a machine-generated electrical signal, an optical It can be encoded on a signal, or an artificially generated propagated signal, such as an electromagnetic signal. A computer storage medium may be a computer readable storage device, a computer readable storage substrate, a random or serial access memory device, or a combination of one or more thereof.

The processes and logic flows described herein are performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. can be performed. The processes and logic flows may also be performed by special purpose logic circuits such as FPGAs (Field Programmable Gate Arrays), ASICs (Application Specific Integrated Circuits), or GPGPUs (General Purpose Graphics Processing Units), and the devices may also be implemented by them. .

A processor suitable for the execution of a computer program may, by way of example, be based on general and/or special purpose microprocessors or any kind of central processing unit. Generally, a central processing unit receives instructions and data from read-only memory and/or random-access memory. The essential elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical or optical disks, for storing data or receives data from such one or more mass storage devices. and/or operably coupled to transfer data to the one or more mass storage devices. However, a computer need not have such devices.

Computer readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media and memory devices including semiconductor memory devices such as EPROM, EEPROM and flash memory devices; Includes magnetic disks such as hard disks or removable disks. The processor and memory may be supplemented by, or incorporated into, special purpose logic circuitry.

While this specification contains details of many specific implementations, these should not be construed as limitations on the scope of any invention or what may be claimed, rather than specific embodiments of the specific invention. It should be construed as a description of what may be a unique feature. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Further, while features may be described above and even initially claimed to act in certain combinations, one or more features from the claimed combination may in some cases be such It may be omitted from the combination and the claimed combination may be directed to subcombinations or variations of subcombinations.

Similarly, although acts have been presented in the figures in a particular order, it should be understood that such acts must be performed in the particular order presented or in a sequential order to achieve desirable results. It should not be understood, or that all illustrated acts must be performed. Multitasking and parallel processing can be advantageous in some situations. Further, the separation of various system modules and components in the above-described embodiments should not be understood to require such separation in all embodiments, and the program components and systems described generally are a single software module. It should be understood that it can be integrated into a product or packaged into multiple software products.

Particular embodiments of the subject matter have been described. Other embodiments are within the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes illustrated in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve desirable results. Multitasking and parallel processing may be advantageous in some implementations.

Claims (25)

An accelerator for accelerating tensor computation, comprising:
The accelerator includes one or more computational units, each computational unit comprising:
a. a first memory bank for storing at least one of input activations or output activations;
b. a second memory bank for storing neural network parameters for use in performing computations, said second memory bank storing a sufficient amount of said neural network parameters on said computing unit to allow a given configured to allow delay below a specified level and throughput above a specified level for a neural network ("NN") model and architecture of
c. Each of said computation units further comprises a plurality of cells each including at least one multiply-accumulate ("MAC") operator for receiving parameters from said second memory bank and performing computations, each of said plurality of cells each comprising : , configured as an array that processes different data with a single instruction,
d. Each computing unit further comprises a first traversal unit in data communication with at least the first memory bank, the first traversal unit providing control signals to the first memory bank to cause associated cells to configured to provide an input activation to a data bus accessible by an associated MAC operator included;
e. The accelerator performs one or more computations associated with at least one element of a data array, the one or more computations performed by the MAC operator and received in part from the data bus. and a multiplication operation of the input activations received from the second memory bank and parameters received from the second memory bank. the one or more computing units are divided into a first set and a second set;
2. The accelerator of claim 1, wherein each of said first set and said second set perform concurrent computation of subsets of output neurons using said single instruction. 3. The accelerator of claim 1 or 2, wherein said first memory bank, said second memory bank, said first traversal unit and said at least one MAC operator are included on the same die. Accelerator according to any of claims 1 to 3, wherein said second memory bank is arranged to store more than 100,000 parameters. Accelerator according to any of claims 1 to 3, wherein said second memory bank is configured to store more than 1,000,000 parameters. Accelerator according to any of claims 1 to 3, wherein said second memory bank is arranged to store more than 100,000,000 parameters. 7. The accelerator according to claim 1, wherein said second memory bank includes SRAM. The accelerator according to any one of claims 1 to 7, wherein said second memory bank comprises 3D SRAM. A computer-implemented method comprising a computation unit for accelerating tensor computation, comprising:
a. sending a first input activation by said first memory bank in response to said first memory bank receiving a control signal from a first traversal unit, said first memory bank being adapted to said computation unit; and wherein said first input activation is provided by a data bus accessible by a plurality of cells of said computation unit, said method further comprising:
b. receiving, by the plurality of cells, one or more parameters from a second memory bank for storing neural network parameters used in performing computations, the second memory bank having a sufficient amount of storing said neural network parameters on said computing unit to allow for a given neural network (“NN”) model and architecture a delay below a certain threshold with a throughput above a certain threshold; The plurality of cells includes at least one sum-of-products (“MAC”) operator, the method further comprising:
c. performing one or more calculations associated with at least one element of a data array by associated MAC operators contained in associated cells, said one or more calculations being partially transferred from said data bus; a multiplication operation of at least the first input activation accessed and at least one parameter received from the second memory bank;
d. The method of claim 1, wherein each of said plurality of cells is configured as an array to process different data with a single instruction. the computational unit comprises a plurality of computational units divided into a first set and a second set;
10. The method of claim 9, wherein each of said first set and said second set perform concurrent computation of subsets of output neurons using said single instruction. 11. A method according to claim 9 or 10, wherein said first memory bank, said second memory bank, said first traversal unit and said at least one MAC operator are housed on the same die. A method according to any of claims 9-11, wherein said second memory bank is arranged to store more than 100,000 parameters. A method according to any of claims 9-11, wherein said second memory bank is arranged to store more than 1,000,000 parameters. A method as claimed in any one of claims 9 to 13, wherein said second memory bank comprises SRAM. A method according to any of claims 9-13, wherein said second memory bank comprises 3D SRAM. A method according to any one of claims 9 to 15, further comprising loading said neural network parameters for use in performing calculations into said second memory bank. A computer-implemented method comprising a computation unit for accelerating tensor computation, comprising:
a. sending a first input activation by said first memory bank in response to a first memory bank receiving a control signal, said first input activation being sent by a data bus; The method further
b. receiving one or more parameters from a second memory bank for storing neural network parameters used in performing calculations by a plurality of multiply-accumulate (“MAC”) operators contained in a plurality of cells; and said second memory bank stores a sufficient amount of said neural network parameters on said computation unit such that for a given neural network (“NN”) model and architecture, delays below a particular threshold and enabling throughput above a certain threshold, the method further comprising:
c. performing, by each of the plurality of MAC operators, one or more calculations associated with at least one element of a data array, the one or more calculations being partially from the data bus; a multiplication operation of at least the first input activation accessed and at least one parameter received from the second memory bank;
d. The method of claim 1, wherein the plurality of cells are included in the computing unit, and wherein each of the plurality of MAC operators is configured to process different data with a single instruction. the computational unit comprises a plurality of computational units divided into a first set and a second set;
18. The method of claim 17, wherein each of said first set and said second set perform concurrent computation of subsets of output neurons using said single instruction. 19. A method according to claim 17 or 18, wherein said first memory bank, said second memory bank, a first traversal unit and said plurality of MAC operators are housed on the same die. A method according to any of claims 17-19, wherein said second memory bank is arranged to store more than 100,000 parameters. A method according to any of claims 17-19, wherein said second memory bank is configured to store more than 1,000,000 parameters. A method according to any of claims 17-19, wherein said second memory bank is configured to store more than 100,000,000 parameters. The method of any of claims 17-21, wherein said second memory bank comprises SRAM. A method according to any of claims 17-21, wherein said second memory bank comprises 3D SRAM. A method according to any of claims 17-24, further comprising loading said neural network parameters for use in performing calculations into said second memory bank.

Images