JP6995851B2 - Neural network calculation tile - Google Patents

Description

Background The present specification generally relates to neural net calculation tiles for the calculation of deep neural network (“DNN”) layers that allow reduction of instruction bandwidth and instruction memory.

Overview In general, one innovative aspect of the subject matter described herein can be carried out in computational units for accelerating tensor computation. The compute unit has a first data width, a first memory bank for storing at least one of input activation or output activation, and a second data larger than the first data width. It has a width and includes a second memory bank for storing one or more parameters used when performing calculations. The compute unit may further include at least one cell containing at least one product-sum (“MAC”) operator that receives parameters from the second memory bank and performs the computation. The computing unit further comprises a first traversal unit that communicates data with at least the first memory bank, the first traversal unit providing control signals to the first memory bank and data accessible by the MAC operator. It is configured to give input activation to the bus. The compute unit performs one or more computations related to at least one element of the data array, and the one or more computations are performed by the MAC operator and, in part, the input activity received from the data bus. Includes a multiplication operation between the conversion and the parameters received from the second memory bank.

In general, another innovative aspect of the subject matter described herein can be carried out in a computer-implemented way to accelerate tensor calculations. The method performed by the computer is to activate the first input by the first memory bank in response to the first memory bank having the first data width receiving the control signal from the first traversal unit. The first memory bank is located within the computing unit and the first input activation is provided by a data bus accessible by at least one cell of the computing unit. The method further comprises receiving one or more parameters from a second memory bank having a second data width greater than the first data width by at least one cell, where at least one cell is at least one. Contains two multiply-accumulate ("MAC") operators. The method further comprises performing one or more computations related to at least one element of the data array by means of a MAC operator, the one or more computations being partially accessed from the data bus. Includes a multiplication operation of at least the first input activation and at least one parameter received from the second memory bank.

Another innovative aspect of the subject matter described herein can be implemented in non-temporary computer-readable storage media. A non-temporary computer-readable storage medium containing instructions that can be executed by one or more processors, the instructions causing one or more processors to perform an operation when such an operation is performed. The first memory bank having the first data width comprises sending a first input activation by the first memory bank in response to receiving a control signal from the first traversal unit. One memory bank is located within the compute unit and the first input activation is provided by a data bus accessible by at least one cell of the compute unit. The operation performed may further include receiving one or more parameters from a second memory bank having a second data width larger than the first data width by at least one cell. A cell contains at least one multiply-accumulate ("MAC") operator. The operation performed may further include performing one or more calculations related to at least one element of the data array by means of a MAC operator, where the one or more calculations are in part data. Includes a multiplication operation of at least one input activation accessed from the bus and at least one parameter received from the second memory bank.

The subject matter described herein can be realized in a particular embodiment so as to realize one or more of the following advantages: Tracking memory address values using registers allows a program to iterate through a deeply nested loop with a single instruction. Tensors accessible from narrow and wide memory units in a single calculation tile are traversed based on the memory address values retrieved from the registers. The memory address value corresponds to the element of the tensor. Tensor calculations occur on individual calculation tiles based on the execution of deep loop nesting. The calculation can be distributed across multiple tiles. Computational efficiency is improved and accelerated based on distributing tensor computations of multi-layer neural networks across several computational tiles. You can traverse tensors and perform tensor calculations with a small number of instructions.

The subject matter described herein can also be realized in certain embodiments to realize other advantages. For example, by adopting a memory hierarchy that combines narrow, low-bandwidth memory with high-bandwidth, wide memory, which allows addressing flexibility to traverse multidimensional arrays in any order. High utilization of MAC operators can be achieved for DNN layers of very different dimensions, and locality in computation can be fully utilized.

Other implementations of this aspect and other aspects include corresponding systems, devices and computer programs encoded on computer storage and configured to perform method actions. A system of one or more computers can be so configured by software, firmware, hardware or a combination thereof that is installed in the system and causes the system to perform actions in action. One or more computer programs can be configured as such by having instructions that cause the device to perform an action when executed by the data processing device.

The subject matter described herein also relates to image recognition and / or classification methods / systems. The system can be implemented using the disclosed technology and the described hardware calculation system with hardware calculation units or calculation tiles. The calculation unit processes a tensor calculation for computing inference using a neural network having a plurality of neural network layers.

Details of one or more implementations of the subject matter described herein are described in the accompanying drawings and in the description below. Other potential features, aspects and advantages of the subject will become apparent from the description, drawings and claims.

It is a block diagram of an exemplary calculation system. An exemplary neural network calculation tile is shown. An exemplary tensor traversal unit (TTU) structure is shown. An exemplary architecture is shown that includes a narrow memory unit that provides input activation to one or more multiply-accumulate (MAC) operators. An exemplary architecture is shown that includes an output bus that provides output activation to the narrow memory units of FIGS. 2 and 4. It is an exemplary flowchart of a process for performing a tensor calculation using the neural network calculation tile of FIG.

Similar reference numbers and designations in various drawings indicate similar elements.
Detailed Description The subject matter described herein relates to a hardware computational system containing multiple computational units configured to accelerate the machine learning inference workload of the neural network layer. Each computational unit in a hardware computational system is self-contained and can independently perform the computations required by a given layer of a multi-layer neural network.

Inference can be calculated using a neural network with multiple layers. For example, given an input, the neural network can compute inferences for that input. The neural network computes this inference by processing the input through each layer of the neural network. In particular, each layer of the neural network has its own set of weights. Each layer receives an input and processes the input according to a set of weights for that layer to produce an output.

Therefore, to compute an inference from the received input, the neural network receives the input, processes it through each neural network layer to generate the inference, and the output from one neural network layer is the next neural network layer. Given as an input to. Data inputs to a neural network layer, such as inputs to a neural network, or outputs of layers below that layer in a sequence to a neural network layer, can be referred to as activation inputs to that layer.

In some implementations, the layers of the neural network are arranged in a sequence. In another implementation, the layers are arranged in a directed graph. That is, any particular layer can receive multiple inputs, multiple outputs, or both. The layers of a neural network can also be configured so that the output of one layer can be sent back as input to the previous layer.

The hardware computation system described herein can perform computations on the neural network layer by distributing the tensor computations across multiple computation tiles. Computational processes performed within the neural network layer may include multiplication of an input tensor containing input activations with a parameter tensor containing weights. The calculation involves multiplying the input activation by the weight in one or more cycles, and performing product accumulation over many cycles.

Tensors are multidimensional geometric objects, and exemplary multidimensional geometric objects include matrices and data arrays. In general, software algorithms are performed by computational tiles to perform tensor calculations by processing nested loops to traverse N-dimensional tensors. In one exemplary computational process, each loop may be responsible for traversing a particular dimension of an N-dimensional tensor. For a given tensor construct, the calculation tile may require access to the elements of that tensor in order to perform multiple dot product calculations related to that tensor. Calculations are made when the input activation given by the narrow memory structure is multiplied by the parameters or weights given by the wide memory structure. Since the tensor is stored in memory, a set of tensor indexes may need to be translated into a set of memory addresses. In general, the tensor traversal unit of a calculation tile performs a control operation that gives the index of each dimension associated with the tensor and the order in which the index elements are traversed and the calculation is performed. The tensor calculation ends when the multiplication result is written to the output bus and stored in memory.

FIG. 1 shows a block diagram of an exemplary computational system 100 for accelerating tensor computations associated with deep neural networks (DNNs). The system 100 generally includes a controller 102, a host interface 108, an input / output (I / O) link 110, a plurality of tiles including a first tile set 112 and a second tile set 114, a classifier portion 116, and a bus map. Includes a data bus identified in 118 (shown for clarity but not included in system 100). The controller 102 generally includes a data memory 104, an instruction memory 106, and at least one processor configured to execute one or more instructions encoded in a computer-readable storage medium. The instruction memory 106 can store one or more machine-readable instructions that can be executed by one or more processors of the controller 102. The data memory 104 may be any of various data storage media for storing various data related to the calculation generated in the system 100 and then accessing the data.

The controller 102 is configured to execute one or more instructions related to the tensor calculation in the system 100, including the instructions stored in the instruction memory 106. In some implementations, the data memory 104 and the instruction memory 106 are volatile memory units (s). In some other implementations, the data memory 104 and the instruction memory 106 are non-volatile memory units (s). The data memory 104 and the instruction memory 106 may also include floppy (registered trademark) disk devices, hard disk devices, optical disk devices, or tape devices, flash memory or other similar solid-state memory devices, or storage area networks or devices of other configurations. It may be another form of computer-readable medium, such as an array of devices including. In various embodiments, the controller 102 may be referred to or so referred to as the core manager 102.

As shown, the host interface 108 is coupled to the I / O link 110, the controller 102, and the classifier portion 116. The host interface 108 receives instructions and data parameters from the I / O link 110 and gives the instructions and parameters to the controller 102. In general, instructions can be given to one or more devices in system 100 via instruction bus 124 (discussed below) and parameters can be given to one or more devices in system 100 via ring bus 128 (discussed below). Can be given to. In some implementations, the instruction is initially received by the controller 102 from the host interface 118 and stored in the instruction memory 106 for later execution by the controller 102.

The classifier portion 116 is similarly coupled to the controller 102 and tile 7 of the second tile set 114. In some implementations, the classifier portion 116 is implemented as a separate tile within the system 100. In an alternative embodiment, the classifier portion 116 is located or located within the controller 102 as a subcircuit or subdevice of the controller 102. The classifier portion 116 is generally configured to perform one or more functions on the cumulative pre-activation values received as the output of the fully connected layer. The fully coupled layer may be divided across the tiles within the tile sets 112 and 114. Therefore, each tile is configured to generate a subset of pre-activation values (ie, linear output) that can be stored in the tile's memory unit. The classification result bus 120 provides a data path from the classifier portion 116 to the controller 102. The data including the post-function value (ie, result) is given from the classifier portion 116 to the controller 102 via the classification result bus 120.

Bus map 118 shows a data bus that provides one or more interconnected data communication paths between the tiles of the first tile set 112 and the second tile set 114. As shown in FIG. 1, the bus map 118 provides an explanation of codes used for identifying the classification result bus 120, the CSR / master bus 122, the command bus 124, the mesh bus 126, and the ring bus 128. In general, tiles are a core component within the accelerator architecture of system 100 and are the focus of tensor calculations that occur within the system. Each tile is an individual computational unit, and multiple tiles can interact with other tiles in the system to accelerate computations across one or more layers of a multi-layer neural network (eg, tensor computations). For example, the calculation can be distributed across multiple tiles. Computational efficiency can be improved and accelerated based on distributing tensor computations of multi-layer neural networks across several computational tiles. The tiles in tile sets 112, 114 can share the execution of tensor calculations associated with a given instruction, but the individual compute units are independent of the other corresponding tiles in tile sets 112, 114. It is a self-contained computational component configured to perform a subset of tensor computations.

The control and status register (CSR) bus 122 sets the program configuration and allows the controller 102 to send one or more instructions to read the status register associated with one or more tiles. It is a slave bus. The CSR bus 122 can be connected in a single daisy chain configuration with one master bus segment and a plurality of slave bus segments. As shown in FIG. 1, the CSR bus 122 provides communication that connects the tiles of the tile sets 112, 114 and the controller 102 via a bus data path connecting the tiles and the controller 102 in a ring to the host interface 110. In some implementations, the host interface 110 is the single master of the CSR bus ring, and the entire CSR bus address space is memory mapped to the memory space within the host interface 110.

The CSR bus 122 may, for example, program a memory buffer pointer in the controller 102 to allow the controller 102 to initiate fetching instructions from the instruction memory 106, statically during one or more operations. One that involves updating / programming various tile settings that remain in place (eg, a coefficient table for polynomial approximation calculations) and / or loading / reloading the firmware for classifier part 116. It can be used by the host interface 110 to perform the above operations. In one example, the firmware reload may include a new function to be applied to the linear output (ie, the pre-activation value). Therefore, every slave having access to the CSR bus 122 will have a separate node identifier (node ID) associated with and identifying it. The node ID is part of the instruction address and is used, inspected, or used by the CSR slave (ie, controller 102, tiles 112, 114 and classifier 116) to determine if the CSR packet is addressed to the slave. It can be examined by other methods.

In some implementations, one or more instructions can be sent through the controller 102 by the host interface 102. The instruction may be, for example, 32 bits wide, with the first 7 bits containing header information indicating the instruction address / destination for which the instruction is to be received and executed. The first 7 bits of the header may contain data parameters representing a particular node ID. Therefore, a slave on the CSR busling (eg, each tile) can inspect the instruction header to determine if the request by the master (host interface 110) is addressed to the tile inspecting the header. .. If the node ID in the header does not indicate that the destination is the inspection tile, the inspection tile copies the input CSR instruction packet to the CSR bus input connected to the next tile for inspection by the next tile.

The instruction bus 124 starts from the controller 102 and, like the CSR bus 122, provides communication in which the tiles in the tile sets 112 and 114 are connected via a bus data path that connects the tiles in the tile set 112 and 114 back to the controller 102 in a ring shape. In one embodiment, the controller 102 broadcasts one or more instructions over the instruction bus 124. The instructions broadcast by the controller 102 may differ from the instructions given via the CSR bus 122. However, the mode in which the tile receives and / or consumes or executes the instructions received via the bus 124 may be similar to the process for executing the instructions received via the CSR bus 122.

In one example, the instruction header (ie, bitmap) indicates to the receive tile that the receive tile needs to consume a particular instruction based on the bitmap associated with that instruction. Bitmaps can have a specific width defined for a bit. Instructions are typically transferred from one tile to the next based on the parameters of the instruction. In one implementation, the width of the instruction bus 124 may be configured to be less than the size / width of the instruction. Therefore, in such a configuration, the transmission of instructions takes place over several cycles, and the bus stop of instruction bus 124 places a decoder to put the instructions received in that tile in the appropriate target instruction buffer associated with that tile. Have.

As further described below, the tiles in tile sets 112, 114 are generally configured to support two broad categories of instructions. The two broad categories are also called instruction types. Instruction types include tensor operation (TensorOp) instructions and direct memory access (DMAOp) instructions. In some implementations, the DMAOp instruction has one or more specializations that are allowed to be simultaneous. One or more specializations may be referred to as DMAOp instruction subtypes or opcodes. In some cases, all unique and / or valid DMAOp instruction type / subtype tuples will have separate instruction buffers within a particular tile.

At a particular tile of tiles 112, 114, the bus stop associated with the instruction bus 124 examines the header bitmap to determine the instruction type / subtype. Instructions may be received by the tile and subsequently written to the tile's instruction buffer prior to execution of the instruction by the tile. The instruction buffer of the tile on which the instruction is written can be determined by the instruction type and subtype indicator / field. The instruction buffer may include a first-in first-out (FIFO) control scheme that prioritizes the consumption of one or more related instructions. Therefore, under this FIFO control scheme, instructions of the same type / subtype will always be executed in the order in which the instructions arrive on the instruction bus.

The different instruction buffers in the tile are the TensorOp instruction buffer and the DMAOp instruction buffer. As mentioned above, the instruction type includes a TensorOp instruction and a DMAOp instruction. For DMAOp instructions, instruction subtypes (indicating the "write destination" buffer position) include: 1) mesh inbound instruction buffer; 2) mesh outbound instruction buffer; 3) narrow-wide DMA instruction buffer; 4) Wide-narrow DMA instruction buffer; and 5) Ringbus DMA instruction buffer. These buffer positions are described in more detail below with reference to FIG. Wide and narrow specifications are used throughout the specification and generally refer to the approximate width size (bits / bytes) of one or more memory units. As used herein, "narrow" may refer to one or more memory units, each having a size or width of less than 16 bits, and "wide" means a size or width, each less than 64 bits. Can refer to one or more memory units with.

The mesh bus 126 provides a data communication path different from that of the CSR bus 122, the command bus 124, and the ring bus 128 (described later). As shown in FIG. 1, mesh bus 126 provides a communication path that joins or connects each tile to its corresponding neighbor tile in both the X and Y dimensions. In various implementations, the mesh bus 126 can be used to transfer the amount of input activation between one or more narrow memory units in adjacent tiles. As shown, mesh bus 126 does not allow input activation data to be transferred directly to non-proximity tiles.

In various implementations, the mesh bus 126 and the various tiles connected via the mesh bus 126 may have the following configurations. The four corner tiles of the mesh have two outbound ports and two inbound ports. The four edge tiles of the mesh have three inbound ports and three outbound ports. All non-edge, non-corner tiles have 4 inbound ports and 4 outbound ports. In general, in the NxN tile layout example, an edge tile is a tile with only three neighbor tiles and a corner tile is a tile with two neighbor tiles. For data flow methodologies over the mesh bus 126, in general, all input activations arriving via the mesh bus 126 for a particular tile must be committed to one or more narrow memory units in that tile. .. Further, for tile configurations with less than four inbound ports, the DMAOp instruction may write a zero value to a location in the tile's narrow memory instead of waiting for data on a nonexistent input port. Similarly, for tile configurations with less than four outbound ports, the DMAOp instruction does not perform transfer-related narrow memory reads and port writes to non-existent ports.

In some implementations, the location or address of the narrow memory unit to which a particular input activation will be written or read is a tensor traversal unit based on the inbound / outbound DMAOp given via the mesh bus 126. (Hereinafter, "TTU") will be generated. The inbound DMAOp and the outbound DMAOp may be executed at the same time, and the required synchronization will be managed by the synchronization flag control method managed by the controller 102. The TTU will be described in more detail below with reference to FIGS. 2 and 3.

The ring bus 128 starts from the controller 102 and, like the CSR bus 122 and the command bus 124, provides communication that connects the tiles 112 and 114 via a bus data path that connects the tiles 112 and 114 back to the controller 102 in a ring shape. In various implementations, the ring bus 128 generally connects or joins all wide memory units (discussed in more detail below with reference to FIG. 2) in all tiles 112, 114. Therefore, the payload width of the ring bus 128 corresponds to the width of the wide memory unit arranged in each tile of the tile sets 112 and 114. As mentioned above, the ring bus 128 also includes a bitmap header indicating the tiles that need to consume payload data, including instructions or parameters communicated over the ring bus 128.

For data (ie, payload) received at a particular tile via the ring bus 128, in response to receiving information, each tile zeros the position data shown in the bitmap header specific to the receiving tile (ie). Clear) and then transfer the data to another tile. Therefore, when the header bitmap has no remaining bitset data indicating the particular tile for which the payload is to be received, the transfer of the payload to another tile will stop. Payload data generally refers to activations and weights used by one or more tiles during a tensor calculation performed based on the execution of a deeply nested loop.

In some implementations, the controller 102 may be described as part of the ring bus 128. In one example, for a DMAOp instruction executed within a particular tile, the controller 102 is used to pop the data / payload from the ring bus stop and transfer that payload to the ring bus stop in the next tile in the ring. May be good. The controller 102 can further commit payload data to one or more wide memory units in the tile, if required by instructions in the bitmap header. The address of one or more wide memory units that need to write data may be generated by the DMAOp instruction within a particular tile.

In various implementations, each tile of tile sets 112, 114 can be either a producer of payload data or a consumer of payload data. If the tile is the producer of payload data, the tile reads data from one or more of its wide memory units and puts the ring bus 128 for consumption by one or more other tiles. Multicast via. If a tile is a consumer of payload data, the tile receives the data, writes it to one or more wide memory units within that tile, and transfers the payload data for consumption by one or more other tiles. .. With respect to the transfer of payload data through the ring bus 128, there is usually only one producer / master of data on the ring bus 128 at any given time. The DMAOp instruction execution order (eg, FIFO control scheme) in all tiles will ensure that there is only one producer / master of data on the ring bus 128 at a given time.

In some implementations, the controller 102 uses a synchronization flag control architecture to ensure that there is only one producer / master of payload data on the ring bus 128 at a given time. In one example, each write to the ring output by the tile will trigger the corresponding increment of the sync flag count. The controller 102 can examine the payload data to determine the number of data chunks or segments that contain the payload. The controller 102 then monitors the execution by the tile to ensure that the expected number of data segments are transferred and / or consumed by that tile before the other tiles execute in master mode.

If there is a local multicast group on the ring bus 128 that has no overlapping space and is connected over the ring bus 128, then there is only one data producer / master on the ring bus 128 at a given time. There are exceptions to guaranteeing. For example, tile 0 (master) multicasts (ie, generates data) to a tile in the grouping of tiles 0 to 3, and tile 4 (master) to a tile in the grouping of tiles 4 to 7. You can do the same for it. An important requirement of this dual-master multicast method is to prevent different multicast groups from referencing each other's data packets, as packet duplication can occur and cause one or more data calculation errors. be.

As shown in FIG. 1, the controller 102 provides a communication data path for coupling or connecting tiles in tile sets 112, 114 to the I / O 110 and includes several core functions. The core function of the controller 102 is generally to supply one or more I / O input activations to the tiles in the tileset 112, 114, one or more input activations and parameters received from the I / O 110. To the tile, to feed the tile one or more instructions received from the I / O 110, to send the I / O output activation to the host interface 108, and to the CSR bus 122 and the ring bus 128. Includes functioning as a ring stop. As described in more detail below, the first tileset 112 and the second tileset 114 perform one or more tensor calculations performed based on deep loop nesting consisting of inner and outer loops, respectively. Includes multiple tiles used to do this.

The system 100 generally operates as follows. The host interface 108 gives the controller 102 one or more instructions that define the direct memory access operation (DMAOp) that occurs for a given calculation. The descriptors associated with the instructions supplied to the controller 102 will contain the information required by the controller to facilitate large-scale dot product calculations associated with the multidimensional data array (tensor). In general, the controller 102 receives input activations, tile instructions, and model parameters (ie, weights) from the host interface 108 to perform tensor calculations on a given layer of neural network. The controller 102 can then multicast the instruction to tiles 112, 114 in the data flow scheme defined by the instruction. As mentioned above, a tile that consumes an instruction can then start broadcasting a new / subsequent instruction to another tile based on the bitmap data in the instruction header.

With respect to the data flow, the input activation and parameters are sent to the tiles of tile sets 112, 114 via the ring bus 128. Each of the tiles 112, 114 will store a subset of the input activations required to calculate the subset of output activations assigned to that particular tile. The DMAOp instruction for the tile moves the input activation from wide memory to narrow memory. Computations within the tile begin when the required input activations, parameters / weights, and calculation instructions (TTU operations, memory addresses, etc.) are available on the tile. For the computations that occur in the tile, the MAC operator in the tile (discussed below) completes all the inner product operations defined by the instruction set, and the preactivation function is applied to the result of the multiplication operation (ie, the product- sum value ). And finish.

The result of one or more tensor computations involves writing the output activation of the compute layer to the narrow memory unit of the tile performing the computation. In one tensor calculation, there will be an output edge activation transfer to neighboring tiles via the mesh bus 126. Transferring the output edge activation to neighboring tiles is required to calculate the output activation for subsequent layers if the calculation spans multiple layers. When the calculations for all layers are complete, DMAOp transfers the final activation to the classifier tile 116 via the ring bath 128. The controller 102 then reads the final activation from the classifier tile 116 and executes DMAOp to move the final activation to the host interface 108. In some implementations, the classifier portion 116 performs the calculation of the output layer (ie, the last layer) of the NN. In another embodiment, the output layer of the NN is one of a classifier layer, a regression layer, or another layer type generally associated with neural networks.

FIG. 2 shows an exemplary neural network (NN) calculation tile 200. In general, the exemplary tile 200 may correspond to any tile in the first tile set 112 and the second tile set 114 described above with reference to FIG. In various implementations, the compute tile 200 may also be referred to or so referred to as the compute unit 200. Each calculation tile 200 is a self-contained calculation unit configured to execute instructions independently for other corresponding tiles in the tile sets 112, 114. As briefly described above, each calculation tile 200 executes two types of instructions, the TensorOp instruction and the DMAOp instruction. In general, each instruction type involves computational operations associated with deep loop nesting, so each instruction type will generally be executed over multiple time epochs to ensure the completion of all loop iterations.

As discussed in more detail below, the different instruction types are performed by independent control units within the calculation tile 200 that synchronize on the data via synchronization flag control managed within the calculation tile 200. The synchronization flag control manages the concurrency between executions of different instruction types within the calculation tile 200. Each computational operation associated with each instruction type is performed in a strict issue order (ie, first in, first out). For the two instruction types, TensorOP and DMAOp, there is no ordering guarantee between these different instruction types, and each type is treated as a separate control thread by calculation tile 200.

With respect to the data flow configuration, the calculation tile 200 generally includes a data path 202 and a data path 205, each of which provides a communication path for data flows in and out of the calculation tile 200. As mentioned above, the system 100 includes three different data bus structures laid out in a ring configuration, namely the CSR bus 122, the instruction bus 124, and the ring bus 128. With reference to FIG. 2, the data path 205 corresponds to the command bus 124, and the data path 202 generally corresponds to one of the CSR bus 122 and the ring bus 128. As shown, the data path 202 includes a ring output 203 that provides an output path for data exiting the calculation tile 200 and a ring input 204 that provides an input path for data entering the calculation tile 200.

The calculation tile 200 further includes a TensorOp control 206 including a TensorOp tensor traversal unit (TTU) 226 and a DMAOp control 208 including a DMAOpTTU228. The TensorOp control 206 generally manages writes to the TensorOpTTU register 232 and reads from the TensorOpTTU register 232 and manages traverse operations for execution by the TensorOpTTU226. Similarly, the DMAOp control 208 generally manages writes to the DMAOpTTU register 234 and reads from the DMAOpTTU register 234, and manages traverse operations for execution by the DMAOpTTU 228. The TTU register 232 includes an instruction buffer for storing one or more instructions including an operation to be performed by the TensorOpTTU226 in the execution of an instruction by the TensorOp control 206. Similarly, the TTU register 234 includes an instruction buffer for storing one or more instructions including an operation to be performed by the TTU 228 in the execution of an instruction by DMAOp control 208.

As further described below, the TTU 226 and / or 228 are used by compute tiles 200 to traverse array elements of one or more tensors, typically residing in narrow memory 210 and wide memory 212. In some implementations, the TTU226 is used by the TensorOp control 206 to provide a tensor operation to traverse the dimensions of a multidimensional tensor based on the execution of deep loop nesting.

In some implementations, an instruction for execution by the calculation tile 200 arrives at the tile via the data path 205 (ie, part of the instruction bus 124). The calculation tile 200 examines the header bitmap to determine the instruction type (TensorOp or DMAOp) and the instruction subtype (read or write operation). The instructions received by the calculation tile 200 are subsequently written to a particular instruction buffer, depending on the instruction type. In general, instructions are received and stored (ie, written to a buffer) prior to the execution of the instruction by the components of calculation tile 200. As shown in FIG. 2, each instruction buffer (ie, TensorOpTTU register 232 and DMAOpTTU register 234) can include a first in, first out (FIFO) control scheme that prioritizes the consumption (execution) of one or more related instructions. ..

Briefly as described above, tensors are multidimensional geometric objects, and exemplary multidimensional geometric objects include matrices and data arrays. An algorithm containing deeply nested loops may be performed by calculation tile 200 to perform tensor calculations by iterating over one or more nested loops and traversing the N-dimensional tensor. In one exemplary computational process, each loop in a loop nest may be responsible for traversing a particular dimension of an N-dimensional tensor. As described herein, the TensorOp control 206 typically traverses the dimensional elements of a particular tensor construct and accesses them to complete the computation defined by the deeply nested loops. Manage one or more tensor operations to drive.

The calculation tile 200 further includes a narrow memory 210 and a wide memory 212. The narrow and wide designation generally refers to the width size (bits / bytes) of the memory units of the narrow memory 210 and the wide memory 212. In some embodiments, the narrow memory 210 comprises a memory unit each having a size or width of less than 16 bits, and the wide memory 212 comprises a memory unit each having a size or width less than 32 bits. In general, the calculation tile 200 receives the input activation via the data path 205 and the DMA control 208 performs an operation to write the input activation to the narrow memory 210. Similarly, the calculation tile 200 receives the parameter (weight) via the data path 202, and the DMA control 208 performs an operation to write the parameter to the wide memory 212. In some implementations, the narrow memory 210 is allowed any controller (eg, TensorOp control 206 or DMAOp control 208) to access the shared memory unit of the narrow memory 210 for each memory cycle. It may include a memory arbiter commonly used in shared memory systems to determine if it is.

The calculation tile 200 further includes an input activation bus 216 and a MAC array 214 each containing a plurality of cells including a MAC operator 215 and a sum register 220. In general, the MAC array 214 uses the MAC operator 215 and the sum register 220 across a plurality of cells to perform tensor calculations, including arithmetic operations related to inner product calculations. The input activation bus 216 provides a data path in which input activation is provided by the narrow memory 210, one for each access by each MAC operator 215 of the MAC array 214. Therefore, on the basis of one broadcast of the input activation, each single MAC operator 215 in a particular cell will receive the input activation. Arithmetic operations performed by the MAC operator of the MAC array 214 typically multiply the input activation given by the narrow memory 210 by the parameters accessed from the wide memory 212 to produce a single output activation value. Including that.

During an arithmetic operation, the partial sums are accumulated and stored in the corresponding, eg, total register 220, or written to wide memory 212 and re-accessed by a particular cell in MAC array 214 to complete subsequent multiplication operations. You may. The tensor calculation can be described as having a first part and a second part. The first part is completed when the multiplication operation generates the product-sum value by, for example, completing the multiplication of the input activation and the parameter to generate the product- sum value . The second part includes the application of the nonlinear function to the sum of products value , and the second part is completed when the output activation is written to the narrow memory 210 after the application of the function.

Computation tile 200 further includes an output bus 218, a nonlinear unit (NLU) 222 including an output activation pipeline 224, an NLU control 238, and a reference map 230 showing the core attributes of the components within the calculation tile 200. Reference map 230 is shown for clarity, but is not included in calculation tile 200. Core attributes include whether a particular component is a unit, storage device, operator, controller, or data path. Generally, when the first part of the tensor calculation is completed, the sum-of-product value is given to the NLU222 from the MAC array 214 via the output bus 218. After arriving at NLU222, the data identifying the activation function received via the activation pipeline 224 is applied to the product-sum value , and then the output activation is written to the narrow memory 210. In some implementations, the output bus 218 includes at least one pipelined shift register 236, and completing the second part of the tensor calculation uses the shift register 236 of the bus 218. Includes shifting the sum-of-product value towards the non- linear unit 222 .

For example, with respect to the inner product calculation of two multidimensional data arrays for a single calculation tile 200, the MAC array 214 provides robust single instruction multiple data (SIMD) functionality. SIMD generally states that all parallel units (multiple MAC operators 215) share the same instruction (based on deep loop nesting), but each MAC operator 215 executes the instruction on different data elements. means. In one basic example, the arrays [1,2,3,4] and [5,6,7,8] are added element by element to form the array [6,8,10,12] in one cycle. To get it, you usually need four arithmetic units to perform the operation on each element. By using SIMD, the four units can share the same instruction (eg, "addition") and perform calculations in parallel. Since the instructions are shared, the requirements for instruction bandwidth and instruction memory are reduced, which improves efficiency. Therefore, the system 100 and the calculation tile 200 provide improved acceleration and parallel processing over the conventional method in tensor calculation.

In one example, and as described in more detail below, a single instruction may generate multiple computational tiles 200 (tile sets 112, 114 of FIG. 1) by the controller 102 due to consumption by the plurality of MAC arrays 214. See). In general, a neural network layer can contain multiple output neurons, which are divided so that tensor calculations related to a subset of output neurons can be assigned to specific tiles in tilesets 112, 114. Can be done. Each tile of tile sets 112, 114 can then perform related tensor calculations on different groups of neurons for a given layer. Thus, the computational tile 200 can provide at least two forms of parallel processing: 1) One form is the output activity (corresponding to a subset of output neurons) among multiple tiles of tile sets 112, 114. 2) Other forms include simultaneous computation (by a single instruction) of multiple subsets of output neurons based on the division between tiles of tile sets 112, 114.

FIG. 3 shows an exemplary tensor traversal unit (TTU) structure 300 containing four tensors to be tracked, each with a depth of eight. The TTU 300 generally includes a counter tensor 302, a stride tensor 304, an initial tensor 306, and a limiting tensor 308. The TTU 300 further includes an adder bank 310 and a tensor address index 312. As mentioned above, a tensor is a multidimensional geometric object and must be given an index for each dimension in order to access the elements of the tensor. Since the tensor is stored in the narrow memory 210 and the wide memory 212, the set of tensor indexes must be translated into the set of memory addresses. In some implementations, the conversion of an index to a memory address is done by making the memory address a linear combination of the indexes and reflecting the address via the tensor address index 312.

There is a TTU for each control thread, and in the calculation tile 200 there is a control thread (TensorOP and DMAOp) for each instruction type. Therefore, as mentioned above, the calculation tile 200 has two sets of TTUs: 1) TensorOpTTU226; and 2) DMAOpTTU228. In various implementations, the TensorOp control 206 causes the TTU 300 to load the TensorOpTTU counter 302, limit 308, stride value 304 at the start of a particular tensor operation and does not change the register value before the instruction is retired. Each of the two TTUs will need to generate an address for the following memory address ports in calculation tile 200: 1) wide memory 212 address port, and 2) presented as 4 address ports 4 Narrow memory 210 with two independent arbitrated banks.

As mentioned above, in some implementations, the narrow memory 210 has access to the shared memory resource of the narrow memory 210 by which controller (eg, TensorOp control 206 or DMAOp control 208) for each memory cycle. It may include a memory arbiter commonly used in shared memory systems to determine if it is allowed to do so. In one example, the different instruction types (TensorOp and DMAOp) are independent control threads that require arbitration and require memory access. When a particular control thread commits a tensor element to memory, that control thread increments the tensor reference counter 302 committed to memory.

In one example, when the TensorOp control 206 executes an instruction to access a particular element of the tensor, the TTU 300 can determine the address of a particular element of the tensor, and the control 206 accesses storage, eg, narrow memory 210. Then, the data representing the activation value of a specific element can be read out. In some implementations, the program can include nested loops, and control 206 is a two-dimensional array variable within the nested loop according to the current index variable value associated with the nested loop. Instructions can be executed to access the elements of.

The TTU 300 may simultaneously maintain a traversal state for up to X TTU rows for a given tensor. Each tensor resident in the TTU 300 at the same time occupies a dedicated hardware tensor control descriptor. The hardware control descriptor can consist of an X-number TTU counter 302 per row position, a stride 304, and a limit register 308 supporting a tensor with a maximum X-number TTU counter per row dimension. In some implementations, the number of rows and the number of counters per row can be different.

For a given position register, the final memory address is calculated from an addition operation involving adding the position registers together. The base address is incorporated in the counter 302. One or more adders are shared for a tensor reference that resides in the same memory. In one implementation, there can only be a single load / store on any given port in the cycle, so multiple tensor references residing in the same narrow or wide memory can cause those counters to be arbitrary. It is the function of loop nesting control to ensure that it is not incremented in a given cycle. The use of registers to calculate memory access address values, including the determination of offset values, is described in patent application serial number 15/014, entitled "Matrix Processing Appliance", filed February 3, 2016. , 265, which is expressly incorporated herein by reference in its entirety.

For example, when a software algorithm processes an N-dimensional tensor, nested loops may be used in which each loop serves to traverse each dimension of the N-dimensional tensor. The multidimensional tensor can be a matrix or a multidimensional matrix. Each dimension of the N-dimensional tensor can contain one or more elements, and each element can store its own data value. For example, a tensor can be a variable in a program, and the variable can have three dimensions. The first dimension may have a length of 300 elements, the second dimension may have a length of 1000 elements, and the third dimension may have a length of 20 elements. May have.

Traversing a tensor within a nested loop may require the calculation of the element's memory address value to load or store the element's corresponding data value. For example, a for loop is a nested loop, and three loops tracked by three loop index variables can be nested to traverse a 3D tensor. In some cases, the processor may need to execute a loop boundary condition, such as setting the loop boundary of the inner loop with an outer loop index variable. For example, when deciding whether to exit the innermost loop of a nested loop, the program may use the current value of the loop index variable of the innermost loop and the loop index variable of the outermost loop of the nested loop. You may compare it with the current value of.

In general, when a processing unit in a compute tile executes an instruction to access a particular element of a tensor, the tensor traversal unit determines the address of a particular element in the tensor, and the processing unit accesses the storage medium (memory). Allows you to read data that represents the value of a particular element. For example, a program can contain a nested loop, where the processing unit executes an instruction and follows a two-dimensional array within the nested loop according to the current index variable value associated with the nested loop. You can access the elements of the variable. Based on the current index variable value associated with the nested loop, the tensor traversal unit may determine the offset value that represents the offset from the first element of the 2D array variable. The processing unit then uses the offset value to access certain elements of the 2D array variable from memory.

The following provide template parameters that may be used to instantiate the specialized TTU300. 1) X-number TTU rows; 2) X-number TTU counters per row; 3) X-number TTU adder units; 4) Shows shared adder references for each TTU row; and 5) Per counter Indicates the X counter size [TTU] [row] [depth]. All TTU registers are architecturally visible. The address of the particular tensor element that needs to be accessed for the calculation (ie, the tensor address 312) is the result of counter addition. When an increment signal is issued from the control thread to a row of TTUs, the TTU 300 performs a single cycle operation, increments the innermost dimension by the stride 304 of that dimension, and propagates the rollover through all depths. do.

In general, the TTU 300 determines a state associated with one or more tensors. The state can include a loop boundary value, the current loop index variable value, a dimensional multiplier for calculating the memory address value, and / or a program counter value for processing the branch loop boundary. The TTU 300 can include one or more tensor state elements and an arithmetic logic unit. Each of the tensor state elements can be a storage element, such as a register or any other suitable storage circuit. In some implementations, the tensor state elements may be organized into physically or logically different groups, as described in more detail in patent application serial number 15/014,265.

FIG. 4 shows an exemplary architecture that includes a narrow memory 210 that broadcasts activation 404 to one or more multiply-accumulate (MAC) operators via input bus 216. The shift register 404 provides a shift function in which the activation 404 is sent to the input bus 216 one at a time for receipt by one or more MAC operators 215 in the MAC cell 410. In general, the MAC cell 410 including the MAC operator 215 can be defined as a calculated cell for calculating the partial sum, and in some implementations it is configured to write the partial sum data to the output bus 218. As shown, cell 410 may consist of one or more MAC operators. In one implementation, the number of MAC operators 215 in the MAC cell 410 is called the cell issue width. As an example, a double-issued cell calculates the multiplication of two activation values (from narrow memory 210) and two parameters (from wide memory 212) and combines the result of the two multipliers with the current partial sum. Refers to a cell with two MAC operators that can perform addition between.

As mentioned above, the input bus 216 is a broadcast bus that provides input activation to the MAC operator 215 of the linear unit (ie, MAC array 214). In some implementations, the same input is shared among all MAC operators 215. The width of the input bus 216 must be wide enough to supply the corresponding number of cells with broadcast inputs for a given MAC array 214. The following example will be considered to illustrate the structure of the input bus 216. When the number of cells in the linear unit is equal to 4 and the activation width is equal to 8 bits, the input bus 216 can be configured to give up to 4 input activations per cycle. In this example, all cells in the MAC array 214 will access only one of the four broadcast activations.

Based on the instruction's TensorOp field setting received by the calculation tile 200, the cells of the MAC array 214 may need to perform the calculation with the same input activation. This may be referred to as a Zout split within the cell of the MAC array 214. Similarly, Zin splitting within a cell occurs when the cell of MAC array 214 requires a different activation to perform the calculation. In the former case, a single input activation is replicated four times and the four activations read from the narrow memory 210 are broadcast over four cycles. In the latter case, reading of the narrow memory 210 is required for each cycle. In the above example, the TensorOp control 206 orchestrate this broadcast method based on the execution of instructions received from the controller 102.

FIG. 5 shows an exemplary architecture that includes an output bus 218 to provide output activation to the narrow memory units 210 of FIGS. 2 and 4. In general, each MAC cell 215 of the MAC array 214 in calculation tile 200 calculates different output activations. However, for an output feature array, if the output feature depth is less than the number of MAC cells 215 in the calculation tile 200, the cells may be grouped to form one or more cell groups. All MAC cells 215 in a cell group compute the same output (ie, for an output feature map), but each cell only computes a subset of outputs that correspond to a subset of Zin dimensions. As a result, the output of MAC cell 215 is here a partial sum, not the final linear output. In some implementations, NLU222 aggregates these partial sums into a final linear output based on the control signal given to NLU222 by NLU control 238.

As mentioned above, the output bus 218 is a pipelined shift register. In various implementations, the first part of the tensor calculation is finished, indicating that the TensorOp control 206 needs to write out the partial sum (by executing the instruction), the partial sum given to the output bus 218. There will be parallel loads. The number of parallel loads will correspond to the number of MAC cells in the calculation tile 200. The TensorOp control 206 then shifts out the partial sum and feeds it through the non-linear pipeline. In some implementations, there may be situations where not all MAC cells in the tile are actually used to perform the calculation. In such a situation, not all partial sums that are shifted to the output bus will be valid. In this example, the TensorOp control 206 may provide a control signal to the MAC array 214 to indicate the number of valid cells to be shifted out. The amount of parallel load loaded on the output bus 218 still corresponds to the number of MAC cells in the calculation tile, but only the valid values will be shifted out and committed to the narrow memory 210.

FIG. 6 is an exemplary flow chart of process 600 for performing a tensor calculation using the 200 neural network calculation tiles of FIG. The process 600 starts at block 602 and the narrow memory 210 of the calculation tile 200 sends (ie, broadcasts) activations one by one to the input activation data bus 216. The activation value is stored in the narrow memory 210. The narrow memory 210 may be a collection of static random access memory (SRAM) banks that allow addressing to specific memory locations to access the amount of input. The activation read from the narrow memory 210 is broadcast via the input activation bus 216 to the linear cells of the MAC array 214 (ie, the linear unit) containing the plurality of MAC operators 215 and the sum register 220. .. At block 604 of process 600, the MAC operator 215 of the calculation tile 200 each receives two inputs, one input (activation) is received from the input activation bus 216; another input (parameter) is the wide memory 212. Received from. Therefore, activation supplies one of the inputs for each MAC operator 215, and each MAC operator 215 in the cell of the MAC array 214 gets their second multiplier input from the wide memory 212.

In block 606 of process 600, the MAC array 214 of the calculation tile 200 performs a tensor calculation including an inner product calculation based on the elements of the data array structure accessed from the memory. The wide memory 212 has a width in bits equal to the width of the linear unit (32 bits). Therefore, the linear unit (LU) is a SIMD vector arithmetic logic unit (ALU) unit that receives data from a vector memory (ie, wide memory 212). In some implementations, the MAC operator 215 may also obtain an accumulator input (partial sum) from the wide memory 212. In some implementations, there is time division for wide memory 212 ports for reads and / or writes for two different operands (parameters and partial sums). In general, the wide memory 212 may have a limited number of ports in order to optimize the area. As a result, if it is necessary to read an operand (eg, a parameter) from the wide memory 212 and write an operand (eg, a partial sum) to the wide memory 212 at the same time, the pipeline associated with the particular operand may stop functioning.

At block 608, the calculation cell of calculation tile 200 (having MAC operator 215 and sum register 220) produces at least one product-sum value based on the multiplication operation performed by the MAC / calculation cell. The result of the MAC cell operation includes either the partial sum written back to the wide memory (during the partial sum arithmetic operation) or the product sum value sent to the output bus 218. At block 610, NLU222 of calculation tile 200 applies a non-linear activation function to the sum-of-product value and then writes the output activation to the narrow memory 210. In some implementations, the output bus 218 is a shift register that can accumulate parallel loads of results / sum of products from the MAC operator 215, but with the application of non-linear functions and the narrow memory 210 of the same tile. Shift them out one at a time for the write operation to.

Embodiments of the subject matter and functional operation described herein are computer software or firmware tangibly implemented in a digital electronic circuit system, including the structures disclosed herein and their structural equivalents. In computer hardware, or in one or more combinations thereof. Embodiments of the subject matter described herein are tangible, non-temporary, as one or more computer programs, i.e., for execution by a data processor or to control the operation of the data processor. It can be implemented as one or more modules of computer program instructions encoded on a program carrier. Alternatively, or in addition, program instructions are generated to encode information for transmission to a receiver device suitable for execution by a data processing device, such as a machine-generated electrical signal, optical. It can be encoded on an artificially generated propagated signal, such as a signal or an electromagnetic signal. The computer storage medium can be a computer readable storage device, a computer readable storage board, a random or serial access memory device, or a combination thereof.

The processes and logical flows described herein are performed by one or more programmable computers running one or more computer programs to operate on input data and perform functions by producing outputs. obtain. Processes and logic flows can also be performed by special purpose logic circuits such as FPGAs (Field Programmable Gate Arrays), ASICs (Application Specific Integrated Circuits), or GPGPUs (General Purpose Graphic Processing Devices), and the devices can also be implemented by them. ..

A processor suitable for executing a computer program may be based, for example, on a general purpose microprocessor, a special purpose microprocessor, or both, or any kind of central processing unit. In general, the central processing unit receives instructions and data from read-only memory and / or random access memory. An essential element of a computer is a central processing unit for executing instructions and one or more memory devices for storing instructions and data. In general, a computer also includes, for example, one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or receives data from the one or more mass storage devices. It is operably coupled to transfer data to the one or more mass storage devices, or both. However, the computer does not need to have such a device.

Computer-readable media suitable for storing computer program instructions and data are all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices; eg internal. Includes magnetic disks such as hard disks or removable disks. Processors and memory can be supplemented by special purpose logic circuits or incorporated into special purpose logic circuits.

Although the present specification contains details of many particular embodiments, they should not be construed as a limitation to the scope of any invention or what can be claimed, and to a particular embodiment of a particular invention. It should be interpreted as a statement that can be a peculiar feature. Certain features described herein in the context of separate embodiments may also be realized in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment can be realized separately in multiple embodiments or in any suitable partial combination. Further, features are described above as acting in a combination and may even be initially claimed as such, but one or more features from the claimed combination may in some cases be the combination. Combinations that can be removed from and claimed from can be directed to partial combinations or variants of partial combinations.

Similarly, the actions are shown in the figure in a particular order, but such actions need to be performed in that particular order or in a contiguous order to achieve the desired result. It should not be understood, or that all indicated actions need to be performed. In some situations, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the program components and systems described are generally single software. It should be understood that it can be integrated into a product or packaged into multiple software products.

Other implementation examples are summarized in the following examples.
Example 1: A computing unit for accelerating tensor computation, with a first memory bank having a first data width and storing at least one input activation and at least one output activation. A second memory bank that has a second data width that is larger than the first data width and stores one or more parameters used when performing calculations, and parameters from the second memory bank. A first traversal unit comprising at least one cell containing at least one product-sum (“MAC”) operator that receives and performs a calculation, and a first traversal unit that communicates data with at least the first memory bank. Is configured to give a control signal to the first memory bank to give input activation to the data bus accessible by the MAC operator, and the compute unit is one associated with at least one element of the data array. Or perform multiple calculations, one or more of which are performed by the MAC operator, partially multiplying the input activation received from the data bus with the parameters received from the second memory bank. Including, calculation unit.

Example 2: A compute unit performs one or more of its computations by performing a loop nest containing multiple loops, and the loop nest structure traverses one or more dimensions of the data array. The computational unit according to Example 1, comprising each loop used by the first traversal unit.

Example 3: One or more computations are performed based in part on the tensor operation given by the first traversal unit, which is a loop nesting to access one or more elements of the data array. The computational unit according to Example 2, comprising the structure.

Example 4: A second memory location configured to access at least one memory location in the first memory bank and at least one memory location in the second memory bank based on instructions received from a source outside the compute unit. The calculation unit according to one of Examples 1 to 3, further comprising 2 traversal units.

Example 5: The computational unit according to Example 4, wherein the first traversal unit is a tensor arithmetic traversal unit, the second traversal unit is a direct memory access traversal unit, and the data array corresponds to a tensor containing a plurality of elements. ..

Example 6: The calculation unit contains a non-linear unit, the first part of the calculation involves generating a product-sum value based on the multiplication operation, and the second part of the calculation is one or more non-linear functions by the non-linear unit. The calculation unit according to one of Examples 1 to 5, which comprises applying to the product-sum value of.

Example 7: The computational unit according to Example 6, wherein the computation performed by the computational unit comprises using a shift register to shift the sum-of-product value to a non- linear unit.

Example 8: Further including a part of a ring bus extending outside the compute unit, the ring bus is between the first memory bank and the memory bank of another adjacent compute unit and between the second memory bank and the other proximity. The calculation unit according to one of Examples 1 to 8, which provides a data path between the calculation unit and the memory bank of the calculation unit.

Example 9: The computational unit according to one of Examples 1-8, wherein the second memory bank is configured to store at least one of a partial sum or one or more pooling layer inputs. ..

Example 10: A computer-implemented method for accelerating tensor calculations in response to a first memory bank having a first data width receiving a control signal from a first traversal unit. The first memory bank is located within the compute unit and the first input activation is accessed by at least one cell of the compute unit. Given by a possible data bus, the method further comprises receiving one or more parameters from a second memory bank having a second data width larger than the first data width by at least one cell. , At least one cell contains at least one multiply-sum (“MAC”) operator, and the method further comprises performing one or more calculations related to at least one element of the data array by means of the MAC operator. One or more computations include, in part, a multiplication operation with at least one input activation accessed from the data bus and at least one parameter received from the second memory bank.

Example 11: One or more computations are performed based in part on the fact that the compute unit performs a loop nest containing multiple loops, and the structure of the loop nest is one or more dimensions of the data array. The method performed by the computer of Example 10, comprising each loop used by the first traversal unit to traverse.

Example 12: A method performed by a computer according to Example 11, further comprising providing a tensor operation, including a loop nesting structure for accessing one or more elements of the data array, by a first traversal unit.

Example 13: The first traversal unit is a tensor arithmetic traversal unit, the second traversal unit is a direct memory access traversal unit, and the data array corresponds to a tensor containing a plurality of elements. The method performed by the computer described in 1.

Example 14: Performed by the computer according to one of Example 10-13, further comprising performing the first part of one or more calculations by generating a product-sum value based on a multiplication operation. How to do it.

Example 15: By the computer of Example 14, further comprising applying a nonlinear function to the sum of products value to perform a second part of one or more computations by obtaining one or more output activations. How it is carried out.

Example 16: A non-temporary computer-readable storage medium containing instructions that can be executed by one or more processors, the instructions causing one or more processors to perform an operation when so executed. The operation involves sending a first input activation by the first memory bank in response to the first memory bank having the first data width receiving a control signal from the first traversal unit. , The first memory bank is located within the compute unit, the first input activation is given by the data bus accessible by at least one cell of the compute unit, and the operation is further by at least one cell. At least one cell contains at least one product-sum (“MAC”) operator, including receiving one or more parameters from a second memory bank that has a second data width that is greater than the first data width. The operation further comprises performing one or more operations related to at least one element of the data array by means of a MAC operator, the one or more operations being partially from the data bus. Includes a multiplication operation of at least the first input activation accessed and at least one parameter received from the second memory bank.

Example 17: One or more computations are performed based in part on the fact that the compute unit performs a loop nest containing multiple loops, and the structure of the loop nest is one or more dimensions of the data array. The non-temporary computer-readable storage medium of Example 16, comprising each loop used by a first traversal unit to traverse.

Example 18: The non-temporary computer-readable storage medium of Example 17, further comprising providing a tensor operation, including a loop nesting structure for accessing one or more elements of the data array, by a first traversal unit. ..

Example 19: The non-temporary computer according to one of Examples 16-18, further comprising performing a first part of one or more calculations by generating a product-sum value based on a multiplication operation. Readable storage medium.

Example 20: The non-discussion of Example 19, further comprising performing a second part of one or more computations by applying a nonlinear function to the sum of products values to obtain one or more output activations. Temporary computer-readable storage medium.

Specific embodiments of the subject have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in different order and still achieve the desired result. As an example, the process shown in the attached figure does not necessarily have to be in the specific order or consecutive order shown to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

Claims (15)

A calculation unit (200) for accelerating tensor calculation of neural networks.
A first memory bank (210) having a first data width and for storing at least one input activation and at least one output activation,
A second memory bank (212) having a second data width larger than the first data width and for storing one or more parameters used when performing a calculation.
Based on instructions received from sources outside the compute unit, access to at least one memory location in the first memory bank to obtain at least one input activation.
A control signal for writing the one or more parameters is given to the second memory bank.
A direct memory access traversal unit configured to access at least one memory location in the second memory bank to obtain the one or more parameters based on the instruction received.
With at least one cell containing at least one product-sum (MAC) operator (215) that receives parameters from the second memory bank and performs calculations.
A tensor traversal unit (226) for data communication with at least the first memory bank is provided, and the tensor traversal unit gives a control signal to the first memory bank and is accessible by the MAC operator. (216) is configured to give input activation,
The compute unit performs one or more computations related to at least one element of the data array, the one or more computations being performed by the MAC operator and partially from the data bus. Includes a multiplication operation between the input activation received and the parameters received from the second memory bank.
The second memory bank is a computational unit configured to store at least one of a partial sum or one or more pooling layer inputs . A calculation unit (200) for accelerating tensor calculation of neural networks.
A first memory bank (210) having a first data width and for storing at least one input activation and at least one output activation,
A second memory bank (212) having a second data width larger than the first data width and for storing one or more parameters used when performing a calculation.
Based on instructions received from sources outside the compute unit, access to at least one memory location in the first memory bank to obtain at least one input activation.
A control signal for writing the one or more parameters is given to the second memory bank.
A direct memory access traversal unit configured to access at least one memory location in the second memory bank to obtain the one or more parameters based on the instruction received.
With at least one cell containing at least one product-sum (MAC) operator (215) that receives parameters from the second memory bank and performs calculations.
A tensor traversal unit (226) for data communication with at least the first memory bank is provided, and the tensor traversal unit gives a control signal to the first memory bank and is accessible by the MAC operator. (216) is configured to give input activation,
The compute unit performs one or more computations related to at least one element of the data array, the one or more computations being performed by the MAC operator and partially from the data bus. Includes a multiplication operation between the input activation received and the parameters received from the second memory bank.
A ring bus extending outside the compute unit provides a data path between the second memory bank and the second memory bank of another adjacent compute unit.
The payload data and the bitmap header are transferred through the ring bus, the payload data includes the parameters, and the bitmap header indicates a calculation tile in which the payload data should be used.
When the calculation tile receives the payload data and the bitmap header, the calculation tile clears the bit set data specific to the calculation tile in the bitmap header, and the payload data and the bitmap header. A calculation unit that sends and to another calculation tile . The compute unit performs one or more of the calculations by performing a loop nest containing the plurality of loops, and the structure of the loop nest traverses one or more dimensions of the data array. The calculation unit according to claim 1 or 2 , comprising each loop used by said tensor traversal unit. The one or more computations are performed based in part on the tensor operation given by the tensor traversal unit, which is a loop nesting structure for accessing one or more elements of the data array. 3. The calculation unit according to claim 3 . The calculation unit includes a non-linear unit, a first part of the calculation comprises generating a product-sum value based on the multiplication operation, and a second part of the calculation is the product of non-linear functions by the non-linear unit. Any one of claims 1 to 4 , comprising obtaining at least one output activation by applying to the sum value , wherein the first memory bank stores the at least one output activation. The calculation unit described in. The calculation unit according to claim 5 , wherein the calculation performed by the calculation unit comprises using a shift register to shift the product-sum value to the non- linear unit. Further comprising a portion of a ring bus extending outside the compute unit, the ring bus may be between the first memory bank and a memory bank of another adjacent compute unit and between the second memory bank and the other. The calculation unit according to any one of claims 1 and 3 to 6, which provides a data path to and from a memory bank of adjacent calculation units. The calculation unit according to any one of claims 2 to 7 , wherein the second memory bank is configured to store at least one of a partial sum or one or more pooling layer inputs. .. A computer-implemented method for accelerating neural network tensor calculations,
The direct memory access traversal unit accesses at least one memory location in the first memory bank to obtain at least one input activation based on an instruction received from a source outside the compute unit.
To give a control signal to write parameters to the second memory bank,
Based on the instruction received, accessing at least one memory location in the second memory bank to obtain one or more parameters.
The first memory bank having the first data width comprises sending a first input activation by the first memory bank in response to receiving a control signal from the tensor traversal unit. One memory bank is located within the compute unit, the first input activation is provided by a data bus accessible by at least one cell of the compute unit, and the method further comprises.
The at least one cell comprises receiving one or more parameters from a second memory bank having a second data width greater than the first data width, wherein the at least one cell is at least one. The product-sum (MAC) operator is included, and the above method further includes
The MAC operator comprises performing one or more calculations related to at least one element of the data array, the one or more calculations being at least partially accessed from the data bus. Includes a multiplication operation between the first input activation and at least one parameter received from the second memory bank.
The second memory bank is performed by a computer to accelerate the tensor calculation of the neural network, which is configured to store at least one of the partial sum or one or more pooling layer inputs. How to do it. A computer-implemented method for accelerating neural network tensor calculations,
The direct memory access traversal unit accesses at least one memory location in the first memory bank to obtain at least one input activation based on an instruction received from a source outside the compute unit.
To give a control signal to write parameters to the second memory bank,
Based on the instruction received, accessing at least one memory location in the second memory bank to obtain one or more parameters.
The first memory bank having the first data width comprises sending a first input activation by the first memory bank in response to receiving a control signal from the tensor traversal unit. One memory bank is located within the compute unit, the first input activation is provided by a data bus accessible by at least one cell of the compute unit, and the method further comprises.
The at least one cell comprises receiving one or more parameters from a second memory bank having a second data width greater than the first data width, wherein the at least one cell is at least one. The product-sum (MAC) operator is included, and the above method further includes
The MAC operator comprises performing one or more calculations related to at least one element of the data array, the one or more calculations being at least partially accessed from the data bus. Includes a multiplication operation between the first input activation and at least one parameter received from the second memory bank.
A ring bus extending outside the compute unit provides a data path between the second memory bank and the second memory bank of another adjacent compute unit.
The payload data and the bitmap header are transferred through the ring bus, the payload data includes the parameters, and the bitmap header indicates a calculation tile in which the payload data should be used.
When the calculation tile receives the payload data and the bitmap header, the calculation tile clears the bit set data specific to the calculation tile in the bitmap header, and the payload data and the bitmap header. A computer-implemented method for accelerating tensor calculations in a neural network that sends and to another calculation tile. The one or more computations are performed based in part on the calculation unit performing a loop nest containing the plurality of loops, and the structure of the loop nest is one or more dimensions of the data array. The method performed by the computer according to claim 9 or 10 , comprising each loop used by the tensor traversal unit to traverse. 11. The method performed by a computer according to claim 11 , further comprising providing the tensor traversal unit with a tensor operation comprising a loop nesting structure for accessing one or more elements of the data array. Performed by the computer of any one of claims 9-12 , further comprising performing the first part of the one or more calculations by generating a product-sum value based on the multiplication operation. How to be done. 13. The computer according to claim 13 , further comprising applying a non-linear function to the sum of products value to perform a second portion of the one or more calculations by obtaining at least one output activation. How it is carried out. A program executed by a computer, wherein the program causes the computer to execute the method according to any one of claims 9 to 14 .

Images