JP7495480B2 - Vector Reduction Using Shared Scratch-Pad Memory. - Google Patents

Description

BACKGROUND This specification relates generally to circuit configurations of hardware circuits used to perform neural network computations.

A neural network is a machine learning model that utilizes one or more layers of nodes to generate an output (e.g., a classification) for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to one or more other layers in the network (e.g., other hidden layers or the output layer of the network). Some of the layers of the network generate outputs from the received input according to the current values of a respective set of parameters. Some neural networks may be convolutional neural networks (CNNs) configured for image processing or recurrent neural networks (RNNs) configured for speech and language processing. Different types of neural network architectures can be used to perform a variety of tasks related to classification or pattern recognition, prediction including data modeling, and information clustering.

A neural network layer of a CNN may have an associated set of kernels, which may correspond to parameters or weights. The associated set of kernels is used to process an input (e.g., a batch of inputs) through the neural network layer to generate a corresponding output of the layer for computing a neural network inference. The batch of inputs and the set of kernels may be represented as a tensor (i.e., a multidimensional array) of inputs and weights. A hardware circuit that executes a neural network includes a memory having locations identified by address values. The memory locations may correspond to elements of the tensor, and the tensor elements may be traversed or accessed using the control logic of the circuit. For example, the control logic may determine or calculate a memory address value of an element to load or store a corresponding data value of the element.

SUMMARY This document describes techniques for performing data storage and vector reduction in a large shared scratch pad memory. In particular, these techniques are used to reduce the overall amount of operations required to perform vector reduction, including reducing values or outputs generated as a result of computations performed in each processor core of a computing system. For example, the system includes a hardware circuit that may have multiple processor cores and an architecture incorporating static random access memory (SRAM) memory resources. The SRAM memory resources are allocated to be shared among multiple respective processor cores of the circuit.

The sets of computations performed in the computing system can be distributed to respective cores of one or more hardware circuits to generate respective vectors of values. The shared memory receives each of the vectors of values from a resource of each of the processor cores using a direct memory access (DMA) data path of the shared memory. The shared memory performs an accumulation operation on each of the vectors of values using an arithmetic unit coupled to the shared memory. The arithmetic unit is configured to accumulate values based on arithmetic operations encoded in the arithmetic unit. A result vector is generated based on the accumulation operation.

One aspect of the subject matter described herein may be embodied in a method performed using a hardware circuit having a shared memory and a plurality of processor cores in communication with the shared memory. The method includes generating a first vector of values based on a calculation performed at a first processor core, the shared memory receiving the first vector of values from the first processor core using a direct memory access (DMA) data path of the shared memory, and performing an accumulation operation in the shared memory between the first vector of values and a vector stored in the shared memory. The accumulation operation is performed using a calculator unit that is i) coupled to the shared memory and ii) configured to accumulate a plurality of vectors. The method includes generating a result vector based on the accumulation operation.

Each of these and other implementations may optionally include one or more of the following features. For example, in some implementations, the vector stored in the shared memory was received from a second processor core, and the method includes the first processor core performing an accumulate-to-memory operation to accumulate each value of the first vector of values to memory locations of the shared memory, and the second processor core performing an accumulate-to-memory operation to accumulate each value of a second vector of values to memory locations of the shared memory, the second vector of values corresponding to the vector stored in the shared memory. In some cases, the second processor sets a flag (e.g., an initial vector/value flag) such that the values of the second vector of values are written as initial values, while the first processor core sets a different flag (e.g., an accumulation flag) such that the values of the first vector of values are accumulated with the values of the second vector of values.

In some implementations, generating the result vector based on the accumulation operation includes generating the result vector without the first processor core performing a step of pre-accumulating products resulting from calculations performed by the first processor core, and generating the result vector without the second processor core performing a step of pre-accumulating products resulting from calculations performed by the second processor core.

In some implementations, generating the result vector includes generating a vector of accumulated values as a result of performing the accumulation operation on a first vector of values, applying an activation function to each value in the vector of accumulated values, and generating the result vector as a result of applying the activation function to each value in the vector of accumulated values. The accumulation may be performed on values of the first vector, or accumulation may include pair-wise accumulation of values of the first vector and values of the vector stored in the shared memory.

In some implementations, each resource of the first processor core is a first matrix computation unit, and the method further includes generating a first vector of accumulated values corresponding to the first vector of values based on a matrix multiplication performed using the first matrix computation unit of the first processor core.

In some implementations, the respective resource of the second processor core is a second matrix computation unit, and the method further includes generating a second vector of accumulated values corresponding to the second vector of values based on a matrix multiplication performed with the second matrix computation unit of the second processor core. The hardware circuit may be a hardware accelerator configured to execute a neural network having multiple neural network layers, and the method includes generating an output of a layer of the neural network based on the result vector.

The method may further include generating a first vector of values based on calculations performed on the first processor core and generating a second vector of values based on calculations performed on the second processor core. The calculations performed on the first processor core and the calculations performed on the second processor core may be part of a mathematical operation controlled by commutativity. In some implementations, the mathematical operation is a floating point multiplication operation, a floating point addition operation, an integer addition operation, or a min-max operation. In some implementations, the mathematical operation includes a floating point addition operation and an integer addition operation. The first processor core and the second processor core may be the same processor core.

In some implementations, the shared memory is configured to function as a shared global memory space including memory banks and registers shared between two or more processor cores of the hardware circuit.

Other implementations of this and other aspects include corresponding systems, devices, and computer programs encoded on computer storage devices and configured to perform the actions of the methods. One or more computer systems may be so configured by software, firmware, hardware, or combinations thereof installed on the systems that, when operated, cause the systems to perform the actions. One or more computer programs may be so configured by having instructions that, when executed by a data processing device, cause the devices to perform the actions.

The subject matter described herein can be implemented in particular embodiments to achieve one or more of the following advantages:

The techniques described herein take advantage of the ability of large shared scratch pad memories that support a DMA mode to atomically shrink incoming vector data to a shared memory location, rather than simply overwriting this data. In other words, multiple processor cores or processors can simultaneously perform shrink operations that update the same shared memory location, so that the resulting shrink values are calculated as if the operations occurred consecutively, even if the shrink operations involve values associated with the same memory location. If instead each processor simply overwrites the shared memory location, a previous value written by another processor may be unintentionally lost (e.g., corresponding to a lost update problem). Based on the control loop, the system can detect an "atomic" shrink of the data and cause the memory location to hold (or store) the final result of the shrink operation by allowing the overwriting of the value in the shared memory location in either way. In some cases, the various techniques described herein are extendable to other memory types (including on-chip and off-chip memory) present throughout the system.

The arithmetic unit is coupled proximate to the shared memory to support various arithmetic operations for accumulating vector values into the shared memory cells/locations. The arithmetic operations may be based on any reduction operator (such as floating point atomic addition, integer addition, maximum, minimum, max pooling, and even multiplication). The arithmetic unit coupled proximate to the shared memory provides the advantage of integrating software-managed addressing of shared resources and commutative mathematical operations into a single memory system.

These techniques include a read-modify-write control loop running in a shared memory control unit to keep track of outstanding operations to ensure atomicity, and to stop or reorder write traffic as necessary to prevent vector values from accumulating against old vector values. The read-modify-write control loop also provides performance and energy improvements over alternative, inefficient approaches that require reading vector data stored in a first processor core, performing arithmetic operations on the read vector values in a computation unit remote from the first core, and then storing/writing back to the first processor core. When a system has a large vector memory, these alternative, inefficient approaches may require moving data significant distances across the chip. Such approaches unnecessarily consume computation cycles in the processor core and wiring bandwidth to and from the core. These inefficiencies also generate deeper computation schedules and unnecessarily consume register bandwidth.

These techniques include an accumulation-to-memory feature based in part on an accumulation flag generated in a processor core and used in conjunction with a shared memory DMA path. This feature allows two or more processor cores to accumulate vectors directly into a shared memory location in the shared memory system. This feature can be particularly useful in multi-node systems by allowing DMA from multiple cores to target the same memory sectors and addresses simultaneously without requiring external synchronization or software locks to arbitrate operations between the cores. For example, this can be useful for configuring shared memory cells as a fully-reduced buffer across multiple chips, or as a distributed system of processor cores.

Some implementations relate to methods, systems, and apparatus, including computer-readable media, for performing vector reduction using a shared scratchpad memory of a hardware circuit, the hardware circuit having processor cores in communication with the shared memory. For each processor core, a respective vector of values is generated based on computations performed in the processor core. The shared memory receives each of the vectors of values from a resource of each of the processor cores using a direct memory access (DMA) data path of the shared memory. The shared memory performs an accumulation operation on each of the vectors of values using an arithmetic unit coupled to the shared memory. The arithmetic unit is configured to accumulate values based on arithmetic operations encoded in the arithmetic unit. A result vector is generated based on the accumulation operation.

Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims.

1 is a block diagram of a computing system having hardware circuitry including an exemplary shared memory. FIG. 2 is a block diagram illustrating an example of a processor core communicating with a shared memory in a hardware circuit. FIG. 2 is a block diagram illustrating an example of a vector processor in communication with a matrix computation unit of a hardware circuit. FIG. 2 is a block diagram illustrating an example accumulation pipeline. FIG. 2 is a diagram illustrating an example of an input tensor, a weight tensor, and an output tensor. 2 is a flow diagram illustrating an example process for performing vector reduction using the shared memory of FIG. 1.

DETAILED DESCRIPTION Like reference numbers and designations in the various drawings indicate like elements.

Reduction operations are often used in computations that utilize linear algebra, such as computationally intensive workloads for operations involving artificial neural networks. For example, a reduction operation may be necessary to average gradient values computed across different processing nodes of a distributed system when training a neural network. Reduction operations can be performed distributed, such as in the case of a full reduction operation, or locally for a given computation (such as a matrix multiplication tile sum operation).

Performance and power concerns can be important factors for efficiently building and executing these operations in a computing system. Typically, a reduction operation requires pulling data through the memory hierarchy of the system (e.g., a distributed system) to the arithmetic logic unit (ALU) of a processor (or processor core), performing the calculation/reduction on the pulled data, and then writing the result back through the memory system. However, performing these various steps in a system comes at a high cost in both performance and power. In addition, performing reduction operations across cores in memory visible to multiple processor cores typically requires synchronization and/or resource reservation in non-overlapping memory regions, which can result in significant performance and capacity overhead, as well as increased programming complexity.

Based on the foregoing context, data processing techniques are described herein for performing vector reduction by accumulating vectors of values in one or more memory address locations in a large shared scratch pad memory. The vector reduction and accumulation are performed in the shared scratch pad memory based on software managed addressing of memory locations used to write (store) results of computations, rather than based on addressing schemes typically used in hardware managed cache memory systems. The shared memory includes resources such as memory cells that are shared across a distributed system of processor cores. The described techniques include accumulation into memory for performing an accumulative reduction step (e.g., for vector reduction) as the vectors of values are processed. For example, the accumulative reduction step may be performed across matrix multiplications performed on different sets of inputs processed through a layer of a neural network to generate the output of the layer.

The data processing techniques including the shared scratchpad memory are implemented using an improved hardware circuit architecture compared to previous designs. The hardware circuit may be a special-purpose processor (such as a neural network processor, an application specific integrated circuit (ASIC), or a hardware accelerator). The hardware circuit is configured to execute a neural network including multiple neural network layers. The improved architecture and data processing techniques described herein enable the circuit representing the hardware accelerator to achieve increased speed and bandwidth to further accelerate computations.

The computation may be a particular type of mathematical operation (such as a floating-point multiplication, floating-point addition, or integer addition operation). The computation may also be included among the operations of an example neural network model that are performed to compute an inference or to train the model. In some examples, the computation is used to process inputs through layers of a CNN or RNN to generate outputs corresponding to neural network inferences, or to calculate gradients with respect to the parameters of a CNN or RNN to update the parameters of the neural network during training.

The mathematical operations are controlled by commutativity and may include atomic reductions (e.g., atomic floating-point reductions). An atomic reduction is treated as an accumulation or vector reduction step in which a vector is accumulated directly into a memory location of a shared memory (or into a vector stored therein) without the need to synchronize activity among the cores providing the vectors of values requiring accumulation. In other words, two or more cores of the hardware circuit may accumulate values into a central address location of a shared memory cell in any order such that the final result vector provides the correct mathematical result of the accumulation. In one example, the accumulation involves a pair-wise accumulation of values of a first vector provided by a first core and values of a second vector provided by a second core.

1 is a block diagram of a computing system 100 including an exemplary hardware circuit 101. As mentioned above, the hardware circuit 101 may represent a hardware accelerator or some other dedicated processor. In some cases, the system 100 is an exemplary computing system for accelerating tensor or neural network calculations associated with an artificial deep neural network (DNN) (such as an RNN or CNN). For example, the system 100 is configured to run the CNN on an exemplary hardware accelerator and pass data values to the hardware accelerator to generate output for computing inference. In some implementations, the system 100 is a system-on-chip. For example, the system-on-chip may include the hardware circuit 101 and some (or all) of the other components and devices described in this document as being included in the system 100.

The hardware circuit 101 may be a hardware accelerator configured to accelerate the execution and/or performance of a neural network model. For example, the execution of the neural network model may be accelerated as compared to the execution of the model on an exemplary general purpose machine (such as a central processing unit (CPU)). Similarly, the performance and execution of the neural network model may be accelerated as compared to the execution of the model on another hardware accelerator (such as a graphics processing unit (GPU)) that does not have the improved hardware features and techniques described herein.

The system 100, including the circuit 101, includes a system memory 102 and a shared memory 104. The system memory 102 may represent a high bandwidth memory ("HBM 102") or an input/output (I/O) device that communicates data to and from the processor cores 105-1, 105-2 of the hardware circuit 101. The data communication may generally include writing data values to or reading data from the vector memories 106, 108 located at a particular processor core 105-1, 105-2. For example, the HBM 102 may communicate data to and from the processor core 105-1 to provide input to the core and receive output generated by one or more computational resources of the core.

The data values may represent arrays of vector elements or vector values. For example, a first vector array may represent a batch of inputs to be processed through a neural network layer, while a second vector array may represent a set of weights for that layer. Relatedly, a third vector array may represent a vector of accumulation values corresponding to outputs generated by processor core 105-1, while a fourth vector array may represent a vector of activation values representing outputs generated by processor core 105-2.

HBM 102 may be a dynamic random access memory (DRAM) asset of system 100. In some implementations, HBM 102 is external or off-chip memory to circuit 101 and is configured to communicate data to and from an on-chip vector memory bank (described below) of system 100. For example, HBM 102 may be located at a physical location that is external to the integrated circuit die representing circuit 101. Thus, HBM 102 may be separate from or non-local to computational resources located within the integrated circuit die. Alternatively, HBM 102, or portions of the resources, may be located within the integrated circuit die representing circuit 101 such that HBM 102 is local to or co-located with the computational resources of the circuit.

System 100 may include one or more processor cores 105-1, 105-2. In some implementations, system 100 includes multiple processor cores 105-n, where n is an integer greater than or equal to 1. In the examples of FIG. 1 and FIGS. 2 and 3 described below, system 100 is shown as including two processor cores, but system 100 including hardware circuitry 101 described herein may have more or fewer processor cores. Generally, processor core 105-n is a separate, self-contained processing/computation unit of system 100 (or hardware circuitry 101).

Each processor core 105 is configured to independently perform computations (e.g., neural network computations) required by one or more layers of a multi-layer neural network. The computations may be necessary to process data of a machine learning workload or to perform a particular task of the workload. The computations performed in the processor cores to process inputs through one or more neural network layers may include multiplication of a first set of data values (e.g., inputs or activations) with a second set of data values (e.g., weights). For example, the computations may include multiplying input or activation values with weight values in one or more cycles and performing an accumulation of products over multiple cycles.

The different values in the first and second sets of data values are stored in specific memory locations of a memory construct in a processor core of the hardware circuit 101. In some implementations, the individual values in the first set of data values may correspond to respective elements of an input tensor, while the individual values in the second set of data values may correspond to respective elements of a weight (or parameter) tensor. As an example, a neural network layer in a series of layers may process a set of inputs (such as inputs of image pixel data) or a set of activation values generated by another neural network layer in the series of layers.

The set of inputs or activation values can be represented as a one-dimensional (1D) or multidimensional tensor (e.g., 2D or 3D) having multiple elements along each dimension. Each of the memory locations that store data values can be mapped to a corresponding element of the one-dimensional or multidimensional tensor, and the tensor elements can be traversed or accessed using the control logic of the circuit. For example, the control logic can determine or calculate a memory address value that maps to an element to load or store the corresponding data value of the element.

The hardware circuit 101 has a dedicated memory hierarchy that includes different memory constructs. Each of these memory constructs may have different bandwidth and latency characteristics compared to the other constructs, and may also have different physical locations within the hardware circuit 101. Exemplary memory constructs include a shared memory 104, vector memories 106, 108, and vector registers 110, 112. In general, the memory constructs are operable to store data values (such as inputs, vector values associated with activations, or gain values) to be processed by a neural network layer, and output activations generated by the neural network layer in response to processing the inputs or activations through the layer. The generation and storage of output activations, as well as the various memory constructs used to perform these operations, are described in more detail below with reference to Figures 2 and 3.

FIG. 2 is a block diagram 200 illustrating an example of how resources or sections of the shared memory 104 may be arranged in the hardware circuit 101 to facilitate data communication between various components of the hardware circuit. As described above, the shared memory 104 provides a basis for improving the hardware architecture and data processing techniques of the system 100. The shared memory 104 is a larger on-chip SRAM compared to the on-chip memory of some other neural network processor chips. In some implementations, the shared memory 104 may be described as being between the HBM 102 and the vector memories 106, 108 of the corresponding processor cores 105-1, 105-2 (e.g., logically or physically). For example, the act of using the shared memory 104 to move data between the HBM 102 and the vector memories 106, 108 may involve the data traversing the shared resources of the shared memory 104.

The shared memory 104 may represent a shared central space on the chip or circuit 101. For example, the shared memory 104 may be configured to function as a shared global memory space, which includes memory resources corresponding to memory banks and registers shared among one or more of the processor cores 105-1, 105-2 of the multiple processor cores that may be present in the system 100 and/or included in the hardware circuit 101. As described in more detail below, the shared memory 104 may be configured to function as a software-controlled scratch pad (e.g., similar to an exemplary vector). In some implementations, some (or all) of the resources of the shared memory 104 may be configured to function as a software-controlled scratch pad (staging resource) rather than a hardware-managed cache.

The system 100 is configured to expose at least two programming interfaces to a user to take advantage of the data transfer functions provided by the shared memory 104. A first interface exposes programmable DMA data transfer functions and operations, while a second, different interface exposes programmable load/store data transfer functions and operations. Each of these interface functions may represent a logical attribute of the shared memory 104, which are described in more detail below.

As discussed above, the memory constructs of the system 100 have various bandwidth and latency characteristics. For example, the shared memory 104 may have a higher bandwidth and lower latency than the DRAM accesses of the HBM 102, but a lower bandwidth and higher latency than the accesses to the vector memories 106, 108. In some examples, the shared memory 104 has a lower data capacity than the DRAM assets of the HBM 102, but a higher data capacity than the respective vector memories of the processor cores. In general, these various bandwidth and latency characteristics represent standard memory hierarchy trade-offs.

The memory constructs of system 100, and in particular shared memory 104, may also vary in their physical arrangement within hardware circuitry 101. Shared memory 104 includes resources such as memory banks and registers that may be physically and logically arranged with respect to the arrangement of the particular computational resources of processor cores 105-1, 105-2. In this context, shared memory 104 may be generally characterized with reference to its physical structure and its logical structure. The physical structure of shared memory 104 is described first, followed by its logical structure.

In terms of its physical structure, the shared memory 104 resources may be physically distributed on dedicated or neural net processor chips corresponding to the hardware circuit 101. For example, different subsets, portions, or sections of the resources forming the shared memory 104 may be physically distributed at various locations of the circuit 101 so that different types of data transfer operations and processing techniques can be performed in the system 100. In some implementations, one section of the shared memory 104 resources may be internal to the processor core of the circuit 101, while another section of the resources may be external to the processor core of the circuit 101. In the example of FIG. 2, a section of the shared memory 104 is external to each of the processor cores 105-1, 105-2 to enable DMA operations to move large blocks of data between memory locations of the HBM 102 and memory locations of the shared memory 104.

Returning briefly to the HBM 102, this type of system memory may be an external memory structure used by the system 100 to provide high bandwidth data to and/or from the vector memory of each processor core. In some implementations, the HBM 102 is configured for various direct memory access (DMA) operations to retrieve data from or provide data to memory address locations of the vector memory within the processor core of the circuit 101. More specifically, DMA operations involving the HBM 102 transferring data to and from the vector memories 106, 108 are enabled by an exemplary control scheme and memory resources of the shared memory 104.

In the example of FIG. 2 and FIG. 3 (described below), the shared memory 104 includes a shared memory control unit 201 ("control unit 201"). The control unit 201 is configured to generate control signals 114 for controlling memory access operations (including the HBM 102, the shared memory 104, the vector memories 106, 108, and each of the vector registers 110, 112).

The control unit 201 executes a control scheme that is distributed to the different memories of the system 100 (e.g., the HBM 102, the shared memory 104, the vector memories 106, 108, and the vector registers 110, 112). In some implementations, this control scheme is distributed to the different memories based on communication between the control unit 201 and the respective control units of each memory. For example, the control scheme can be distributed to the memories based on control signals provided by the control unit 201 that are processed locally by the respective control units of these different memories. Data path sharing can be used to move data between the HBM 102 and the respective vector memories of the processor cores 105-1, 105-2. Once this is done, the system 100 will activate any (and all) necessary control units for a given memory or data path to manage the data handoff that needs to take place at the appropriate touchpoint.

The control unit 201 is configured to execute software instructions and generate control signals that cause a first portion of the memory resources of the shared memory 104 to function as a DMA memory unit. The first portion of the resources can be represented by a shared core data path 204 referenced to the processor core 105-1 and a shared core data path 224 referenced to the processor core 105-2. This representative DMA memory unit is operable to move data between the HBM 102 and each of the first and second processor cores 105-1 and 105-2 based on the control signals generated by the control unit 201.

For example, control signals may be generated to perform DMA operations to move blocks of data (e.g., vectors) between memory locations in shared memory 104 and vector memory 106 using data path 202, shared core data path 204, or data path 206, and between memory locations in shared memory 104 and vector memory 108 using data path 222, shared core data path 224, or data path 226. In some implementations, shared memory 104 may alternatively be referred to as shared CMEM 104.

In this document, CMEM generally corresponds to a block of physically contiguous memory (CMEM) that provides a configuration useful as a data buffer and on-chip SRAM storage. As described in more detail below, in system 100, blocks of CMEM resources are physically distributed in hardware circuit 101 and arranged to be shared among components of a processor core, which may be configured as hardware accelerators or other types of dedicated processors. Each of shared core data paths 204 and 224 are exemplary nodes that may exhibit static contention that may occur on the shared data path due to the movement of vector data across these points in the system.

As shown in the example of FIG. 2, the hardware circuit 101 and system 100 shown in FIG. 1 are configured to include multiple load/store data paths 202, 206, multiple CMEM load data paths 208, 214, 228, 234, and multiple CMEM store data paths 215, 235. The hardware circuit 101 and system 100 also include multiple shared staging blocks 210, 230 (described below). In the example of FIG. 2, each of the data paths 202, 222 can be configured as a data path for routing data (e.g., vector or scalar values) in response to performing a DMA operation, as a data path for routing data in response to performing a CMEM load/store operation, or both. The DMA operations and data paths 202, 206, 222, and 226 supported by the shared memory 104 can be used to move data between different memory structures with reference to specific memory offsets and stride parameters.

For example, the system 100 may be configured to perform a DMA operation with the shared memory 104, the DMA operation including moving or copying 1 megabyte of data from one set of memory locations to another set of memory locations at an offset of 0x04. The shared memory 104 and the system 100 are operable to support various stride functions when performing the DMA operation. For example, the DMA operation to move 1 megabyte of data may include a stride operation that inserts an address interval every 200 kilobytes relative to an address base or address value of a base memory location.

In some implementations, the stride operation is used to insert address intervals based on a desired read sequence that is later executed to read the 1 megabyte of data after it has been moved to a destination location. For example, a 1 megabyte block of data may be stored based on a stride operation that corresponds to how the data is read or retrieved and processed in different layers of a neural network, or across different sets of filters or weights in a particular neural network layer.

The control unit 201 of the shared memory 104 is also configured to cause various load and store operations to be performed. For example, the control unit 201 generates control signals to perform load and store operations that move various amounts of data (e.g., vectors or vector values) a) between memory locations of the shared memory 104 and memory locations of the shared staging block 210 using data path 202, shared core data path 204, or data path 208 (in the case of a load operation on core 105-1), and b) between memory locations of the shared memory 104 and memory locations of the shared staging block 230 using data path 222, shared core data path 224, or data path 228 (in the case of a load operation on core 105-2).

Similarly, control signals may be generated to perform load and store operations that move various amounts of data (e.g., vectors or vector values) a) between memory locations in shared memory 104 and vector register 110 using data path 202, shared core data path 204, or data path 215 (for store operations in core 105-1), and b) between memory locations in shared memory 104 and vector register 112 using data path 222, shared core data path 224, or data path 235 (for store operations in core 105-2).

Turning now to the logical structure of the shared memory 104, as described above, the system 100 is configured to expose at least two programming interfaces to a user to take advantage of the data transfer capabilities provided by the shared memory 104. At least one interface exposes programmable DMA capabilities and another interface exposes programmable CMEM load/store capabilities, each of which may represent logical attributes of the shared memory 104. For load/store purposes, the shared memory 104 is logically exposed as a parallel memory to the vector memories 106, 108. In this manner, each load/store data path is operable to provide an additional (or parallel) data path for moving a block of data or specific data through the memory system (such as through the vector registers of the respective processor cores 105-1, 105-2, or through multiple cores of the circuit 101). For example, load/store operations may be performed on the memory resources of the shared memory 104 simultaneously with DMA operations.

More specifically, DMA operations may be performed to move vectors of values between memory locations in shared memory 104 and vector memory 106 using DMA data path 206, and load and store operations may be performed simultaneously with the DMA operations to move different vectors of values between memory locations in shared memory 104 and shared staging block 210. Similar concurrent operations may be performed in processor core 105-2 (or other cores) using resources of processor core 105-2 that correspond to the resources of processor core 105-1.

Load/store operations performed using the CMEM resources of the shared memory 104 may represent a high performance feature of the shared memory 104, or a high performance method of using the shared memory 104, as compared to DMA operations. In some implementations, the control unit 201 is configured to execute software instructions and generate control signals that cause a second portion of the memory resources of the shared memory 104 to function as a software controlled staging resource used to perform the load/store operations.

The second portion of resources may be represented by a shared staging block 210 referenced to processor core 105-1 and a shared staging block 230 referenced to processor core 105-2. Each of shared staging blocks 210, 230 may thus represent a software-controlled staging resource (or scratch pad) formed from a subset of memory resources of shared memory 104. In some examples, the software-controlled staging resources of system 100 are configured to manage the flow of vector data values from HBM 102 to vector registers 110 or 112 of first processor core 105-1 or second processor core 105-2, respectively.

The shared memory 104 and its resources have the property that they are uniquely configurable, for example, not only as a DMA memory for moving data between memory constructs (such as the HBM 102 or vector memories 106, 108), but also as a load/store memory for moving data directly into the respective vector registers 110, 112 on each processor core 105-1, 105-2. These configurable aspects of the shared memory 104 allow its resources and addressing to be scheduled at a fine granularity by software running on the cores. For example, the shared memory 104 can be a software-managed (not hardware-managed) SRAM resource, in which the processor core's compiler specifically manages the addressing of that memory (including types of data that may or may not reside at memory address locations of the shared memory 104).

In some implementations, the software-controlled staging resources of the shared memory 104 are configured as a first-in-first-out (FIFO) memory structure (e.g., shared staging block 210 or 230) along the load portion of the load-store data path of the processor core, including a CMEM store data path 215 or 235 for routing data to be stored in the shared CMEM 203 or HBM 102. The FIFO memory structure is configured to temporarily store a set of data values for a threshold number of processor cycles before routing the set of values to the vector registers 110, 112 of the first processor core 105-1 or second processor core 105-2, respectively. The FIFO memory structure is used to mitigate register pressure and scheduling complications that may be caused by CMEM load operations having a particular load latency.

In some implementations, the threshold number of clock cycles is determined based on an exemplary high latency (e.g., 50 cycles) CMEM load operation that may cause register pressure and scheduling complexities associated with reserving a given register for the entire 50 cycles. To alleviate or mitigate concerns regarding register pressure, a CMEM result FIFO ("CRF") is physically instantiated in hardware circuit 100 using resources of shared memory 104. In the example of FIG. 2, a first CRF is represented by staging block 210 of processor core 105-1, while a second CRF is represented by staging block 230. Each of the CRFs may divide an exemplary CMEM load operation into at least two phases: i) a CMEM → CRF phase (where CMEM address information is provided), and ii) a CRF → register phase (where a vector register target is provided).

For example, each of the shared staging blocks 210, 230 is configured to receive a data value (e.g., a scalar or vector value) and temporarily store the data value for a threshold number of processor cycles. In the processor core 105-1, the data value is routed to the shared staging block 210 along the load data path 208 (and the shared core data path 204) that connects the staging block 210 to other memory locations in the shared memory 104. In the processor core 105-2, the data value is routed to the shared staging block 230 along the load data path 228 (and the shared core data path 224) that connects the staging block 230 to other memory locations in the shared memory 104.

The shared staging block 210 is configured to provide a data value to the vector register 110 of the processor core 105-1 in response to temporarily storing the data value for the threshold number of processor cycles. Similarly, the shared staging block 230 is configured to provide a data value to the vector register 112 of the processor core 105-2 in response to temporarily storing the data value for the threshold number of processor cycles.

The system 100 is configured to issue multiple CMEM load instructions in the same cycle. For example, the system 100 can issue a CMEM load instruction executed using the data path 208 (or 214) and the shared staging block 210, and issue a load to the vector memory 106 executed using the data path 212 in the same cycle. In some examples, from a software control perspective, each of a Cmem load operation traversing the data path 214 between the resource 210 and the vector register 110, and a Vmem load operation traversing the data path 212 between the vector memory 106 and the vector register 110 can be issued and executed in the same cycle. In some implementations, the vector registers 110, 112 are adapted to include additional ports that allow the vector registers 110, 112 to receive concurrent load operations, as compared to previous designs.

For example, vector register 112 is configured to include additional ports that allow the register to receive respective vector payloads from vector memory 108 and shared staging block 230 during concurrent load operations executed by processor core 105-2. In some examples, a single datum of payload loaded into each of vector registers 110, 112 includes 128 separate loads based on up to 128 data items that may be moved to vector register 110 or vector register 112 during a single load operation.

The CMEM load/store functions of shared memory 104 can provide higher peak performance compared to previous designs because they do not require routing data through vector memory macros. For example, loads and stores (along data paths 215, 235) can be performed in parallel with vector memory loads and stores, such as by using available additional register ports in vector registers 110, 112.

In some implementations, the system 100 includes an exemplary load-store interface that provides a parallel interface to each of the shared staging blocks 210, 230 that bypasses some (or all) of the bandwidth limitations that may exist when traversing a data path through the vector memories 106, 108. This exemplary load-store interface can effectively provide higher memory bandwidth that allows additional performance to be extracted from the exemplary workload. For example, the system 100 can be configured to perform various load/store operations using resources (e.g., software-controlled staging resources) of the shared memory 104, and the load/store operations can be performed to bypass moving data through the vector memory in the processor core.

For example, a component of hardware circuit 101 may communicate with shared memory 104 to read data from a single address location of a memory bank or register file of shared memory 104. In some examples, data stored at a single address in memory is read and the single datum may be moved to a register file or staging block located internal to the processor core. For example, the single datum may be read from an address location of shared CMEM 104, moved through shared core data path 224, and moved to an address location of shared staging block 230 in processor core 105-2 for further processing. This operation may be performed to save processor clock cycles in core 105-2 and bandwidth in the data path connecting to vector memory 108 by bypassing moving the data through the memory system via vector memory 108.

Figure 3 is a block diagram 300 illustrating an example of a vector processor in communication with a matrix computation unit of hardware circuit 101. More specifically, in some implementations, a tensor processor core 302-1 of hardware circuit 101 includes a vector processing unit 304 ("vector processor 304") and a matrix computation unit 308 coupled to vector processor 304. Similarly, another tensor processor core 302-2 of hardware circuit 101 includes a vector processor 306 and a matrix computation unit 310 coupled to vector processor 306. Thus, matrix computation units 308, 310 are resources of each of processor cores 302-1, 302-2.

In general, the hardware circuit 101 is configured to perform calculations to generate the outputs of the neural network layers. Each of the matrix computation units 308 and 310 included in the circuit 101 is configured to perform a subset of the calculations to generate the accumulation values used to generate the outputs of the neural network layers. In some implementations, the software-controlled staging resources described above (e.g., staging blocks 210, 230) are configured to manage the flow of data (e.g., vector operands) from the HBM 102 shown in FIG. 1 to each of the matrix computation units 308, 310. In some cases, the operands are inputs and weights provided by the HBM 102. The operands may be structured as vector arrays based on data operations performed using an arithmetic logic unit (ALU) of the vector processor 304 or 306.

In the example of FIG. 3, the control unit 201 generates control signals to manage the retrieval (or reading) of batches of inputs and sets of weights from memory locations in the shared memory 104, the vector memories 106, 108, and the vector registers 110, 112. The retrieved inputs and weights may be processed through neural network layers to generate accumulated values based on calculations performed in the matrix computation units 308, 310. The accumulated values may be processed in the vector processors 304, 306 to generate activation values corresponding to the outputs of the neural network layers.

Control signals generated by the control unit 201 are used to store (or write) multiple sets of outputs or output activations generated by the vector processors 304, 306 in other memory locations of the HBM 102 or hardware circuit 101 for processing in one or more other neural network layers. More specifically, the system 100 is configured to execute data processing techniques for performing vector reduction, which includes accumulating vectors of values in one or more memory address locations in a large shared scratch-pad memory (such as the shared memory 104). As described above, vector reduction and accumulation can be performed in the shared scratch-pad memory 104 based on software-managed addressing of locations in memory cells of the shared memory 104. Address locations in memory cells of the shared memory 104 can be used to write (store) results of computations performed in different components of the system 100.

The system 100 includes a calculator/accumulator unit 320 ("calculator 320") that is coupled (or may be coupled) to the shared memory 104. The calculator 320 is configured to accumulate values based on one or more arithmetic operations. The arithmetic operations may be programmed or encoded in the calculator 320 in software, firmware, hardware, or a combination of each. The calculator 320 may represent a dense portion of computational logic that is coupled near the memory cells of the shared memory 104 and performs accumulation operations on vector values being routed to the shared memory cells of the shared memory 104.

In some implementations, the operator 320 is a computation unit that includes hardware circuits for performing different types of adders (e.g., normalizing adders) and multipliers, each configured to perform different types of mathematical operations on values having different types of numeric formats. For example, the operator 320 is configured to perform mathematical operations (such as floating-point multiplication, floating-point addition, integer addition operations, and min-max operations). In some other implementations, the operator 320 is included in the system 100 as a hardware feature of the shared memory 104. One or more arithmetic operations or functions of the operator 320 may also be implemented in software and hardware.

The operator 320 may include logic 325 for selecting a particular arithmetic operation or for selecting circuitry in the operator 320 configured to perform a particular arithmetic operation. In some implementations, the operator 320 is instantiated in the shared memory 104 and/or the hardware circuitry 101 based on one or more numeric formats (e.g., two's complement integer and floating point) of the values in the vector of values. For example, the numeric format corresponds to a data format used to represent the numbers or values in the vector. In some implementations, the operator 320 includes circuitry for a normalization unit, a pooling unit, or both.

As mentioned above, the described techniques include an accumulate to memory function for performing an accumulated reduction step (e.g., for vector reduction) when processing a vector of values. In the example of FIG. 3, each of the processor cores 302-1, 302-2 may generate a respective accumulation flag 330, 335 to cause the control unit 201 of the shared memory 104 to perform an accumulate to memory function on an exemplary vector of values. The vector of values may be moved to the shared memory 104, for example, using a DMA operation that moves the vector to the shared memory 104 using data path 206 or data path 226.

FIG. 4 is a block diagram illustrating an exemplary accumulation pipeline 400 ("pipeline 400"). Pipeline 400 illustrates exemplary data processing steps for an exemplary operation of accumulating a vector of values in a shared memory cell 445 of shared memory 104.

Vector operands, such as individual input and weight values, may be represented as tensor values that are multiplied using multiplication cells of an exemplary matrix unit of the processor core and then stored in the core's vector memory (402). In some implementations, the inputs of the vector operands correspond to partitions of an input matrix or input tensor. For example, an input tensor may be split into two sections, and input values from different respective dimensions of each section may be sent to a particular processor core to be multiplied with weight values to generate output values. The input tensors, along with the weight tensors and output tensors, are described in more detail below with reference to FIG. 5.

The final result vector 450 may be based on a final set of output values representing the outputs calculated for the layer of the neural network using each of the input matrices/tensors of input. So, even if the data/input values of the input tensors may be split to be processed by different processor cores, generating the correct and accurate final result vector 450 actually depends on the correct and accurate accumulation of at least two different sets of output values generated by each core. For example, the different sets of output values generated by each core may need to be summed or accumulated to generate the correct final result vector 450.

In the example of FIG. 4, each processor core is shown as core_0 (e.g., processor core 302-1) and core_1 (e.g., processor core 302-2). Depending on the matrix multiplication performed by each matrix unit (e.g., matrix 308 or 310) of each processor core, multiple output values may be generated. In some implementations, the output values are stored in a vector memory of the processor core performing the matrix multiplication and then sent to the shared memory 104 for an accumulation operation. The final result vector 450 can be obtained based on both processor cores aggregating their respective halves of the calculations assigned to them. In some implementations, the aggregation to obtain the final result vector corresponds to a "pre-accumulation operation".

Previous approaches to accumulating vector values required one core to move its results to another core. These approaches required additional processor cycles, memory resources, computational bandwidth, and specific software control to move different sets of result values between different cores in the system. The accumulation reduction techniques herein allow these aggregations to occur in shared memory systems based on accumulation functions natively executable in the shared memory 104.

Each of the processor cores may generate a respective accumulation flag 330, 335 to cause the control unit 201 of the shared memory 104 to perform an accumulation to memory function on the exemplary vector of values (404). The vector of values generated by each processor core 105 may be moved to the shared memory 104 using DMA operations as described above. The technique for accumulating the vector of values into shared memory cells or address locations of the shared scratch pad memory 104 may be performed via the programmable DMA data transfer functionality of the system 100. For example, any DMA operation operable to move data into memory cells of the shared memory 104 may use the accumulation techniques described in this document. In this manner, each of the cores 0 and 1 in the examples of Figures 2 and 3 may both accumulate vector values into the same address locations of particular shared memory cells of the shared memory 104. In one example, the accumulation involves a pair-wise accumulation of values of a first vector provided by core 0 and corresponding values of a second vector provided by core 1.

In some implementations, the system 100 is configured to provide large vector "store accumulate" in a load/store usage mode of the shared memory 104, rather than using the DMA mode of the shared memory. For example, a shared load/store memory layer between multiple processor cores can be used to perform the "store accumulate" function, which isolates the need for some (or all) synchronization between the processor cores. In some implementations, the shared load/store memory layer between multiple processor cores used to perform the store accumulate function includes at least the data paths 212, 232 described above with reference to FIG. 2.

The shared memory 104 and the control unit 201 perform an accumulation operation on each of the vectors of values using the operator 320 (406). For example, the control unit 201 performs an accumulation reduction step across matrix multiplications performed on different sets of inputs processed through the layer of the neural network to generate the output of the layer. In some implementations, the vectors of values can be each of the vectors of accumulated values generated as a result of the matrix multiplications described above.

The control unit 201 is configured to mask one or more vector elements to enable or disable accumulation on particular vector elements, perform control to manage accumulation of different vectors, and keep track of outstanding accumulation operations (408).

With respect to mask elements, the system 100 may include machines (such as computational servers or associated hardware circuitry) that each include a 16B (16-bit) wide vector unit (e.g., a vector processor). The vector units may be configured to operate on 16-bit wide data elements, but the vectors of values generated by the hardware circuit's (or server's) resources may be only 9B wide vectors. In some implementations, the system 100 operates on one or more 9-element wide vectors, each of which includes nine data values that are 16 bits each. In this example, the control unit 201 may identify a data structure for the vector of values to be accumulated in a shared memory location of the shared memory 104. Based on this data structure, the control unit 201 may determine that the values to be accumulated in the shared location are 9B wide vectors for the 16B wide vector configuration of the vector units.

The control unit 201 may perform a mask function 430 that causes the arithmetic unit 320 to apply arithmetic operations to, for example, only the first 9 fields in a vector when performing accumulation or reduction. For example, a request from the processor core 302-1 to the shared memory 104 to accumulate a vector in the shared memory cell 445 may be presented as a 16B wide vector based on the configuration of the vector processing unit 304 of the processor core 302-1. The control unit 201 is configured to determine whether the value being accumulated is represented by the second half of the 16B wide vector or by a 9B wide vector represented by the first 9 fields of the 16B wide vector. Thus, the system 100 is operable to identify and select or otherwise control which particular elements in the vector are accumulated in the shared memory cell 445.

With respect to accumulation control, control unit 201 is configured to execute read-modify-write control 435 ("control 435") to control and manage the accumulation of distinct vectors of values in the shared memory system. Control 435 provides performance and energy improvements over inefficient alternative approaches that require reading data in a first core, performing computations on the read values in a computation unit separate from the first core, and then storing/writing back to the first core.

With respect to tracking of outstanding operations, the control unit 201 is configured to execute an operation tracker 440 to track outstanding requests and current (or pending) operations to accumulate different vectors of values in the shared memory system. For example, the control unit 201 uses the operation tracker 440 to track each write operation that requests to write a vector of values to a memory location (such as the shared memory cell 445) of the shared memory. In some implementations, the control unit 201 tracks the operations based on an accumulation flag 330, 335 that accompanies the write request from the processor core. The accumulation flag 330, 335 indicates that the vector of values should be written as an initial value or accumulated with an existing value at a particular memory location of the shared memory 104.

The control unit 201 sends control signals to the calculator 320 to cause the calculator 320 to perform an accumulation operation between the current value stored in a particular memory address location and the vector of values being written to that shared memory location. In some implementations, a request from a processor core to write a vector of values to a shared memory cell 445 requires at least two clock cycles to process. Because this write request may require at least two clock cycles to process, a read/write hazard may occur if the control unit 201 is attempting to read a value at a shared memory location while another vector is being written to the same shared memory location. In this case, the value being read is not the most current value because the write operation was not fully processed before performing the read of the value.

The control unit 201 uses the operation tracker 440 to determine which requests have been sent to the shared memory 104 in the last few clock cycles and determines whether the value stored at a particular memory location is stale or new. The control unit 201 can determine whether the value is stale or new based on a timestamp of the last write request or based on the time required to process the last write request (e.g., more than one clock cycle). For example, the timestamp can indicate that more than two clock cycles have passed since the last request was initiated or processed in the shared memory 104. If the value is determined to be new, the control unit 201 reads the value. If the value is determined to be stale, the control unit 201 stops reading the value until the number of clock cycles required to indicate that the value is new again for reading or accumulation has passed.

The system 100 is configured to receive a value (e.g., a vector) and accumulate it onto an existing value in a shared memory location (410) without losing any previous accumulations stored in the shared memory location 445. For example, the system 100 is configured to perform an accumulation operation without requiring external software locks to mitigate race conditions that may overwrite a vector accumulation previously stored in a memory location (e.g., the shared memory cell 445). The system 100 performs the accumulation operation without requiring a local pre-accumulation operation to be performed on each processor core and without requiring pre-synchronization between the processor cores. For example, a local pre-accumulation operation may be performed to accumulate each set of partial sums that are calculated locally on a given processor core.

The shared memory 104 is configured to natively support this functionality, which represents the atomic aspect of the vector reduction feature of the present technology. For example, multiple cores (e.g., 10 cores) of the system 100 may all be generating different vectors of values, and each core may submit a request to accumulate their respective vectors in a shared memory location. In some implementations, the request includes an accumulation flag 330, 335 and a corresponding core ID (e.g., core 0, core 1, core N, etc.), and the value to be accumulated in the memory location. In some implementations, a large matrix multiplication job may be split between at least two processor cores of the system 100, and this accumulation/vector reduction technique is used to simplify the accumulation of partial sums or dot products generated from the matrix multiplication.

In some implementations, these techniques for accumulating vectors of values in a shared memory cell are used during training of a neural network model. For example, these techniques can be used to perform a full reduction operation for gradient accumulation that reduces gradients calculated as part of a training step across a distributed system of processor cores. In particular, based on the disclosed accumulation reduction techniques, this gradient accumulation for training a neural network model can be performed natively in system 100 as a function of the memory system or shared memory 104.

5 illustrates an example of a tensor or multi-dimensional matrix 500 that includes an input tensor 504, a transformation of a weight tensor 506, and an output tensor 508. In FIG. 5, each of the tensors 500 includes elements that correspond to data values for a computation performed at a given layer of the neural network. The computation may include multiplying the input/activation tensor 504 with the parameter/weight tensor 506 on one or more clock cycles to generate an output or output value. Each output value in the set of outputs corresponds to a respective element of the output tensor 508. Multiplying the activation tensor 504 with the weight tensor 506 includes multiplying the activations from the elements of the tensor 504 with the weights from the elements of the tensor 506 to generate partial sum(s).

In some implementations, the processor cores of the system 100 operate on vectors that correspond to i) discrete elements in a multidimensional tensor, ii) vectors of values that include multiple discrete elements along the same or different dimensions of a multidimensional tensor, or iii) combinations of each. Each discrete element or multiple discrete elements in a multidimensional tensor can be represented using X,Y coordinates (2D) or using X,Y,Z coordinates (3D), depending on the dimensionality of the tensor.

The system 100 can calculate multiple partial sums corresponding to products generated by multiplying the batch inputs with corresponding weight values. As described above, the system 100 can perform the accumulation of products (e.g., partial sums) over multiple clock cycles. For example, the accumulation of products can be performed in the shared memory 104 based on techniques described herein. In some implementations, the input-weight multiplication can be written as a sum of products where each weight element is multiplied with a discrete input of the input volume (e.g., a row or slice of the input tensor 504). The row or slice can represent a given dimension (e.g., a first dimension 510 of the input tensor 504, or a second, different dimension 515 of the input tensor 504).

In some implementations, an exemplary set of computations can be used to compute the output of a convolutional neural network layer. The computations for a CNN layer may include performing a 2D spatial convolution between a 3D input tensor 504 and at least one 3D filter (weight tensor 506). For example, convolving one 3D filter 506 with a 3D input tensor 504 may produce a 2D spatial plane 520 or 525. The computations may include computing a sum of dot products for a particular dimension of the input volume.

For example, spatial plane 520 may include sum-of-products output values calculated from inputs along dimension 510, while spatial plane 525 may include sum-of-products output values calculated from inputs along dimension 515. The calculations to generate the sum-of-products output values in each of spatial planes 520 and 525 may be performed in shared memory 104 (e.g., in shared memory cells 445) using the cumulative reduction steps described herein.

FIG. 6 is a flow diagram illustrating an example process 600 for performing vector reduction using a shared scratchpad memory in a hardware circuit having processor cores in communication with the shared memory. In some implementations, process 600 is part of a technique used to accelerate neural network computations using the shared memory of FIG. 1.

Process 600 may be implemented or performed using system 100 described above. A description of process 600 may refer to the computational resources of system 100 described above. Steps or actions of process 600 may be enabled by programmed firmware or software instructions executable by one or more processors of the devices and resources described in this document. In some implementations, steps of process 600 correspond to a method of performing computations to generate outputs of neural network layers using hardware circuitry configured to execute a neural network.

Now referring to process 600, vectors of values are generated (602) in system 100. For example, for each processor core included in one or more hardware circuits of system 100, a respective vector of values is generated based on computations performed by the processor core.

The shared memory of the system 100 receives each of the vectors of values (604). For example, the shared memory 104 receives the vectors of values from the resources of each of the processor cores using the direct memory access (DMA) data path of the shared memory 104. In some implementations, the vectors or vectors of values are generated by a single processor core (or each of multiple processor cores) and then provided to the shared memory of the system 100, which performs a calculation using the vector of values. For example, the shared memory can obtain a vector from a first processor core and perform a reduction operation using the obtained vector and one or more other vectors. The one or more other vectors may be received or obtained from processor cores other than the first processor core.

In some other implementations, the system 100 is configured to perform direct store operations in conjunction with accumulation operations. For example, the system 100 can generate accumulation flags 330, 335 that are used to directly store one or more vectors of values in shared memory locations in the shared memory 104. The vectors may be from a single processor core or may be from multiple different processor cores. For example, the processor cores 105-1 or 302-2 can generate control signals representing the accumulation flags and pass the control signals to the control unit 201 of the shared memory 104. The system 100 can be configured to store the vectors of values in the vector memories 106, 108 and then perform a DMA operation to move the vectors of values from the vector memories to the shared memory 104.

The system 100 performs an accumulation operation on each of the vectors of values (606). More specifically, the shared memory 104 performs the accumulation operation as each of the vectors of values is written to a shared memory location. For example, the system 100 causes the shared memory 104 to perform an accumulation operation on each of the vectors of values using an operator 320 coupled to the shared memory 104. The system 100 is operable to accumulate values corresponding to different elements (or values) of the same vector and elements of different vectors. The operator 320 is configured to accumulate values based on arithmetic operations encoded in the operator units. In some implementations, the arithmetic operations are mathematical operations controlled by commutativity. The arithmetic operations may include atomic reductions (e.g., atomic floating-point reductions).

For example, atomic reduction is treated as an accumulation or vector reduction step in which a vector of values is accumulated directly into a memory location (e.g., a shared cell) of a shared memory. In one example, the system 100 accumulates multiple vectors generated from multiple different cores as part of an accumulation operation. In another example, the system 100 accumulates values (e.g., vectors) already stored in the shared memory 104 (e.g., in a shared cell of the memory) with values generated by the core. In another example, the system 100 accumulates multiple vectors generated from multiple different cores with one or more values already stored in the shared memory 104. The above examples involving vectors generated by the cores and values already stored in the shared memory are also applicable to reduction operations, and other types of arithmetic operations that may be performed using the arithmetic unit 320.

In some other implementations, each of the processor cores 302-1, 302-2 provides a vector that needs to be accumulated, and the values are accumulated directly in memory locations without synchronizing activity between the processor cores 302-1, 302-2. Similarly, values can be accumulated directly in memory locations without either of the processor cores 302-1, 302-2 performing a step of pre-accumulating products (e.g., partial sums) that may result from calculations performed on either of the processor cores. In other words, two or more cores of the system 100 can accumulate vectors of values containing partial sums in address locations (e.g., central address locations) of shared memory cells of the memory 104 in any order. The system 100 is configurable in some implementations such that no pre-accumulation operations need to be performed locally at a core, and in some other implementations such that some of the partial sums or certain types of partial sums can be accumulated at a given core.

The system 100 generates a result vector (e.g., a final result vector) based on the accumulation operation (608). For example, the system 100 generates the final result vector based on performing an accumulation operation with one or more vectors of values and a vector stored in the shared memory. In some implementations, the system 100 generates a result vector whose individual elements result from pairwise application of the accumulation operation to each element of the first vector and each corresponding element of the vector stored in the shared memory. The result vector provides the correct mathematical result of the accumulation regardless of the order in which each of the vectors accumulated to generate the final result arrive at the shared memory cell.

In some implementations, to achieve this desired result, one or more DMA operations may be detected, prioritized, and ordered based on control 435 (e.g., a read-modify-write control loop) performed using control unit 201 and at least control logic 325 of operator 320. For example, control unit 201 may detect a set of incoming vectors/vector values with corresponding cores providing vectors, and use operator 320 to serialize individual accumulation operations based on a given priority scheme specified by control 435. The priority scheme may be used to stop or reorder write traffic as necessary to prevent vector values from accumulating against older vector values.

The result vector may be a final result vector representing a set of outputs of the neural network layer. For example, the neural network layer may be a convolutional neural network layer, and the output may be a set of activation values generated in response to convolving each kernel (e.g., parameters/weights of tensor 506) over a particular input volume of input tensor 504.

The system 100 can generate a vector of accumulated values as a result of performing an accumulation operation on each of the vectors of values. In some implementations, each of the vectors of values is a partial sum corresponding to a dot product. For example, referring back to the convolutional neural network layer, the inputs of the input volume described above are processed by performing a dot product operation with i) each input value along a given dimension (e.g., dimension 510) of the input tensor 504 and ii) a set of parameters of the convolutional layer. In response to convolving at least one kernel of the weight tensor 506 with a portion of the input along the given dimension of the input volume, a corresponding set of dot products or partial sums can be accumulated in memory locations of the shared memory 104 to generate a set of accumulated values.

The system 100 can apply an activation function to each value in the vector of accumulated values. For example, the layer of the neural network may (or may not) have an activation function (such as ReLU, sigmoid, or tanh) that represents a nonlinear function that provides nonlinearity in the neural network. The system 100 generates a result vector in response to applying the activation function to each value in the vector of accumulated values. In some implementations, the hardware circuit 101 is a hardware accelerator configured to execute a neural network that includes multiple neural network layers, and the system 100 generates an output of the layer of the neural network based on the result vector. For example, processing the layer inputs at the neural network layer may include the layer applying an activation function to generate a set of activation values that are the output of the neural network layer. The activations generated by the first neural network layer can be processed through a second or subsequent layer of the neural network.

The embodiments and functional operations of the subject matter described herein may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed herein and structural equivalents thereof, or in a combination of one or more of these. The embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or for controlling the operation of a data processing apparatus.

Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information that is transmitted to an appropriate receiving device for execution by a data processing device. The computer storage medium may be a machine-readable storage medium, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these.

The term "computing system" encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, computer, or multiple processors or computers. An apparatus may include special purpose logic circuitry (e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit)). In addition to hardware, an apparatus may also include code that creates an execution environment for a target computer program (e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations of these).

Computer programs (which may also be referred to or described as programs, software, software applications, modules, software modules, scripts, or code) may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages) and may be deployed in any form (such as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment).

A computer program may, but need not, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple associated files (e.g., a file that stores one or more modules, subprograms, or portions of code). A computer program may be deployed to run on one computer, or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communications network.

The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data to generate output. These processes and logic flows may also be performed by, and devices may be implemented as, special purpose logic circuitry (e.g., an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (general purpose graphics processing unit)).

A computer suitable for executing a computer program may include or be based on, by way of example, a general-purpose or special-purpose microprocessor or both, or other types of central processing units. Typically, the central processing unit receives instructions and data from a read-only memory or a random-access memory or both. Some elements of a computer are a central processing unit for implementing or executing instructions, and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or is operatively coupled to receive, transmit, or transmit data to or from one or more of these mass storage devices. However, a computer need not have such devices. Additionally, a computer may be incorporated in another device (e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few).

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

To provide for user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device (e.g., an LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices may also be used to provide user interaction. For example, feedback provided to the user may be any form of sensory feedback (e.g., visual, auditory, or tactile feedback), and input from the user may be received in any form, including acoustic, speech, or tactile input. Additionally, the computer may provide for user interaction by sending and receiving documents to and from a device used by the user (e.g., by sending a web page to a web browser on the user's client device in response to receiving a request from the web browser).

Embodiments of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., as a data server, or a middleware component (e.g., an application server), or a front-end component (e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein), or any combination of one or more such back-end, middleware, or front-end components. These components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include local area networks ("LANs") and wide area networks ("WANs") (e.g., the Internet).

A computing system may include clients and servers. Clients and servers are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship.

Although the specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the specification in terms of separate embodiments may also be realized in combination in a single embodiment. Conversely, various features described in the specification in terms of a single embodiment may also be realized in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described as functioning in a particular combination and initially claimed as such, one or more features included in a claimed combination may, in some cases, be omitted from the combination, and the claimed combination may relate to a subcombination or a variation of the subcombination.

Similarly, although operations are shown in a particular order in the figures, this should not be understood as requiring such operations to be performed in the particular order or sequential order shown, or that all of the operations shown must be performed to achieve a desired result. Multitasking and parallel processing may be advantageous in certain situations. Furthermore, the separation of various system modules and components in the above embodiments should not be understood as requiring such separation in all embodiments. It should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims (25)

1. A method implemented using a hardware circuit having a shared memory and a plurality of processor cores in communication with the shared memory, the method comprising:
generating a first vector of values based on vector operands operated on by a vector processing unit of the first processor core;
the shared memory receiving, using a direct memory access (DMA) data path of the shared memory, the first vector of values from the first processor core;
performing an accumulation operation between the first vector of values and a vector stored in the shared memory;
The accumulation operation is performed using a calculator unit, the calculator unit comprising:
i) configured to accumulate respective values of one or more vectors;
ii) a first vector of values is routed to the shared memory such that the first vector of values is accumulated into the vector stored in the shared memory external to the first processor core, the method further comprising:
generating a result vector based on the accumulation operation. The vector stored in the shared memory is received from a second processor core, and the method further comprises:
performing an accumulate operation into memory to accumulate each value of the first vector of values using memory locations of the shared memory;
and performing an accumulate operation into memory to accumulate each value of a second vector of values using the memory locations of the shared memory. generating the result vector based on the accumulation operation,
generating the result vector without the first processor core performing a step of pre-accumulating products resulting from calculations performed by the first processor core;
and generating the result vector without the second processor core performing a step of pre-accumulating products resulting from calculations performed by the second processor core. Generating the result vector comprises:
generating a vector of accumulated values as a result of performing the accumulation operation on the first vector of values;
applying an activation function to each value in the vector of accumulated values;
and generating the result vector as a result of applying the activation function to each value in the vector of accumulated values. The resource of each of the first processor cores is a first matrix calculation unit, and the method further comprises:
4. The method of claim 2 or 3, comprising generating a first vector of accumulated values corresponding to the first vector of values based on a matrix multiplication performed using the first matrix computation unit of the first processor core. Each resource of the second processor core is a second matrix calculation unit, and the method further comprises:
6. The method of claim 5, comprising generating a second vector of accumulated values corresponding to the second vector of values based on a matrix multiplication performed using the second matrix computation unit of the second processor core. the hardware circuit is a hardware accelerator configured to execute a neural network including a plurality of neural network layers;
The method of any one of claims 1 to 6, wherein the method comprises generating an output of a layer of the neural network based on the result vector. generating a first vector of values based on calculations performed on the first processor core;
generating a second vector of values based on calculations performed on the second processor core;
7. The method of claim 2, 3, 5, or 6, wherein the calculations performed by the first processor core and the calculations performed by the second processor core are part of a mathematical operation that is controlled by commutativity. The mathematical operation is
Floating-point multiplication operations,
Floating-point addition operations,
The method of claim 8, wherein the operation is an integer addition operation; or a min-max operation. The method of claim 8, wherein the mathematical operations include floating point addition operations and integer addition operations. The method according to any one of claims 2, 3, 5, 6, and 8 to 10, wherein the first processor core and the second processor core are the same processor core. The method of any one of claims 1 to 11, wherein the shared memory is configured to function as a shared global memory space including memory banks and registers shared between two or more processor cores of the hardware circuit. 1. A system comprising:
A processing device;
a hardware circuit having a shared memory and a plurality of processor cores in communication with the shared memory;
and a non-transitory machine-readable storage device for storing instructions executable by the processing device to perform operations, the operations including:
generating a first vector of values based on vector operands operated on by a vector processing unit of the first processor core;
the shared memory receiving, using a direct memory access (DMA) data path of the shared memory, the first vector of values from the first processor core;
performing an accumulation operation between the first vector of values and a vector stored in the shared memory;
The accumulation operation is performed using a calculator unit, the calculator unit comprising:
i) configured to accumulate respective values of one or more vectors;
ii) the first vector of values is routed to the shared memory such that the first vector of values is accumulated into the vector stored in the shared memory external to the first processor core, the operations further comprising:
generating a result vector based on the accumulation operation . the vector stored in the shared memory is received from a second processor core, and the operation comprises:
performing an accumulate operation into memory to accumulate each value of the first vector of values using memory locations of the shared memory;
and performing an accumulate operation into memory to accumulate each value of a second vector of values using the memory locations of the shared memory. generating the result vector based on the accumulation operation ,
generating the result vector without the first processor core performing a step of pre-accumulating products resulting from calculations performed by the first processor core;
and generating the result vector without the second processor core performing a step of pre-accumulating products resulting from calculations performed by the second processor core. Generating the result vector comprises:
generating a vector of accumulated values as a result of performing the accumulation operation on the first vector of values;
applying an activation function to each value in the vector of accumulated values;
and generating the result vector as a result of applying the activation function to each value in the vector of accumulated values. The resource of each of the first processor cores is a first matrix calculation unit, and the operation further comprises:
16. The system of claim 14 or 15, further comprising generating a first vector of accumulated values corresponding to the first vector of values based on a matrix multiplication performed using the first matrix computation unit of the first processor core. The respective resources of the second processor core are second matrix calculation units, and the operations further include:
20. The system of claim 17, further comprising generating a second vector of accumulated values corresponding to the second vector of values based on a matrix multiplication performed using the second matrix computation unit of the second processor core. the hardware circuit is a hardware accelerator configured to execute a neural network including a plurality of neural network layers;
The system of any one of claims 13 to 18, wherein the operations include generating an output of a layer of the neural network based on the result vector. generating a first vector of values based on calculations performed on the first processor core;
generating a second vector of values based on calculations performed on the second processor core;
19. The system of claim 14, 15, 17, or 18, wherein the computations performed by the first processor core and the computations performed by the second processor core are part of a mathematical operation controlled by commutativity. The mathematical operation is
Floating-point multiplication operations,
Floating-point addition operations,
21. The system of claim 20, wherein the operation is an integer addition operation; or a min-max operation. The system of claim 20, wherein the mathematical operations include floating point addition operations and integer addition operations. The system of any one of claims 14, 15, 17, 18, and 20-22, wherein the first processor core and the second processor core are the same processor core. The system of any one of claims 13 to 23, wherein the shared memory is configured to function as a shared global memory space including memory banks and registers shared between two or more processor cores of the hardware circuit. A computer program storing instructions executable by a processor to cause the processor to perform operations, the operations comprising:
generating a first vector of values based on vector operands operated on by a vector processing unit of the first processor core;
a shared memory receiving the first vector of values from the first processor core using a direct memory access (DMA) data path of the shared memory;
performing an accumulation operation between the first vector of values and a vector stored in the shared memory;
The accumulation operation is performed using a calculator unit, the calculator unit comprising:
i) configured to accumulate respective values of one or more vectors;
ii) the first vector of values is routed to the shared memory such that the first vector of values is accumulated into the vector stored in the shared memory external to the first processor core, the operations further comprising:
generating a result vector based on the accumulation operation.

Images