WO2021212045A1

WO2021212045A1 - Synchronization of processing elements and software managed memory hierarchy in a machine learning accelerator

Info

Publication number: WO2021212045A1
Application number: PCT/US2021/027787
Authority: WO
Inventors: Nishit Shah; Srivathsa DHRUVANARAYAN; Reed KOTLER
Original assignee: SiMa Technologies, Inc.
Priority date: 2020-04-17
Filing date: 2021-04-16
Publication date: 2021-10-21

Abstract

A method, system, and apparatus are disclosed herein for bridging a deterministic phase of instructions with a non-deterministic phase of instructions when those instructions are executed by a machine learning accelerator while executing a machine learning network. In the non-deterministic phase, data and instructions are transferred from off-chip memory to on-chip memory. When the transfer is complete, processing elements are synchronized and, upon synchronization, a deterministic phase of instructions is executed by the processing elements.

Description

SYNCHRONIZATION OF PROCESSING ELEMENTS AND SOFTWARE

MANAGED MEMORY HIERARCHY IN A MACHINE LEARNING

ACCELERATOR

CROSS REFERENCE TO RELATED APPLICATION [0001] This application claims priority to U.S. Patent Application No. 16/852,326, filed April 17, 2020, and U.S. Patent Application No. 16/852,338, filed April 17, 2020, both of which are incorporated herein by reference in their entirety.

BACKGROUND

Technical Field

[0002] This disclosure relates generally to the implementation of machine learning networks on hardware.

Description of Related Art

[0003] Machine learning is one of the most powerful recent trends in technology. In machine learning, a model is developed to perform a certain task. The model, which will be referred to as a machine learning network, is trained and deployed in order to carry out that task. For example, a model may be developed to recognize the presence of objects within images captured by a set of cameras. Once the model is deployed, images captured by the cameras are input to the machine learning network, which then outputs whether (or to what confidence level) objects are present within the images.

[0004] Machine learning networks typically require the handling of a large volume of data and the execution of a large number of computations. As a result, they are commonly implemented in compute facilities with access to significant resources, such as in the cloud or on server clusters. However, the sources of input to machine learning networks may be located remotely from these compute facilities. For example, cameras and other types of sensors may be located on the edge of the network. Example applications for edge devices include automotive and other forms of transportation including autonomous transportation, agricultural, industrial, robotics, drones, surveillance and security, smart environments including smart cities, medical and personalized health. Example tasks include computer vision, image analysis, image understanding, speech recognition, audio analysis, audio understanding, natural language processing, classification and pattern recognition tasks. For edge devices, it may be desirable to perform certain tasks in real-time. In addition to memory and other programmable processors, an edge device may also include sensors, such as cameras (both still image and video cameras), microphones, temperature sensors, pressure sensors and other types of sensors. The sensors may capture samples that are used as inputs to a computing pipeline within the edge device. Thus, one common paradigm is for the input sources to be web-based so that they can continuously send their captured data to the cloud- based compute facility, which then executes the machine learning network and returns the result.

[0005] There can be many advantages if the machine learning network was instead embedded on edge devices, such as combined with the camera system. However, many types of edge devices (e.g., cameras) have resource limitations (e.g., limited memory, power consumption, etc.). This may be due to power consumption, build complexity, size, and other considerations. Thus, when machine learning networks are placed on edge devices, those edge devices may be adapted to deal with limited resources. For example, limited memory may be an issue because memory banks add to the size, complexity, and power consumption of those devices. This holds true especially for memory that resides on the same die as the processor itself (e.g., static random-access memory (SRAM)). Therefore, memory optimization both in hardware and software may be advantageous to execute machine learning networks on edge devices. As another example, processing elements may be simplified to conserve resources, but this may give rise to a need to synchronize processing elements in their execution.

BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Embodiments of the disclosure have other advantages and features which will be more readily apparent from the following detailed description and the appended claims, when taken in conjunction with the examples in the accompanying drawings, in which:

[0007] Figure (FIG.) 1A is a block diagram of a system with a machine learning accelerator (MLA) and corresponding compiler.

[0008] FIG. IB illustrates actions that the MLA may perform during a deterministic phase of instructions.

[0009] FIG. 2A is a block diagram of a hardware system including a Machine Learning Accelerator (MLA).

[0010] FIG. 2B is a block diagram of a Tile within the MLA. [0011] FIG. 3 is a block diagram of a software development environment including a machine learning (ML) compiler.

[0012] FIG. 4 illustrates a process for bridging a non-deterministic phase and a deterministic phase.

[0013] FIG. 5 illustrates an architecture for determining whether a memory location is storing live data and marking a memory location for storing live data or not storing live data. [0014] FIG. 6 illustrates bridging execution of a non-deterministic phase of instructions and a deterministic phase of instructions.

[0015] FIG. 7 illustrates actions that the MLA may perform during a non-deterministic phase of instructions.

[0016] FIG. 8 illustrates a mesh of Tiles organized into rows and columns.

[0017] FIG. 9 illustrates a Tile having two instruction queues.

[0018] FIG. 10 illustrates an alternative process for synchronizing Tiles for execution of a deterministic phase.

[0019] FIG. 11 A includes an MLA with a 6x6 mesh of Tiles.

[0020] FIG. 11B illustrates deterministic and non-deterministic phases for partitions.

[0021] FIG. 12 is a block diagram of an integrated circuit that includes an MLA.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0022] The figures and the following description relate to preferred embodiments by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily recognized as viable alternatives that may be employed without departing from the principles of what is claimed.

I. Machine learning accelerator and compiler

[0023] FIG. 1A is a block diagram of one example of a system with a machine learning accelerator (MLA) 170 and corresponding compiler 120, according to the invention. The compiler 120 receives a description of a machine learning network 100 and generates a computer program 150 that implements the machine learning network using MLA 170. The computer program 150 includes instructions that are executed by processing elements (Tiles) in the MLA according to a schedule determined by the compiler. For convenience, these will be refereed to as statically scheduled instructions. The instructions executed by the Tiles (Tile instructions) are statically scheduled because the compiler can determine which instructions are executed by which Tiles at what times, as will be explained in greater detail below. For example, for the statically scheduled instructions, there are no conditions, branching or data dependencies that can be resolved only at run-time, and which would affect the timing and order of the execution of the instructions. Note that the static schedule determined by the compiler may or may not be included as part of the instructions and computer program. In some embodiments, the computer program may expressly include the schedule, specifying that instruction A is executed at cycle X, instruction B is executed at cycle X+4, instruction C is executed at cycle X+12, etc. In alternate embodiments, the computer program may specify only that instruction A is executed, followed by instruction B, and then instruction C, but without any scheduling information. Even though the static schedule is not expressly specified, these instructions will still execute according to the schedule determined by the compiler because the compiler knows how long it takes to execute each instruction. As a result of the static scheduling, the MLA and instruction set for the MLA may be simplified, with the complexity offloaded to the compiler. A simpler MLA can result in lower cost, lower power consumption and higher performance, all of which are desirable for implementation in edge devices.

[0024] In more detail, the MLN 100 may be described by an architecture and parameters. A depiction of an MLN is shown to the right of box 100 in FIG. 1A. Most MLNs include multiple layers 112, each with one or more nodes which are represented by circles in FIG. 1 A. The lines between nodes in FIG. 1A represent interconnections between the nodes (and layers). Each node calculates a weighted sum of the values received from its connected nodes, possibly also applying a bias. Examples are matrix multiplication and convolution. Each node may also apply certain functionality (operators), such as nonlinear functions (e.g., tanh function), softmax operator, etc. A typical node may compute an output: y = F(åwiXi + b) (1) where xi are the inputs received from other nodes i, wi are weights, b is a bias and F() is a nonlinear operator. The MLN architecture includes the number of nodes (and layers) and their interconnectivity, and the operators applied at nodes. The operators may be described in a parameterized form. The MLN parameters include the weights, biases, and parameters for the operators.

[0025] MLNs may vary in size, depending on the desired task. Small MLNs may have 5-10 or fewer layers, medium size MLNs may have 30-50 layers, and large MLNs may have 100 or more layers. Examples of inputs include text, images and video. Some of the layers may be fully interconnected (i.e., every node in one layer provides input to every node in the next layer), and others may be more locally interconnected (e.g., to implement convolutions). Each weighted interconnect represents a scalar multiplication. The total number of scalar multiplications required to implement an MLN may be on the order of millions, billions, tens of billions or even more. These may be carried out by matrix multiplications.

[0026] The MLA 170 includes a plurality of Tiles 180 and an on-chip memory system implemented on a semiconductor die. The Tiles are organized into one or more meshes of interconnected Tiles. A depiction of a Tile mesh is shown to the right of box 170 in FIG. 1 A. In each mesh, the Tiles 180 are organized in a regular pattern and the interconnections within each mesh provide data transfer paths between Tiles in the mesh. The Tiles execute computations according to instructions received by the Tiles and using data stored in the on- chip memory system. These instructions may be for computations and/or for data transfer. Computations include multiply (including matrix multiply), add, and operators (e.g., nonlinear functions, lookup table, min/max, pooling). These are computations that implement the MLN. In the example of FIG. 1A, the computations performed by layers 112A-D are allocated to groups 182A-D of Tiles as indicated. The allocation is not required to be 1:1. For example, multiple layers could be allocated to a single Tile or vice versa. Not every computation required to implement an MLN need be executed by a Tile; some computation may be executed outside the MLA (e.g., floating point operations, if the Tiles only do integer arithmetic). Tiles typically will at least perform matrix multiplication.

[0027] The compiler 120 receives a description of the MLN 100 and generates a computer program 150 that implements the MLN using the MLA 170. The computer program 150 receives an input sample for the MLN and executes the operations of the MLN to produce the output for the MLN. The computer program 150 includes instructions to be executed by the Tiles for implementing computations in the MLN and may also include instructions to be executed by other elements, such as a controller outside the Tiles. For additional examples and description of the MLA and related components, see U.S. Appl. No. 16/840,216, “Machine Learning Network Implemented by Statically Scheduled Instructions, with Compiler,” which is incorporated by reference herein in its entirety.

[0028] As shown in FIG. 1 A, the compiler partitions the Tile instructions into one or more deterministic phases 190 which typically utilize multiple Tiles. The instructions in a deterministic phase 190 may be statically scheduled by the compiler. For example, a deterministic phase 190 may include a series of computations required to implement a portion of the MLN, where the time required for each computation and associated data transfers is known. As a result, the compiler may statically schedule the Tile instructions within that deterministic phase relative to the other Tile instructions in the phase. The resulting computer program produced by the compiler then implements an allocation of instructions to Tiles and a schedule for executing the instructions as determined by the compiler, although these may not be expressly contained with the computer program. In the example of FIG.

1A, the computations performed by layers 112A-D are allocated to groups 182A-D of Tiles as indicated. In addition, all the Tile instructions (including both for computation and for data transfer) are executed in a single deterministic phase.

[0029] The computer program may also include non-deterministic phases 192. For example, non-deterministic phases 192 may include data fetch or instruction fetch from off- chip memory where the time required to execute the operation varies too much to allow reliable synchronization with other operations. Other examples include computations that occur off-chip, and conditions, branching and other programmatic constructs that depend on values not known until run-time. The breaks in the rectangles for the non-deterministic phases 192 indicate that the timing is not deterministic, whereas the deterministic phases 190 are represented by rectangles without breaks. In FIG. 1A, the deterministic and non- deterministic phases are shown as alternating. This is not required. For example, deterministic and non-deterministic phases may execute concurrently.

[0030] When the non-deterministic phase is completed and the Tiles are ready to execute instructions, the deterministic phase may begin. FIG. IB shows more detail of a deterministic phase, which shows the static schedule computed by the compiler for executing Tile instructions in this phase. The phase begins at some time with when all of the Tiles are synchronized, which for convenience is marked as cycle cO in FIG. IB. The Tiles may have circuitry that synchronizes the Tiles. For example, each Tile may monitor when it is ready to begin execution of a deterministic phase and then actual execution begins when all Tiles signal that they are ready. Alternatively, an external controller may synchronize the Tiles and start the deterministic phase when all tiles are ready.

[0031] FIG. IB illustrates actions that the MLA may perform during a deterministic phase. In this example, the instructions are executed by three Tiles, as denoted by Tl, T2 and T3. Each Tile has two pipelines: a “D” pipeline for executing data transfer instructions and a “C” pipeline for executing compute instructions. The row labeled Tl D shows instructions executed by the Tile 1 D (data transfer) pipeline, and the row labeled Tl C shows instructions executed by the Tile 1 C (compute) pipeline. For this example, assume that all the data transfer instructions are instructions that load new data into that Tile for consumption by the compute pipeline. The white regions of each row denote the execution of instructions and the hashed regions indicate that the pipeline is idling or executing a NO-OP (no operation).

[0032] For Tile 1, instruction 155a transfers data into Tile 1 and instruction 155b then performs a computation that consumes that data. Instruction 155b is dependent on instruction 155a. Here, the T1 C pipeline is not required to continuously poll the T1 D pipeline at run time for when the data is available, and run-time message passing between the pipelines is not required to indicate that the data is available. Rather, because the duration (i.e., time required to execute) of instruction 155a is known, the compiler knows when the data will be available (for convenience, marked as cycle cl in the figure) and can construct a static schedule in which instruction 155b starts execution then. The duration of instruction 155b is also known, so the compiler knows that compute instruction 155d may start after instruction 155b. In this case, the compiler determines a static schedule in which instruction 155d starts at cycle c3. Compute instruction 155d depends on data brought into the Tile by instruction 155c. The duration of instruction 155c is known, so the compiler knows that in the static schedule, instruction 155c must start at cycle c2 or earlier. This pattern is repeated for pairs of data transfer instructions and compute instructions 155e-f, 155g-h, 155i-j.

[0033] For Tile 2, compute instruction 1551 depends on data from data transfer instruction 155k. However, instruction 155k does not start immediately at cycle cO. Rather, it has a delayed start at cycle c4. This may be because the data transfer path required by instruction 155k is occupied by some other data transfer instruction and is not available until cycle c4. The start time of instruction 155k in the static schedule is not determined by run time arbitration or contention mechanisms for the shared data transfer path. Rather, the compiler knows that the data transfer path is occupied since the compiler knows the start times and durations of all the instructions, so the compiler simply creates a static schedule in which instruction 155k does not start until cycle c4 when the compiler knows the data transfer path will be available. Similarly, data transfer instruction 155m has a delayed start time. Perhaps the T2 D pipeline is being used to transfer out the results of computation 1551 and does not become available until cycle c5.

[0034] For Tile 3, computation 155n starts immediately at cycle cO. Perhaps the required data was loaded into Tile 3 during some prior phase. Data transfer instructions 155o and 155p load data for compute instruction 155q. They are separated in time, perhaps because different pieces of data were not available or the data transfer paths were not available until those times. As a final example, data transfer instruction 155r loads data for compute instruction 155s. In the static schedule, the compiler places instruction 155r well in advance of when the data is required, but this may be because that is when the data transfer path is available or perhaps the data was transferred out of the sourcing Tile in order to make room in that Tile.

[0035] Execution of the instructions according to the static schedule at run-time may be implemented in different ways. In one approach, the computer program includes an express schedule for the execution of the instructions. Continuing the example of FIG. IB, the computer program may specify that instruction 155a executes at cycle cO, instruction 155b at cycle cl, instruction 155c at cycle c2, etc. Alternatively, the compiler may fill each instruction stream with NO-OPs to achieve the correct timing. A NO-OP (no operation) is an instruction that occupies a certain number of cycles without other activity. For example, the compiler knows that instruction 155a will end at cycle cl and instruction 155b is supposed to begin at cycle cl. It may fill the space between cycles cO and cl with NO-OPs for the T1 C pipeline. The T1 C pipeline then just continuously executes instructions from its queue, and the NO-OPs ensure that instruction 155b is executed according to the compiler’s static schedule. In yet another approach, the static schedule may be implemented by hardware.

The T1 C pipeline may just stall on the execution of instruction 155b until the data from instruction 155a is ready. The compiler knows that data will be ready at cycle cl and, therefore, instruction 155b will execute starting at cycle cl even though the Tiles are unaware of the static schedule. Regardless of the implementation, for convenience, all of these situations will be described using the phrase “static schedule.” Thus, a statement that the compiler statically schedules the instructions is intended to include all of the above implementations and is not meant to imply that the computer program expressly includes a scheduled time for each instruction.

[0036] In order to statically schedule the instructions in a deterministic phase, the compiler typically will know the duration of each instruction (i.e., how long each instruction takes to execute), the capabilities of each Tile (which Tiles can execute which instructions), the topology of data transfer paths to and from Tiles (including between Tiles, and between Tiles and on-chip memory), and the computations required and their dependencies (i.e., the MLN description). With this information, the compiler can schedule unconditional start times for the Tile instructions. Here, unconditional refers to run-time conditions. The execution order of statically scheduled instructions will not change as a result of run-time conditions, branching or dependence on input values. As a result, compute instructions may be scheduled for start times when all of the required data for the computation is known to be available and the compute pipeline is also known to be available. The need for run-time determination of whether data has arrived and whether the compute pipeline is available may be avoided. Analogously, data transfer instructions may be scheduled for start times when the data transfer path is known to be available. The need for circuitry to handle arbitrations, or to check for or resolve contentions and collisions on shared data transfer paths at run-time may be avoided. The need for routing tables and other circuitry to determine routing at run time may also be avoided.

[0037] Since the deterministic phase is statically scheduled, the compiler knows when the deterministic phase will end. A next non-deterministic phase of instructions may start based on the known end time for the current deterministic phase. The phases may continue while the MLN is executed by the device hosting the MLA.

[0038] The compiler may not be able to statically schedule all instructions because some data may be stored in off-chip memory where access times may not be known prior to run time. Thus, a deterministic phase of statically scheduled instructions may be followed by anon-deterministic state of instructions. FIG. IB also shows rows 160 and 161 that illustrate deterministic and non-deterministic phases. Row 160 represents a non-deterministic phase while row 161 represents a deterministic phase. The white regions of each row denote the execution of instructions and the hashed regions indicate idling. Non-deterministic instructions are indicated by breaks in the rectangles. Pipeline 160 includes instructions to load data for the computations of layers 1-10 from off-chip memory (e.g., DRAM) into the MLA. This data will be consumed by a Tile partition 182B. Referring to FIG. IB, this is performed by the microcontroller. Phase 165 is non-deterministic because it includes loads from off-chip memory (e.g., DRAM). Row 161 includes deterministic phase 166 of statically scheduled Tile instructions that implement the computations for layers 1-10.

[0039] The suffixes indicate different inputs samples. The phases that end in -a apply the MLN to one input sample, the phases that end in -b apply the MLN to the next input sample, etc. The arrows indicate dependencies. Consider first input sample a. A microcontroller loads 165a the relevant data (input values, weights, biases, operator parameters) from off-chip memory (e.g., DRAM) into the MLA memory (e.g., L2 memory). After this is completed, the Tiles may perform their computations 166a using this data.

[0040] The processing of input samples b and c have the same dependencies and general flow. For input sample b, the data load 165b completes early but the data must wait until partition 182B is available for computation. Once begun, the Tile computation 182B has a known duration. For input sample c, the data load 165c takes a long time. As a result, all the downstream operations are delayed. [0041] FIGs. 2-3 are more detailed descriptions of an example system that includes an MLA and corresponding compiler. FIG. 2 shows the hardware component and FIG. 3 shows the software development environment.

[0042] FIG. 2A is a block diagram of a hardware system including an MLA 270. The MLA 270 includes all the components shown in FIG. 2A, except the off-chip L3 memory 290. The MLA components are implemented on a single die as part of a single chip. The MLA 270 includes one or more mosaics 272A-N. In this example, all of the mosaics are the same. Each mosaic 272 includes a mesh of Tiles 280, an on-chip memory system and a controller 277. In FIG. 2A, the on-chip memory system is a multi-level memory system, which includes a level 1 (LI) memory distributed among the Tiles (see FIG. 2B) and a level 2 (L2) memory 274 shared by the Tiles. If there are multiple mosaics 272, the MLA 270 may include a dedicated interconnect 279 for connecting the different mosaics. Each mosaic also includes an interface 278 to the interconnect 279.

[0043] FIG. 2B is a block diagram of a Tile 280 within the MLA. In this example, all the Tiles are the same. Each Tile 280 includes an LI memory 282. Each Tile 280 also includes a data transfer pipeline that executes instructions for transferring data to and from the LI memory 282. Here, the Tiles 280 are arranged in a rectangular array as shown in FIG. 2A, with each Tile connected to its adjacent neighbors. Interior Tiles are connected to four adjacent Tiles. Edge Tiles are connected to adjacent Tiles and also to L2 memory 274. In FIG. 2B, the LI memory 282 may receive data from any of its adjacent Tiles and/or from L2 memory if it is an edge Tile. Similarly, it may transfer data to any of its adjacent Tiles and/or to L2 memory if it is an edge Tile. The data transfer operations are controlled by data transfer instructions received and executed by the Tiles.

[0044] Each Tile 280 also includes a compute pipeline 285 for executing computations using data stored in the LI memory 282. The LI memory acts as software-configurable registers for the compute pipeline 285. The compute pipeline 285 includes matrix multiplication circuitry 286, such as a systolic array, and circuitry for implementing different types of operators 287. The computations are controlled by compute instructions received and executed by the Tiles.

[0045] In this particular example, all of the data transfer instructions and compute instructions executed by the Tiles are statically scheduled. These instructions include data transfer between LI memories in different Tiles, and data transfer between LI memory and L2 memory. Data transfer instructions may specify one hop at a time (e.g., transfer data to the east neighbor Tile) or may specify destination and path through intermediate Tiles (e.g., transfer data to Tile (5,5) using path east-east-north-north-east). The instructions also include matrix multiplies performed by the Tiles and operators applied by the Tiles. These operations do not require very many different instructions to implement, so the overall instruction set may be fairly small, for example not more than 20 instructions, or not more than 50 instructions.

[0046] The L3 memory 290 is off-chip. In this example, the LI and L2 memories are implemented as on-chip SRAM and the L3 memory is implemented as DRAM (flash memory and SSD drives are other alternatives). Because the LI and L2 memories are implemented as SRAM, the data transfers between LI memories or between LI and L2 memories have deterministic timing, so these data transfer instructions can be statically scheduled by the compiler. However, data transfer from off-chip DRAM is more unpredictable in timing. As a result, these instructions are non-deterministic in nature and they are executed by the microcontroller 277. Therefore, they are executed in one of the non- deterministic phases and they are not statically scheduled.

[0047] In one approach, the instructions in the computer program and the data required for computation (e.g., input, weights, biases, parameters for operators) are initially loaded into L3 memory 280. From time to time, instructions and associated data are transferred from L3 memory into L1/L2 memory during anon-deterministic phase since the timing of data transfers from DRAM is not deterministic. Once these instructions and data are loaded into L1/L2 memory, the computer program enters a corresponding deterministic phase in which the Tiles execute the loaded instructions according to a static schedule. The non- deterministic and deterministic phases may occur concurrently. For example, data may be continuously streamed into the L1/L2 memory during the non-deterministic phase, with the corresponding statically scheduled instructions from the deterministic phase consuming that data. In one approach, the Tiles execute only statically scheduled instructions, and all non- statically scheduled instructions are executed by processing elements outside the Tile mesh, for example the microcontroller 277.

SRAM has predictable timing so implementing the LI and L2 memories as SRAM allows the compiler to statically schedule data transfers from those memories into the Tiles for computation. However, there is a limit to the amount of SRAM that may be implemented on a die. In order to increase the effective size of SRAM, a virtual SRAM approach may be used. In one approach, the compute instructions that consume certain data are not fetched into the Tiles until after the corresponding data have been transferred from DRAM (L3 memory) to SRAM (L1/L2 memory). This guarantees that the compute instructions will not be executed by the Tiles before the data is available. All data effectively will appear as if it is transferred to the Tiles from SRAM for computation, even if all of the data would not fit into the available SRAM.

[0048] L2 memory may also be used to temporarily store interim values that are too voluminous to store in LI memory. For example, a layer K of the MLN may produce a large amount of data at its output, to be used as input to the next layer K+l. The layer K output may be stored in L2 memory and then retrieved from L2 memory as needed for the next layer’s computations. This may be implemented using a ping pong buffer approach when multiple input samples are processed as a pipeline. The L2 memory is divided into two regions A and B. When a first input sample is processed, the layer K output is stored in region A of the L2 memory. The computations for layer K+l retrieve the stored values from region A. At the same time, the second input sample is processed and the layer K output is stored in region B of the L2 memory. The two regions then alternate, with the Tiles implementing layer K storing to one region while the Tiles implementing layer K+l read from the other region. The synchronization is implemented by the static scheduling. The compiler knows when regions A/B will be ready and the instructions to implement layer K+l will execute after that time. No synchronization primitives are needed.

[0049] FIG. 3 is a block diagram of a software development environment including an ML compiler 320. In this example, the software development environment also includes a model optimizer 330. The model optimizer 330 receives a description of the MLN 300 and produces an optimized graph 335 of the MLN. It may apply optimizations such as quantization 331, pruning 332 and/or compression 333. Quantization 331 reduces the resolution of calculated values. For example, floating point values may be quantized to a certain number of bits and then integer math used instead of floating point math. This reduces the complexity and power consumed by the Tiles. Pruning 332 removes parts of the MLN that do not contribute significantly to the overall results. For example, if certain weights are zero or close to zero, those weighted interconnects may be pruned. Finally, because MLNs contain a large amount of data, compression may be used successfully to reduce data transfer bandwidths.

[0050] The resulting optimized description 335 of the MLN may be expressed as a graph, in which the nodes of the graph represent nodes in the MLN and the edges of the graph represent the weighted interconnects. The compiler 320 receives the optimized graph 335 and produces the resulting computer program 350. The compiler 320 may perform operations including static scheduling 322, PPA (power performance area) optimizations 324, graph optimizations 326 and/or partitioning 328. Static scheduling 322 of the appropriate instructions was described above.

[0051] PPA optimization 324 includes different optimizations of the computer program 350. For example, the allocation of MLN computations to Tiles may be optimized to reduce power consumption, to increase performance (such as reducing latency or increasing throughput) and/or to reduce area (e.g., number of Tiles used). Examples of this are described in FIG. 4.

[0052] For a given graph representation of an MLN, the number of computations required to execute the MLN is fixed. As a result, in one approach, the compiler may optimize to increase the utilization of compute resources in the Tiles - to keep the compute pipelines as busy as possible. However, for a Tile to execute a computation, the data for that computation must be available. This means that any prior computations must be completed and that those results must be transferred to the Tile doing the next computation. Thus, rather than focusing on computations, the compiler may optimize with respect to data transfer to reduce the wait times of computations. It may also allocate computations to Tiles in order to reduce data transfers between Tiles in the same mesh, to reduce data transfers from outside the MLA and/or to reduce data transfers that cross the boundary of the mesh (e.g., reducing data transfers between LI and L2 memory and trying to keep all data in LI memory).

[0053] The compiler 320 may also optimize 324 the computer program 350, subject to constraints on power, performance, area and/or any of the quantities described above. Graph optimization 326 includes analysis of the graph representing the MLN to prune, merge or quantize links, parameters, values, and layers to achieve better performance. Partitioning 328 concerns mapping the computations in the MLN to an implementation on the MLA. This includes determining which computations are allocated to which Tiles and how data flows through the mesh of Tiles during computation. If there are multiple mosaics, it also includes determining which computations are allocated to which mosaics.

[0054] The resulting computer program 350 may be loaded into memory for execution on a machine learning accelerator 370. For example, one possible application is object detection. In this case, the inputs are images captured by a video camera. The MLN 300 has been trained to identify certain objects in the video images. The computer program 350 implementing the MLN is loaded onto memory that is accessible by the MLA 370, which is implemented as a chip inside the camera. This way, images captured by the video camera may be immediately analyzed by the computer program 350 running on the MLA 370. [0055] In addition to the MLA 370, the computer program 350 or parts of it may be run on a software simulator 336 and/or hardware emulator 338 (including FPGAs configured as MLAs). These may be used for product development, debugging and/or prototyping. For some purposes, a full simulation or emulation is not necessary. For example, to check that there are no collisions or conflicts between statically scheduled instructions, only the flow of data may be simulated or emulated. It is not necessary to compute actual values.

II. Software managed memory hierarchy

[0056] When machine learning networks are placed on edge devices, those edge devices may be adapted to deal with limited resources. For example, limited memory may be an issue because memory banks add to the size, complexity, and power consumption of those devices. This holds true especially for memory that resides on the same die as the processor itself (e.g., static random-access memory (SRAM)). Therefore, memory optimization both in hardware and software may be advantageous to execute machine learning networks on edge devices.

[0057] As discussed above, the MLA uses Tiles to execute machine learning instructions. However, sometimes the data or instructions that a Tile uses may reside originally in the off-chip memory (L3 memory). The data is brought from the off-chip memory to the on-chip memory before a Tile can use the data. Off-chip memory may be memory with variable access times (e.g., Dynamic Random-Access Memory (DRAM)).

Thus, when data is transferred from the off-chip memory, the amount of time (e.g., the number of cycles) that the transfer takes may not be determined prior to execution. Thus, operations that include data transfer with data transfer times that are unknown prior to execution may be implemented as part of a “non-deterministic” phase of execution. The non- deterministic phase instructions may be executed by a microcontroller.

[0058] An exemplary microcontroller may be an ARM M4 processor that orchestrates the off-chip memory accesses. This microcontroller may have an MLC (ML Compiler) thread running on it and the compiled code is responsible for initiating the data transfer from off- chip (e.g., DRAM) memory and subsequent notification to the respective Tiles. The compiled MLC thread is responsible for retrieving and posting data to external memory as well as any interactions with external interfaces.

[0059] In some implementations, access to off-chip memory is performed by a Direct Memory Access (DMA) controller that is managed by the microcontroller. The microcontroller communicates with the rest of MLA using one or more interfaces. The microcontroller is also responsible for generating the first ‘kickstart’ instruction (a ‘Fetch Instructions’ instruction) to each of the Tiles that consumes the data that has been received and written into the L2 memory. In addition, the microcontroller may include a core for executing instructions. The core may include a floating-point unit.

[0060] When a Tile executes an instruction, the Tile accesses the data for the instruction from the on-chip memory dedicated to the Tile (LI memory). However, it is possible for the Tile to access the data directly from L2 memory. Because the LI and L2 memories are implemented as SRAM, the length of time (e.g., number of clock cycles) used to transfer data between LI memories or between LI and L2 memories are known, so data transfer instructions for these transfers can be statically scheduled. As a result, these instructions may be included in a “deterministic phase” of instructions.

[0061] When Tiles execute an MLN, a deterministic phase of instructions may follow a non-deterministic phase of instructions and vice versa. One mechanism for bridging the non- deterministic phase of instructions and the deterministic phase of instructions includes marking addresses in on-chip memory as occupied (e.g., setting a flag) when live data has been transferred into the memory corresponding to those addresses. When a microcontroller executes an instruction to transfer data from the off-chip memory to the on-chip memory, the microcontroller may determine whether potential target addresses in the on-chip memory are storing live data (i.e., whether those addresses are occupied). The microcontroller may make this determination by checking a flag associated with each potential target address. If the flag indicates that live data is not being stored in memory corresponding to the target address (i.e., the corresponding memory is free), the microcontroller may write new data into that memory. If the flag indicates that live data is being stored in the memory corresponding to the target address (i.e., memory is occupied), the microcontroller may not write new data into that memory. When the microcontroller writes the data into memory that corresponds to the target addresses that have been marked as not storing live data (i.e., free), the microcontroller marks the target addresses as storing live data (i.e., occupied).

[0062] When the live data has been written into the on-chip memory, the live data is now ready to be consumed by the Tiles. As referred to herein, the term live data refers to data that a Tile has not yet consumed for execution (e.g., data that the Tile has not yet transferred from L2 memory into LI memory). Thus, marking target addresses after live data is transferred from the off-chip memory to the on-chip memory is one component of bridging a non-deterministic phase and a deterministic phase. Another component of bridging a non- deterministic phase and a deterministic phase involves synchronizing execution of instructions for a particular deterministic phase. [0063] A deterministic phase of instructions where instructions are statically scheduled may follow the non-deterministic phase. The deterministic phase of statically scheduled instructions is executed by the Tiles. The deterministic phase can include instructions for accessing the data stored in the on-chip memory, transferring the data between on-chip memories (e.g., between LI and L2 memory), performing computations from the MLN, and other suitable instructions. Thus, each Tile that is participating in a given deterministic phase may access (e.g., consume) the live data and perform operations using the live data.

[0064] When a Tile has finished consuming the live data stored in the on-chip memory (e.g., L2 memory), the Tile may mark the addresses where the consumed data was stored as no longer storing live data (i.e., free). For example, a Tile may transfer the data from L2 memory into LI memory for a matrix multiplication operation. When the data is transferred, the Tile may mark the addresses associated with the memory from where the data was transferred as no longer storing live data (i.e., that the data can be overwritten).

[0065] Various types of Tile instructions may be used for marking addresses in memory as no longer storing live data. In some implementations, the statically scheduled Tile instructions that access the live data stored in the on-chip memory also mark the addresses as not storing live data. For example, an instruction may be used to transfer the data from L2 memory to LI memory (e.g., a read/write instruction). The instruction may also mark the addresses where the transferred data was stored as not storing live data. Thus, the instruction to transfer data from L2 memory to LI memory may be different from, for example, an instruction to transfer data between different LI memories because the transfer instruction from L2 to LI memory performs the marking.

[0066] In some implementations, multiple statically scheduled Tile instructions access the same live data stored in the on-chip memory, and the addresses are marked as not storing live data only after the last access by said multiple Tile instructions. For example, the compiler discussed above, may determine that multiple Tiles need the data transferred into certain addresses in L2 memory. The compiler may generate a first Tile instruction for the first Tile to transfer the data into LI memory associated with the Tile. That first instruction may not mark the associated addresses in the L2 memory as not storing live data. In other words, when the data is consumed by the first Tile the addresses in L2 memory are not marked as free because a second Tile needs that data to perform MLN operations. The compiler may generate a second Tile instruction for a second Tile to transfer the data into LI memory associated with the second Tile. The second Tile instruction may mark the associated addresses as not storing live data. In other words, the second Tile instruction marks the memory as free because the live data has now been consumed by both Tiles. Thus, the compiler may schedule different instructions for transferring data from L2 to LI memory depending on whether the associated addresses in L2 memory are to be marked as no longer storing live data (i.e., free).

[0067] In some implementations, the addresses are marked as not storing live data by Tile instructions that are separate from the Tile instructions that access the live data stored in the on-chip memory. For example, the compiler may determine that a data transfer instruction is needed to transfer some live data from L2 memory to LI memory. Based on that determination, the compiler may add a new instruction after the transfer instruction to mark the associated addresses as no longer storing any live data.

[0068] When Tile instructions mark certain addresses in the on-chip memory (e.g., L2 memory) as not storing any live data, those addresses may be used by the microcontroller to transfer more data from the off-chip memory (e.g., L3 memory). Thus, when the compiler schedules these instructions the compiler partitions the instructions into deterministic and non-deterministic phases of instructions.

[0069] As will be described in more detail below, one method of optimizing execution of a machine learning network is to use a compiler that can, prior to run-time, statically schedule instructions for executing a machine learning network. For example, the compiler can statically schedule instructions including read and write memory instructions enabling execution with no conditions, branching or dependencies, which may result in lower power consumption, resource usage, and lower cost. However, because on-chip memory is limited, at least some data may be stored in off-chip memory before it is transferred to the on-chip memory for processing. Because off-chip memory has variable access time, the compiler, prior to run-time, may not be able to predict when data retrieved from the off-chip memory is ready to be processed. Thus, some instructions (or sets of instructions) may not be statically scheduled.

[0070] To address this issue, data transfer instructions or groups of instructions (e.g., instructions for read and write operations) may be split into non-deterministic phases (e.g., off-chip memory access) and deterministic phases (e.g., on-chip memory access). To transfer the data from the off-chip memory to the on-chip memory (during a non-deterministic phase of instructions), for example, a microcontroller (sometimes referred to as a controller) may determine whether certain on-chip memory is already storing data that has not been consumed yet (e.g., certain memory locations on-chip may be storing data for future consumption and should not be overwritten). Based on determining that the certain on-chip memory is not storing data that has not been consumed yet, the microcontroller may transfer the data from the off-chip memory to those locations in the on-chip memory. When the data is transferred to the on-chip memory, the target memory locations may be marked as storing data that has not been consumed yet (live data). Subsequently, the deterministic phase of instructions (e.g., transfer between memory locations on chip and data processing) may be started.

[0071] FIG. 4 illustrates an exemplary process for bridging a non-deterministic phase of instructions and a deterministic phase of instructions. At 402, a microcontroller determines whether one or more addresses in the on-chip memory are storing live data for future use by the Tiles. The microcontroller may determine whether each address is storing live data using various methods.

[0072] FIG. 5 illustrates an exemplary memory bank architecture used in determining whether a memory location is storing live data and marking a memory location for storing live data or not storing live data (action 402). Memory bank 500 is an exemplary memory bank for L2 memory that includes storage 502 for storing data. Storage 502 is split into a multitude of chunks (e.g., storing 256 kilobytes of data). Each chunk can store data and can be addressed for reading and writing data. In addition, memory bank 500 includes a storage area 504 for storing liveness indicators. Each chunk of memory in storage 502 has a corresponding liveness indicator in storage area 504. Each liveness indicator stores a value that indicates to the system whether the corresponding chunk is storing live data or is not storing live data. For example, a liveness indicator can be a Boolean where when the Boolean is true, it indicates that live data is being stored in the corresponding chunk of memory. If the Boolean is false, it indicates that live data is not being stored in the corresponding chunk of memory. The liveness indicator may also use a different system (other than Boolean) to indicate that live data is stored or that live data is not stored in the corresponding chunks. For example, a value of one may indicate that the corresponding chunk is storing live data and a value of zero may indicate that the corresponding chunk is not storing live data.

[0073] Using the architecture of FIG. 5, the microcontroller may determine whether one or more addresses in the on-chip memory are storing live data by accessing the liveness indicator for a particular address space (e.g., a 256-kilobyte chunk) and retrieving the indicator for that memory space. In some implementations, instead of using the liveness indicator, the microcontroller may have access to a memory map indicating which addresses within each memory store live data and which do not. The memory map may be a data structure that includes an indicator for each memory address, whether that memory address is storing live data. Thus, when the microcontroller attempts to determine whether a specific address in the memory stores live data, the microcontroller may query the memory map to determine if that address stores live data. In some embodiments, whenever a liveness indicator indicates that the corresponding memory is free (e.g., when a Tile sets a liveness indicator to a value of zero), a memory structure (e.g., memory bank) may transmit an interrupt to a microcontroller to inform the microcontroller that some data (e.g., corresponding to certain addresses) in that memory structure (e.g., memory bank) is no longer live. The microcontroller may maintain a listing of all available addresses.

[0074] Returning to FIG. 4, at 404, responsive to determining that the addresses are not storing any live data for future use, the microcontroller transfers the data from the off-chip memory to the addresses in the on-chip memory. For example, the microcontroller, may determine that a certain number of chunks is needed to transfer data from L3 memory. The microcontroller may identify the chunks for transfer based on whether, for example, those chunks have a liveness indicator that is false, and transfer the data over (e.g., perform a read operation on the L3 memory and write operation into L2 memory). When the data has been transferred to on-chip memory, at 406, the microcontroller marks the address as storing live data. As discussed in relation to FIG. 5, the microcontroller may access the liveness indicators associated with one or more chunks that are now storing live data and write a value of true for those chunks.

[0075] At 408, one or more Tiles may access the live data stored in the on-chip memory. The LI and L2 memories may be connected by direct data transfer paths. The one or more Tiles that need access to the live data that has been transferred to the on-chip memory (e.g., L2 memory) may access the data and transfer the data to each Tile’s respective memory (e.g., LI memory for each Tile). Upon transfer, each Tile may execute computations using the live data stored in the LI memory.

[0076] Referring to FIG. 2B, when a given Tile executes an instruction to transfer the live data from one or more addresses in the L2 memory into one or more address in the LI memory of the Tile, the Tile may access, within an L2 memory bank from where the live data was transferred, liveness indicators for each of the addresses and set each of the use- indicators to a value indicating that live data is no longer stored in the corresponding address of the memory bank. For example, the Tile may execute an instruction to set a liveness indicator for one or more chunks of data storage to a value of false. [0077] When a Tile has finished computations on a set of data, the Tile may transfer the data (whether modified or not) from LI memory back into L2 memory. This is a statically scheduled instruction that is scheduled by the compiler prior to run time. When the data has been transferred back to L2 memory from LI memory, the Tile transferring the data may mark (e.g., set a liveness indicator) a portion of the L2 memory as storing live data. For example, the Tile instruction that is generated by the compiler may include a parameter to mark one or more chunks of memory within an L2 memory bank as storing live data.

[0078] The Tile may transfer data from LI memory to L2 memory for a number of uses. For example, the Tile may transfer the data because that Tile is finished executing Tile operations and the data is transferred out because the Tile may use that data in a subsequent operation. Thus, the data may be an intermediate result. In some embodiments, the compiler may have determined prior to run-time that the data is to be stored in L3 memory because it is a result of some type of operation. Thus, the Tile transfers the data from its LI memory into a location in the L2 memory and sets the liveness indicator to True. That is, the liveness indicator indicates that the memory location is now in use. If the data is to be transferred into L3 memory (i.e., to the off-chip memory), the microcontroller may execute the instruction to transfer the data. However, the microcontroller must wait to execute the transfer instruction until the liveness indicator is set to True. When the transfer is complete, the microcontroller may set the liveness indicator for the location in the L2 memory as False. That is, the memory is now free to be overwritten.

III. Synchronization of processing elements

[0079] As described previously, when machine learning networks are placed on edge devices, those edge devices may be adapted to deal with limited resources. In another aspect, processing elements may be simplified to conserve resources, but this may give rise to a need to synchronize processing elements in their execution.

[0080] One method of optimizing execution of an MLN is to use a compiler that, prior to run-time, generates a computer program with statically scheduled instructions for executing the MLN. For example, the compiler may determine which instructions are executed by which processing elements in the MLA at what time. Static scheduling enables one or more processing elements in the MLA to execute the instructions with no run-time conditions, branching or dependencies. This may result in lower power consumption, simpler MLA design, and lower cost.

[0081] However, because on-chip memory accessible by the processing elements (which may be referred to as local memory) is limited, the statically scheduled instructions and corresponding data may be stored in off-chip memory and then transferred to the on-chip memory for execution by the processing elements. On-chip memory such as SRAM has a known access time, so data transfers from SRAM may be statically scheduled. However, SRAM typically has limited space. Off-chip memory such as DRAM has much more space available, but it has variable access time, so the compiler is not able to predict when the instructions (or corresponding data) fetched from the off-chip memory are ready for execution and then cannot statically schedule data transfers from DRAM. This issue may be compounded if instructions are to be executed by multiple processing elements according to a common static schedule because the processing elements must be synchronized in their operation. One processing element cannot start execution of its statically scheduled instructions later than another processing element, just because its instructions or data arrived later.

[0082] In one approach, to facilitate execution, the compiler may group statically scheduled instructions into phases referred to as deterministic phases. All the instructions in the deterministic phase are statically scheduled (i.e., have deterministic timing) with respect to each other. Each deterministic phase of instructions may be preceded by a corresponding non-deterministic phase. The non-deterministic phase includes instructions for fetching the statically scheduled instructions from the off-chip memory to instruction queues for the processing elements, and instructions for transferring the corresponding data from the off- chip memory to the on-chip memory. Because the compiler may not be able to predict when the non-deterministic phase is completed, it is advantageous to efficiently bridge execution of the non-deterministic phase of instructions and the deterministic phase of instructions.

[0083] FIG. 6 illustrates bridging execution of a non-deterministic phase of instructions and a deterministic phase of instructions. Although the actions of FIG. 6 are shown as following one another, in some embodiments, one or more of actions of FIG. 6 may be performed in parallel. For example, actions 602 and 604 may be executed in parallel with action 610. At 602, an MLA executes a non-deterministic phase of instructions. Those instructions may include instructions to retrieve, from the off-chip memory, both data and statically scheduled instructions. Retrieving both data and statically scheduled instructions from off-chip memory is discussed in more detail later in this disclosure.

[0084] At 604, the MLA determines that execution of the non-deterministic phase of instructions has been completed. The MLA may determine that all instructions to transfer data from the off-chip memory and instructions to fetch statically scheduled instructions from the off-chip memory have been executed, for the current non-deterministic phase. In other words, the non-deterministic phase includes (a) transferring data to be used in the deterministic phase, to memory accessible by the processing elements during the deterministic phase, and (b) transferring instructions to be executed by the processing elements during the deterministic phase to instruction queues for the processing elements. Thus, execution of the deterministic phase is dependent on completion of (a) and (b) above.

In some embodiments, the MLA transfers the data prior to transferring the statically scheduled instructions that use the data and therefore only need check that (b) is completed because that implies that (a) was also completed.

[0085] The data to be used in the deterministic phase may include parameters for the MLN, such as weights in the MLN. As the MLN is executed by the MLA, various prior deterministic phases may output intermediate values. The MLA may write those intermediate values into off-chip memory and when those intermediate values are to be used, the MLA may retrieve those values during the non-deterministic phase to be used during the deterministic phase.

[0086] At 606, the MLA triggers synchronization of a plurality of processing elements. For example, the MLA may use a flag in a register, an interrupt, or another suitable method to trigger synchronization. Thus, synchronization may be triggered based on all statically scheduled instructions being transferred into the corresponding processing elements. At 608, the MLA synchronizes the processing elements. The synchronization process may include each processing element generating and transmitting a ready signal that indicates that the processing element is ready for execution of the deterministic phase of instructions. Each ready signal may be received at a centralized location (e.g., at a predetermined processing element). When all the signals are received at the centralized location, the centralized location may transmit an acknowledgment signal indicating to each processing element that the deterministic phase is ready for execution.

[0087] In some embodiments, the Tiles may be synchronized in a different way. For example, a microcontroller sets a value of a predetermined location in memory to a number of Tiles to be synchronized. Each Tile signals that it is ready to receive data and begins to poll a flag that can be referred to as a resume flag. Each Tile signaling that it is ready to execute instructions causes the value in the predetermined location to be decremented, thus indicating the number of Tiles left to be synchronized. When the number of Tiles to be synchronized reaches zero (i.e., the value becomes zero), the resume flag is set to True. This process may be implemented in circuitry in combination with the microcontroller to initialize the circuitry. [0088] At 608, the processing elements execute the deterministic phase of instructions. For example, each processing element may execute the set of statically scheduled instructions that was loaded during the non-deterministic phase. The execution may include performing mathematical operations on the data retrieved from the off-chip memory during the non- deterministic phase. Prior to executing the statically scheduled instructions, the MLA (e.g., during the non-deterministic phase), may determine whether the data to be used during the next deterministic phase has been transferred to memory accessible by that processing element. In some embodiments, the transfer of data may be followed by transfer of instructions, as discussed above.

[0089] The deterministic phase has a known duration of execution as the compiler is able to statically schedule all the instructions in the deterministic phase. For example, based on the number and type of instructions and the static scheduling, the compiler may determine when the deterministic phase ends. Thus, in some embodiments, the MLA may start execution of a subsequent non-deterministic phase of the instructions based on the known duration of the deterministic phase. In some embodiments, the next non-deterministic phase may start prior to a previous deterministic phase ending, so that non-deterministic and deterministic phases are executing concurrently. Furthermore, the static schedule for executing the instructions in the deterministic phase is based on the known duration for executing each instruction and on a known topology of data transfer paths to and from the processing elements. Thus, execution of the instructions according to the static schedule avoids collisions or the need for arbitrations along the data transfer paths.

[0090] As discussed above, an MLA may be part of an edge device. The MLA may be executing both non-deterministic and deterministic phases of instructions.

[0091] FIG. 7 illustrates actions that the MLA may perform during a non-deterministic phase. To demonstrate the actions of a non-deterministic phase, FIG. 7 shows a Direct Memory Access controller (DMA), a microcontroller, a Tile Group, a Designated Tile and a Master Tile. It should be noted that not all components shown in FIG. 7 are necessary. For example, the actions of FIG. 7 may be performed without a microcontroller.

[0092] The microcontroller may generate and transmit a command 730 to the DMA to transfer data from the off-chip memory (e.g., L3 memory) to the on-chip memory (e.g., L2 memory). The DMA generates and transmits a command 732 to the off-chip memory to retrieve the data and receives the data transfer 734 from the off-chip memory. In response, the DMA transmits an acknowledgment 736 to the microcontroller that the data has been transferred. [0093] In some embodiments, the compiler adds one or more commands to a computer program to retrieve data from the off-chip memory. The microcontroller may send one or more commands to the DMA. The DMA may access the off-chip memory and retrieve the data from addresses in the off-chip memory. The data is then written to the on-chip memory. In some embodiments, the on-chip memory may be divided into two levels: level 1 (LI) memory and level 2 (L2) memory.

[0094] The microcontroller may be an ARM M4 processor that orchestrates the off-chip memory accesses (e.g., transmits commands to the DMA for off-chip memory access). This microcontroller may have an MLC (ML Compiler) thread running on it and the compiled code that is responsible for initiating the data transfer from off-chip memory (e.g., DRAM) and subsequent notification to the respective Tiles. The compiled MLC thread is responsible for retrieving and posting data to external memory as well as any interactions with external interfaces. The microcontroller may communicate with the rest of MLA using one or more interfaces. The microcontroller may also generate the first “kickstart” instruction (a “Fetch Instructions” instruction) to each of the Tiles that consumes the data that has been written into on-chip memory. In addition, the microcontroller may include a core for executing instructions which in turn may include a floating-point unit

[0095] The microcontroller may be external to the processing elements (e.g., Tiles) and may communicate with those processing elements (e.g., via an interface). In some embodiments, the microcontroller may execute the instructions to transfer data to be used in the deterministic phase, to memory accessible by the processing elements (e.g., on-chip memory) during the deterministic phase. For example, the microcontroller may determine, for each processing element, when the data for each processing element has been transferred to the memory accessible by a corresponding processing element, and upon that determination, the microcontroller may issue a command to each processing element to fetch instructions to be executed by that processing element.

[0096] Referring to FIG. 7, the microcontroller transmits one or more commands 738 to a plurality of processing elements (e.g., Tiles in the Tile Group) to fetch statically scheduled instructions from the off-chip memory (e.g., from dynamic random-access memory (DRAM)). For example, the microcontroller may write a corresponding fetch instruction into a designated register associated with each Tile. Each fetch instruction may include a corresponding address in the off-chip memory (e.g., DRAM) for starting to fetch the statically scheduled instructions. Each Tile reads the fetch instruction from the Tile’s corresponding register and transmits one or more fetch commands 740 to the DMA. The DMA may receive the fetch instructions and, transmit a fetch command 742 to retrieve the instructions from the off-chip memory. The DMA receives instructions 744 from the off-chip memory. In some embodiments, the transfer may be performed in bulk (e.g., the DMA may request statically scheduled instructions for multiple Tiles. The DMA may resolve contention by employing one or more various algorithms (e.g., round robin). The DMA transmits corresponding instructions to each Tile (e.g., each Tile in the Tile group).

[0097] When each Tile receives the statically scheduled instructions for the Tile, each Tile may send a ready signal 746 to a designated Tile indicating that the statically scheduled instructions have been transferred to its instruction queue and the Tile is ready to execute those instructions. Thus, determining that execution of the non-deterministic phase has completed includes determining that all processing elements have generated ready signals. In some embodiments, the processing elements may be a part of a mesh of processing elements (e.g., mesh of tiles) and determining that all processing elements have generated ready signals is performed at least in part by circuitry within the mesh. A more detailed description of generating and processing ready signals is discussed later in this disclosure in relation to FIGs. 8 and 10.

[0098] When the Designated Tile receives all the ready signals 746 from all Tiles in the Tile Group, The Designated Tile transmits the ready signal 746 to the Master Tile. In some embodiments, there are multiple groups of Tiles that will be participating in the deterministic phase. Therefore, each Tile Group transmits a ready signal to a corresponding Designated Tile and each Designated Tile transmits the ready signal to the Master Tile. When the Master Tile receives all ready signals 746, the Master Tile transmits start signals 748 to each Designated Tile. Each Designated Tile transmits starts signal 748 to each Tile within the group to start execution of the deterministic state.

[0099] Referring to the synchronization process during the non-deterministic phase,

FIG. 8 illustrates a mesh of Tiles organized into rows and columns. FIG. 8 shows rows 802a - 802j each including ten Tiles with a total of ten rows of Tiles. Each row may include a Tile 804 that has been selected as a designated tile. Thus, the mesh of Tiles in FIG. 8 includes ten designated tiles (Tiles 804a - 804j). Moreover, Tile 804a may be designated as a Master Tile. As shown in FIG. 8, each Tile in a row may include an interconnect to a corresponding designated Tile 804. Thus, when all statically scheduled instructions have been transferred to each Tile, each Tile may transmit a ready signal (e.g., ready signal discussed in relation to FIG. 1C) to a corresponding Tile 804. The Tiles that are not participating in the next deterministic may also transmit a ready signal so that each of the designated Tiles and the master Tile know when to transmit the start signal. Except the Tiles that are not participating in the next deterministic phase will not start execution based on the start signal. Each Tile 804 may transmit the ready signal to Tile 804a which may have been designated as the master Tile (e.g., as a Master Tile shown in FIG. 1C). Thus, each processing element determines that execution of the non-deterministic phase has completed with respect to that processing element and generates a ready signal upon such determination. As a result, synchronizing the processing elements includes generating a synchronization signal (i.e., a ready signal) for the processing elements, once all processing elements have generated the ready signal.

[0100] When Tile 804a receives all the expected ready signals, Tile 804a sends an acknowledgment signal to each other Tile 804 (b-j). Each Tile 804 transmits the acknowledgment signal to each Tile in a corresponding row. That is, one of the processing elements determines that all processing elements have generated corresponding ready signals and generates and transmits the synchronization signal to all the other processing elements. [0101] When each Tile receives the acknowledgment signal, the Tile begins execution of the deterministic phase. When a Tile begins execution of the deterministic phase of instructions, the Tile pulls instructions from a queue where the instructions have been written during the non-deterministic phase. When instructions are executed from the queue, other instructions cannot be written to the queue because this would break the deterministic phase of execution. Thus, it may be advantageous to have a mechanism that enables transferring instructions for executing a next deterministic phase while current instructions are being executed. Thus, the processing elements (e.g., the Tiles) are part of a mesh of processing elements and actions for determining that the non-deterministic phase has completed and actions synchronizing the processing elements are performed by circuitry within the mesh. [0102] FIG. 9 illustrates a Tile with two instruction queues for the compute pipeline 285. The components of FIG. 9 may be the same as components of FIG 2B, except the Tile of FIG. 9 now illustrates a queue 902 and a queue 904. While the Tile may be executing statically scheduled instructions in queue 902, the next set of statically scheduled instructions for the next deterministic phase may be loaded into queue 904. When all the statically scheduled instructions have been executed from queue 902, the Tile may switch (e.g., using an active queue pointer) to queue 904. The Tile may determine whether all the instructions have been loaded into queue 904 before sending a ready signal. In other words, subsequently to execution of a last instruction in a first instruction queue of a processing element (e.g., a Tile), the processing element may determine that it has finished transferring the instructions into a second instruction queue, and transmitting a synchronization signal that indicates that the processing element is ready for executing the instructions in the second instruction queue. [0103] In some embodiments, a processing element (e.g., a Tile) may concurrently (a) execute the deterministic phase of instructions from one of the instruction queues; and (b) execute a subsequent non-deterministic phase of fetching instructions into the other instruction queue for execution in a subsequent deterministic phase.

[0104] In some embodiments, the synchronization process may be performed in a different manner. FIG. 10 illustrates an alternative process for synchronizing Tiles for execution of a deterministic phase. In the embodiments of FIG. 10, the microcontroller that is external to the processing elements, determines at least in part, that the non-deterministic phase has completed. A microcontroller may, as illustrated by arrows 1002 and 1008, transfer data and statically scheduled instructions (e.g., using the DMA) from the off-chip memory (e.g., L3 memory described in relation to FIGs. 2A and 2B) to the on-chip memory (e.g., L2 memory described in relation to FIGs. 2A and 2B). Arrows 1006 and 1008 are bi directional, at least, because the microcontroller may transfer processed or intermediate result data to the off-chip memory. The microcontroller may have other bidirectional communication with off-chip memory and the on-chip memory. When the data and statically scheduled instructions are transferred into L2 memory, the microcontroller may, as indicated by arrow 1004, send a signal to each Tile that is participating in a given deterministic phase a ready signal indicating that the data and statically scheduled instructions are ready for the deterministic phase.

[0105] When each Tile receives the ready signal, the Tile may, as indicated by arrow 1006, access a location within the L2 memory that includes the statically scheduled instructions. The Tile may begin executing the statically scheduled instructions from within the L2 memory by, for example, transferring each instruction into a predetermined location in the LI memory. The Tile may also transfer the data for executing the instruction from L2 memory into LI memory and begin executing the instruction. One or more Tiles may also transfer result data and/or intermediate data to L2 memory as illustrated by arrow 1006 being bi-directional. In some embodiments, the Tile may transfer a number of instructions (e.g., thirty -two instructions) from the L2 memory into a queue within the Tile and execute those instructions. Thus, the microcontroller may synchronize execution of each deterministic phase. IV. MLA system and use

[0106] The embodiments described above may be used to execute a single MLN. However, the MLA is enabled to execute multiple MLNs. Figures 11 A and 1 IB illustrate partitioning the mesh of Tiles to different subnets of an MLN and to different MLNs. In FIG. 11 A, the MLA includes a 6x6 mesh (element 1180 in FIG. 11 A). From time tO to tl, the mesh 1180 is utilized to implement two different MLNs: MLN A and MLN B. The Tiles are divided into three partitions 1182A, 1182B 1 , and 1182B2. Partition 1182A implements MLN A, partition 1182B1 implements the first 10 layers of MLN B, and partition 1182B2 implements the remaining 15 layers of MLN B. MLN B may be partitioned in this manner because some off-Tile operations may be required between layers 10 and 11. Maybe the output of layer 10 requires a computation that is performed off-Tile in a non-deterministic manner, or maybe layers 11-25 require data that cannot be loaded in a manner consistent with the static scheduling of layers 1-10. After time tl, the mesh 1180 continues to implement MLN B using partition 582A, but MLN A is replaced by MLNs C and D using partitions 1182C and 1182D, respectively.

[0107] FIG. 11 A shows a progression over time. The front diagram shows the partitioning at an earlier time and the subsequent diagrams show the partitioning at later times. The times are indicated to the lower right of the diagrams. At time tO, the mesh is partitioned so that the bottom 2x6 Tiles implement MLN A, the upper left 4x3 Tiles implement MLN B layers 1-10, and the upper right 4x3 Tiles implement MLN B layers 11- 25. At time tl, MLN B is no longer required and is replaced by MLNs C and D. 11-25. The upper left 4x2 Tiles now implement MLN C, and the upper right 4x4 Tiles now implement MLN D.

[0108] Note that each of these partitions may run deterministic and non-deterministic phases separately from each other. Partition 1182A implements MLN A, which is independent of MLN B implemented by partitions 1182B1 and 1182B2. Thus, partition 1182A may run separately from the other two partitions. At time tl, partition 1182A may continue to run, unaffected by the change from MLN B to MLNs C and D.

[0109] FIG. 11B illustrates deterministic and non-deterministic phases for partitions 1182B1 and ll82B2. Each row represents different phases of instructions. The white regions of each row denote the execution of instructions and the hashed regions indicate idling. Non-deterministic phases are indicated by breaks in the rectangles. From top to bottom, the rows are the following. The top row has instructions to load data for the computations of layers 1-10 from DRAM into the MLA. This data will be consumed by Tile partition 1182B1. Referring to FIG. 2A, this is performed by the controller 277 and these phases 1155 are non-deterministic because they are loads from DRAM. The second row has the deterministic phases 1156 of statically scheduled Tile instructions that implement the computations for layers 1-10. Similarly, the fourth row has non-deterministic phases 1158 for loading data for the computation of layers 11-25, and the bottom row has deterministic phases 1159 of statically scheduled Tile instructions that implement these computations, respectively. The middle row has other non-deterministic instructions 1157. In this example, this includes non-deterministic computations 1157a-c that occur outside the MLA, and instructions 1157d to repartition the MLA.

[0110] The suffixes indicate different input samples. The phases that end in -a apply the MLN to one input sample, the phases that end in -b apply the MLN to the next input sample, etc. The arrows indicate dependencies. Consider first input sample a. A controller loads 1155a the relevant data (input values, weights, biases, operator parameters) from DRAM into the MLA memory. After this is completed, the Tiles 1182B1 may perform their computations 1156a using this data. The Tile output is transferred off-chip for a computation 1157a that is not performed by the Tiles. In the meantime, the controller loads 558a the relevant data for layers 11-25. When data from both non-deterministic phases 1157a and 1158a are available, Tile partition 1182B2 performs its computations 1159a. The Tile computations within each phase 1156a and 1159a are statically scheduled within their respective non-deterministic phases, but the time between phases 1156a and 1159a may vary. The processing of input samples b and c have the same dependencies and general flow.

[0111] At some point (time tl in FIG. 11 A), the controller ends execution of MLN B and starts execution of MLNs C and D. The compiler has provided different schedules with different partitions of the Tiles as partition 1182C for MLN C and partitions 1182D for MLN D. This is phase 1157d. The process then continues with each of the active partitions 1182A, C, D execution instructions to implement their respective MLNs.

[0112] The approach based on static scheduling described above is not restricted to the examples described above. For example, different network topologies of Tiles may be used. [0113] In some examples, the Tiles in the mesh all had the same capability. Each Tile had a compute pipeline that could perform matrix multiplication and implement certain operators. Each Tile also had a data transfer pipeline to transfer data to and from its local LI memory.

[0114] As discussed above, the MLA includes various components that are on the same die. The ML A may be integrated into a larger integrated circuit product (e.g., as part of an edge device). FIG. 12 is a block diagram of an integrated circuit that includes an MLA 1270. In other words, other components may be included on the same die as the MLA. This example includes the following additional blocks: application processor 1210 (e.g., general purpose CPU running applications), computer vision processor 1212 (or other types of application-specific processors), safety 1214, security 1216, additional SRAM (memory)

1220 and input/output circuitry 1222. It also includes a network 1230 for communication between the different components. This type of semiconductor chip may be referred to as a system-on-chip (SoC).

[0115] The connections to the external world include camera inputs 1240 for the computer vision processors, ports for debug 1242 and configuration 1244, a connection 1246 to external memory (e.g., DRAM), chip-to-chip connections 1248, and network connections 1250 (e.g., Ethernet and PCIe).

[0116] The SoC of FIG. 12 may be combined with other components to perform various tasks in edge devices. Example applications for edge devices include automotive and other forms of transportation including autonomous transportation, agricultural, industrial, robotics, drones, surveillance and security, smart environments including smart cities, medical and personalized health. Example tasks include computer vision, image analysis, image understanding, speech recognition, audio analysis, audio understanding, natural language processing, classification and pahem recognition tasks. For edge devices, it may be desirable to perform certain tasks in real-time.

[0117] In addition to memory and other programmable processors, an edge device may also include sensors, such as cameras (both still image and video cameras), microphones, temperature sensors, pressure sensors and other types of sensors. The sensors may capture samples that are used as inputs to a computing pipeline within the edge device. For example, image samples may be input to the computer vision processors 1212, which perform initial operations such as edge detection and enhancement, contrast enhancement, motion detection, and optical flow. Raw and/or processed images may be then input to the MLA 1270 for analysis by the machine learning network. The MLA may also receive other inputs, such as metadata from other sources and data from other sensors. The application processors 1210 may also perform various functions in the overall pipeline and may also serve as a master controller that coordinates operation of the MLA and the other programmable processors in the pipeline.

[0118] Edge devices may be portable with less power available for computations compared to, for example, cloud-based server farms. It may also be desirable for the computing pipeline within the edge device to perform tasks without utilizing cloud-based or other remote compute resources. In some implementations, the MLA implements computations in the machine learning network at a speed of at least 50 TOPs (50 trillion operations per second) at a power consumption of not more than 5 watts. The speed may be increased by increasing the number of Tiles in the mesh or the number of Tile meshes on the die.

[0119] Although the detailed description contains many specifics, these should not be construed as limiting the scope of the invention but merely as illustrating different examples. It should be appreciated that the scope of the disclosure includes other embodiments not discussed in detail above. Various other modifications, changes and variations which will be apparent to those skilled in the art may be made in the arrangement, operation and details of the method and apparatus disclosed herein without departing from the spirit and scope as defined in the appended claims. Therefore, the scope of the invention should be determined by the appended claims and their legal equivalents.

[0120] In the claims, reference to an element in the singular is not intended to mean “one and only one” unless explicitly stated, but rather is meant to mean “one or more.” In addition, it is not necessary for a device or method to address every problem that is solvable by different embodiments of the invention in order to be encompassed by the claims.

Claims

WHAT IS CLAIMED IS:

1. A method for implementing a machine learning network (MLN) by executing a computer program of instructions on a machine learning accelerator (MLA), the MLA comprising a plurality of Tiles and an on-chip memory implemented on a semiconductor die, wherein Tiles in the plurality of tiles are configured to execute Tile instructions that access the on-chip memory, and wherein the Tiles are organized into one or more meshes of interconnected Tiles, the method comprising: executing a non-deterministic phase of instructions that transfer data from an off-chip memory to addresses in the on-chip memory by: determining whether the addresses in the on-chip memory are storing live data for future use by the Tiles; and responsive to determining that the addresses are not storing any live data for future use, transferring the data from the off-chip memory to the addresses in the on- chip memory and marking the addresses as storing live data; and the Tiles executing a deterministic phase of statically scheduled Tile instructions that implement computations from the MLN, the statically scheduled Tile instructions accessing the live data stored in the on-chip memory.

2. The method of claim 1, further comprising, subsequently to accessing the live data stored in the on-chip memory, marking the addresses as not storing live data.

3. The method of claim 2, wherein the statically scheduled Tile instructions that access the live data stored in the on-chip memory also mark the addresses as not storing live data.

4. The method of claim 2, wherein multiple statically scheduled Tile instructions access the same live data stored in the on-chip memory, and the addresses are marked as not storing live data only after the last access by said multiple Tile instructions.

5. The method of claim 4, wherein a compiler converts the machine learning network to the computer program of instructions, including statically scheduling an order of execution of the multiple Tile instructions.

6. The method of claim 2, wherein the addresses are marked as not storing live data by Tile instructions separate from the Tile instructions that access the live data stored in the on- chip memory.

7. The method of claim 6, wherein a compiler converts the machine learning network to the computer program of instructions, including statically scheduling an order of execution of the Tile instructions that access the live data and the Tile instructions that mark the addresses as not storing live data.

8. The method of claim 1, wherein the on-chip memory is a multi-level memory system that includes level 1 (LI) memory, level 2 (L2) memory that includes for each address space in the L2 memory a flag indicating whether that address space is storing live data, and direct data transfer paths between the LI memory and the L2 memory; and wherein the Tiles execute computations using data sets stored in the LI memory, further comprising: during the non-deterministic phase, transferring the live data from the off-chip memory to the L2 memory; and during the deterministic phase, transferring the live data from the L2 memory to the LI memory and executing computations using the live data stored in the LI memory.

9. The method of claim 8, wherein transferring the live data from the L2 memory to the LI memory comprises: executing, using a Tile an instruction to transfer the live data from one or more addresses in the L2 memory into one or more addresses in the LI memory associated with the Tile; accessing, within an L2 memory bank from where the live data was transferred, a plurality of liveness indicators corresponding to the addresses from where the live data was transferred; and setting each of the plurality of liveness indicators to a value indicating that live data is not stored in the corresponding addresses of the memory bank.

10. The method of claim 1, further comprising: a compiler converting the machine learning network to the computer program of instructions, including the non-deterministic phase of instructions and the deterministic phase of statically scheduled Tile instructions.

11. The method of claim 1 , further comprising: subsequently to executing the statically scheduled Tile instructions transferring, using the Tiles, processed live data from a first on-chip memory to a second on-chip memory; and marking one or more addresses in the second on-chip memory as storing processed live data.

12. The method of claim 1, wherein the non-deterministic phase of instructions is executed by a controller.

13. The method of claim 1, wherein marking the addresses as storing live data comprises: setting a corresponding live data flag for each memory bank represented by the addresses to indicate live data in each memory bank; or setting, in a data structure representing a memory space containing the addresses, each memory location representing the addresses to indicate live data.

14. The method of claim 1, wherein execution of the deterministic phase of statically scheduled Tile instructions starts subsequently to marking the addresses as storing live data.

15. The method of claim 1, wherein the off-chip memory that is not on the same die as the on-chip memory system, and wherein the Tile instructions include instructions for data transfers within the on-chip memory system and do not include instructions for data transfers to and from the off-chip memory system, and a compiler statically schedules the Tile data transfer instructions and does not statically schedule the instructions for data transfers to and from the off-chip memory system.

16. The method of claim 1 wherein the on-chip memory system comprises static random- access memory (SRAM) and the off-chip memory comprises dynamic random-access memory (DRAM).

17. The method of claim 1, wherein the on-chip memory is a multi-level memory system that includes level 1 (LI) memory, level 2 (L2) memory that includes for each address space in the L2 memory a flag indicating whether that address space is storing live data, and direct data transfer paths between the LI memory and the L2 memory; and wherein the Tiles execute computations using data sets stored in the LI memory, further comprising: during the deterministic phase, transferring, using a Tile, new data from the LI memory to the L2 memory; and during the non-deterministic phase, transferring, using the microcontroller, the new data from the L2 memory to the off-chip memory.

18. A method for implementing a machine learning network (MLN) by executing a computer program of instructions on a machine learning accelerator (MLA) comprising a plurality of interconnected processing elements, the instructions partitioned into one or more non-deterministic phases and one or more deterministic phases, the method comprising: executing a non-deterministic phase of the instructions; determining that execution of the non-deterministic phase has completed; subject to the determination that the execution of the non-deterministic phase has completed, executing a deterministic phase of the instructions, wherein execution of the deterministic phase is dependent on completion of the non- deterministic phase, the instructions in the deterministic phase are executed by a plurality of processing elements according to a static schedule; and prior to executing the deterministic phase synchronizing the plurality of processing elements upon completion of the non-deterministic phase for execution of the statically scheduled instructions.

19. The method of claim 18, wherein: the non-deterministic phase comprises (a) transferring data to be used in the deterministic phase, to memory accessible by the processing elements during the deterministic phase, and (b) transferring instructions to be executed by the processing elements during the deterministic phase, to instruction queues for the processing elements; and execution of the deterministic phase is dependent on completion of (a) and (b) above.

20. The method of claim 19, wherein the data to be used in the deterministic phase comprises parameters for the machine learning network.

21. The method of claim 19, wherein the data to be used comprises intermediate values for the machine learning network computed during prior deterministic phases.

22. The method of claim 19, wherein step (a) of transferring data occurs before step (b) of transferring instructions that use the data.

23. The method of claim 18, wherein determining that the non-deterministic phase has completed comprises: determining that data to be used in the deterministic phase has been transferred to memory accessible by the processing elements during the deterministic phase.

24. The method of claim 18, wherein determining that the non-deterministic phase has completed comprises: determining that the instructions to be executed by the processing elements during the deterministic phase have been transferred to instruction queues for the processing elements.

25. The method of claim 24, wherein determining that the instructions to be executed by the processing elements during the deterministic phase have been transferred to instruction queues for the processing elements comprises: each processing element determining whether its instructions have been transferred to its instruction queue.

26. The method of claim 18, wherein the non-deterministic phase comprises: a controller executing instructions to transfer data to be used in the deterministic phase, to memory accessible by the processing elements during the deterministic phase, wherein the controller is external to the processing elements; the controller determining, for each processing element, when the data for that processing element has been transferred to memory accessible by that processing element; and upon said determination, the controller issuing a command to that processing element to fetch instructions to be executed by that processing element.

27. The method of claim 26, wherein: the non-deterministic phase further comprises each processing element determining whether its instructions have been transferred to its instruction queue; and each processing element generating a ready signal that its instructions have been transferred to its instruction queue; and determining that execution of the non-deterministic phase has completed comprises determining that all processing elements have generated ready signals.

28. The method of claim 27, wherein the processing elements are part of a mesh of processing elements, and determining that all processing elements have generated ready signals is performed at least in part by circuitry within the mesh.

29. The method of claim 18, wherein determining that the non-deterministic phase has completed is performed at least in part by a controller external to the processing elements.

30. The method of claim 18, wherein the processing elements are part of a mesh of processing elements, and determining that the non-deterministic phase has completed and synchronizing the processing elements are performed by circuitry within the mesh.

31. The method of claim 18, wherein: each processing element determines that execution of the non-deterministic phase has completed with respect to that processing element and generates a ready signal upon such determination; and synchronizing the processing elements comprises generating a synchronization signal for the processing elements, once all processing elements have generated the ready signal.

32. The method of claim 31, wherein one of the processing elements determines that all processing elements have generated the ready signal, and generates and transmits the synchronization signal to all of the other processing elements.

33. The method of claim 18, wherein executing the deterministic phase of statically scheduled instructions has a known duration, the method further comprising: starting execution of a subsequent non-deterministic phase of the instructions, based on the known duration of the deterministic phase.

34. The method of claim 18, wherein the processing elements have pairs of instruction queues, the method further comprising: concurrently (a) executing the deterministic phase of instructions from one of the instruction queues; and (b) executing a subsequent non-deterministic phase of fetching instructions into the other instruction queue for execution in a subsequent deterministic phase.

35. The method of claim 18, wherein the processing elements have pairs of instruction queues, the method further comprising: subsequently to execution of a last instruction in a first instruction queue of a processing element, determining that the processing element has finished transferring the instructions into a second instruction queue; and transmitting a synchronization signal that indicates that the processing element is ready for executing the instructions in the second instruction queue.

36. The method of claim 18, wherein: executing each of the statically scheduled instructions in the deterministic phase has a known duration; and the static schedule for executing the instructions in the deterministic phase is based on the known duration for executing each instruction and on a known topology of data transfer paths to and from the processing elements, wherein execution of the instructions according to the static schedule avoids arbitrations and collisions along the data transfer paths.

37. The method of claim 18, wherein: the MLA comprises a local memory implemented on a same semiconductor die as the processing elements; the deterministic phase comprises instructions executed by the processing elements to transfer data to and from the local memory; and the non-deterministic phase comprises instructions to transfer data to the local memory from locations not accessible by the processing elements during the deterministic phase.

38. The method of claim 18, wherein the processing elements execute instructions during the deterministic phase to perform matrix multiplication.

39. A non-transitory computer-readable storage medium storing executable computer program instructions for executing any of the methods of claims 1-38.

40. A computer system comprising: a storage medium for receiving and storing instructions; and a processor system having access to the storage medium and executing the instructions to execute any of the methods of claims 1-38.