JP7525101B2 - Systems, methods and apparatus for heterogeneous computing - Google Patents

Description

The present disclosure relates generally to the field of computing devices, and more specifically to heterogeneous computing methods, devices and systems.

In modern computers, the CPU performs general-purpose computing tasks, such as running application software and operating systems. Specialized computing tasks, such as graphics and image processing, are handled by graphics processors, image processors, digital signal processors, and fixed function accelerators. In modern heterogeneous machines, each type of processor is programmed in a different way.

The era of big data processing demands higher performance at lower energy compared to today's general-purpose processors. Accelerators (e.g., either custom fixed-function units or bespoke programmable units) help meet these demands. The field is undergoing rapid evolution in both algorithms and workloads, and the set of available accelerators is difficult to predict in advance and will most likely evolve with product types, branching out across stock units within product types.

The embodiments will be readily understood from the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. The embodiments are illustrated by way of example and are not intended to be limiting to the figures of the accompanying drawings.

It represents a heterogeneous multiprocessing execution environment.

It represents a heterogeneous multiprocessing execution environment.

1 illustrates an exemplary implementation of a heterogeneous scheduler.

1 illustrates an embodiment of system boot and device discovery for a computer system.

1 shows an example of thread migration based on a mapping of program phases to three types of processing elements.

1 illustrates an exemplary implementation flow performed by a heterogeneous scheduler.

1 illustrates an example of a method for thread destination selection by a heterogeneous scheduler.

1 illustrates the concept of using stripe mapping for logical IDs.

13 shows an example of the use of stripe mapping for logical IDs.

An example of a core group is shown below.

An example is given of a method for thread execution in a system that utilizes a binary translator switching mechanism.

1 illustrates an exemplary method for hot code to accelerator core allocation.

1 illustrates an example method for potential core allocation for wakeup or write on page directory base register events.

1 shows an example of a serial phase thread.

1 illustrates an example method for potential core allocation for thread response to a sleep command event.

1 illustrates an example method for potential core allocation for threads in response to phase change events.

Here is an example of code that describes an acceleration region.

1 illustrates an embodiment of a method for execution with ABEGIN in a hardware processor core.

1 illustrates an embodiment of a method for implementation with AEND in a hardware processor core.

We present a system that uses pattern matching to provide ABEGIN/AEND equivalence.

1 illustrates an embodiment of a method for non-accelerated description threads subjected to pattern recognition.

1 illustrates an embodiment of a method for non-accelerated description threads subjected to pattern recognition.

We present various types of memory dependencies, their semantics, ordering requirements and use cases.

1 shows an example of a memory data block pointed to by an ABEGIN instruction.

25 shows an example of a memory 2503 that is configured to use ABEGIN/AEND semantics.

Illustrates how to operate in different modes for implementation using ABEGIN/AEND.

Illustrates how to operate in different modes for implementation using ABEGIN/AEND.

1 provides additional details regarding one embodiment.

1 illustrates an embodiment of an accelerator.

1 illustrates a computer system including an accelerator and one or more computer processor chips coupled to a processor via a multi-protocol link.

4 illustrates a device bias flow according to an embodiment.

4 illustrates an exemplary process according to one embodiment.

1 illustrates the process when an operand is released from one or more I/O devices.

An example is shown using two different types of work queues.

1 illustrates an embodiment of a Data Streaming Accelerator (DSA) device having multiple work queues that RECEIVE descriptors submitted via an I/O fabric interface.

Two work queues are shown.

13 illustrates another configuration using engines and grouping.

An example of a descriptor is given below.

13 shows an example of a completion record.

1 illustrates an exemplary no-op descriptor and no-op completion record.

1 shows an exemplary batch descriptor and no-op completion record.

1 illustrates an exemplary drain descriptor and drain completion record.

1 illustrates an exemplary memory migration descriptor and memory migration completion record.

1 illustrates an exemplary fill descriptor.

1 illustrates an exemplary comparison descriptor and comparison completion record.

13 illustrates an exemplary comparison intermediate descriptor.

13 illustrates an exemplary production data record descriptor and a production delta record completion record.

This shows the format of the differential recording.

1 illustrates an exemplary adaptive differential record descriptor.

13 illustrates one embodiment of the use of the create difference record and adapt difference record operations.

13 illustrates an exemplary dual-cast memory copy descriptor and dual-cast memory copy completion record.

1 illustrates an exemplary CRC generation descriptor and CRC generation.

13 shows an example copy with a CRC generated descriptor.

1 illustrates an exemplary DIF insertion descriptor and a DIF insertion completion record.

1 illustrates an exemplary DIF strip descriptor and a DIF strip completion record.

1 illustrates an exemplary DIF update descriptor and a DIF update completion record.

1 illustrates an exemplary cache flush descriptor.

This indicates 64 bytes of enqueue store data generated by ENQCMD.

1 illustrates an embodiment of a processor-performed method for processing a MOVDIRI instruction.

1 illustrates an embodiment of a method performed by a processor to process a MOVDIRI64B instruction.

1 illustrates an embodiment of a method performed by a processor to process an ENCQMD instruction.

The format for the ENQCMDS command is shown below.

1 illustrates an embodiment of a method performed by a processor to process an ENCQMD instruction.

1 illustrates an embodiment of a method performed by a processor to process a UMONITOR instruction.

1 illustrates an embodiment of a processor-performed method for processing a UMWAIT instruction.

1 illustrates an embodiment of a method performed by a processor to process a TPAUSE instruction.

An example of execution using the UMWAIT and UMONITOR instructions is given below.

An example of execution using the TPAUSE and UMONITOR instructions is given below.

1 illustrates an exemplary implementation in which an accelerator is communicatively coupled to multiple cores through a cache coherent interface.

2 illustrates another diagram of an accelerator including a data management unit, multiple processing elements, and high-speed on-chip storage, as well as other components previously mentioned.

4 illustrates an exemplary set of operations performed by a processing element.

1 illustrates an example of sparse matrix multiplication for vector x to produce vector y.

4 shows a CSR representation of a matrix A where each value is stored as a (value, row index) pair.

We denote the CSC representation of matrix A using (value, column index) pairs.

The pseudocode for the calculation pattern is shown below. The pseudocode for the calculation pattern is shown below. The pseudocode for the calculation pattern is shown below.

1 illustrates a process flow for one embodiment of a data management unit and processing elements.

The paths for the spMspV_csc and scale_update operations are highlighted (with dotted lines).

1 shows the path for spMdV_csr operation.

Here is an example of representing a graph as an adjacency matrix: Here is an example of representing a graph as an adjacency matrix:

The vertex program is shown below.

1 shows exemplary program code for executing a vertex program.

1 shows the formulation of GSPMV.

The framework is presented.

It is shown that a customizable logic block is provided within each PE.

The processing of each accelerator tile is shown.

11 summarizes customizable parameters for one embodiment of a template.

Tuning considerations are presented.

One of the most common sparse matrix formats is shown below.

1 shows steps for an embodiment of sparse matrix-dense vector multiplication using the CRS data format.

An embodiment is shown for an accelerator that includes an accelerator logic die and one or more stacks of DRAM.

1 illustrates one embodiment of an accelerator logic chip from a top perspective facing a stack of DRAM dies. 1 illustrates one embodiment of an accelerator logic chip from a top perspective facing a stack of DRAM dies.

1 provides a high level overview of the DPE.

1 shows an example of a blocking scheme.

Indicates the block descriptor.

Shows a two-row matrix that fits into the buffer of a single dot product engine.

An embodiment of the hardware in a dot product engine using this format is shown.

Indicates the contents of the match logical unit to be captured.

13 provides details of a dot product engine design supporting sparse matrix-vector multiplication according to an embodiment.

Here is an example using specific values:

We show how sparse-dense and sparse-sparse dot product engines can be combined to yield a dot product engine that can handle both types of calculations.

An implementation of socket swapping with 12 accelerator stacks is shown.

A multi-chip package (MCP) implementation using a set of processors/cores and eight stacks is shown.

Shows the accelerator stack.

1 shows a potential layout of an accelerator intended to sit underneath a WIO3 DRAM stack, including 64 dot-product engines, eight vector caches, and an integrated memory controller.

Seven DRAM technologies are compared.

1 shows a stacked DRAM. 1 shows a stacked DRAM.

1 shows a breadth-first search (BFS) listing.

1 illustrates the format of a descriptor used to define a lambda function according to one embodiment.

4 shows the lower 6 bytes of a header word in the embodiment.

A matrix value buffer, a matrix index buffer, and a vector value buffer are shown.

1 illustrates details of one embodiment of a lambda data path.

1 shows an example of instruction encoding.

Indicates the encoding for a particular set of instructions.

1 illustrates an encoding of an exemplary comparison predicate.

13 shows an embodiment using a bias.

1 illustrates a memory-mapped I/O (MMIO) space register for use with a work-queue based implementation. 1 illustrates a memory-mapped I/O (MMIO) space register for use with a work-queue based implementation.

Here is an example of matrix multiplication:

1 shows octoMADD instruction processing using a binary tree reduction network.

1 illustrates an embodiment of a method performed by a processor to process a multiply-accumulate instruction.

1 illustrates an embodiment of a method performed by a processor to process a multiply-accumulate instruction.

1 illustrates exemplary hardware for executing the MADD instruction. 1 illustrates exemplary hardware for executing the MADD instruction. 1 illustrates exemplary hardware for executing the MADD instruction.

1 illustrates an example of a hardware heterogeneous scheduler circuit and its interaction with memory.

1 illustrates an example of a software heterogeneous scheduler.

1 illustrates an embodiment of a method for post-system boot device discovery.

An example of movement for threads in shared memory is shown.

1 illustrates an example method for thread migration that may be performed by a heterogeneous scheduler.

1 is a block diagram of a processor that represents an abstract execution environment as described in detail above.

FIG. 1 is a simplified block diagram illustrating an exemplary multi-chip configuration.

1 shows a block diagram depicting at least a portion of a system including an example implementation of a multi-chip link (MCL).

1 illustrates a block diagram of an exemplary logical PHY of an exemplary MCL.

1 illustrates a simplified block diagram showing another representation of the logic used to implement an MCL.

Here is an example of execution when ABEGIN/AEND is not supported.

FIG. 2 is a block diagram of a register architecture according to one embodiment of the present invention;

2 is a block diagram illustrating both an exemplary in-order pipeline and an exemplary register renaming out-of-order issue/execution pipeline according to an embodiment of the present invention.

1 is a block diagram illustrating both an exemplary embodiment of an in-order architecture core and an exemplary register renaming, out-of-order issue/execution architecture core included in a processor according to an embodiment of the present invention;

A block diagram of a more specific exemplary in-order core architecture is shown, where a core may be one of several logic blocks (including other cores of the same type and/or different types) within a chip. A block diagram of a more specific exemplary in-order core architecture is shown, where a core may be one of several logic blocks (including other cores of the same type and/or different types) within a chip.

FIG. 2 is a block diagram of a processor that may have more than one core, may have an integrated memory controller, and may have integrated graphics according to an embodiment of the present invention.

1 shows a block diagram of a system according to an embodiment of the present invention.

FIG. 2 is a block diagram of a first more specific exemplary system according to an embodiment of the present invention.

FIG. 2 is a block diagram of a second, more specific, exemplary system according to an embodiment of the present invention.

FIG. 2 is a block diagram of a SoC according to an embodiment of the present invention.

1 is a block diagram contrasting the use of a software instruction converter to convert binary instructions in a source instruction set to binary instructions in a target instruction set according to an embodiment of the present invention.

In the following detailed description, reference is made to the accompanying drawings which form a part hereof, where like numerals refer to like parts throughout and illustrate exemplary embodiments which may be practiced. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense, and the scope of the embodiments is defined by the appended claims and their equivalents.

Various operations may be described sequentially as multiple separate actions or processes in a manner that is most helpful in understanding the claimed subject matter. However, the order of description should not be construed as implying that these processes are necessarily order dependent. In particular, these processes need not be performed in the order presented. Processes described may be performed in a different order than in the described embodiment. In additional embodiments, various additional processes may be performed and/or processes described may be omitted.

For purposes of this disclosure, the term "A and/or B" means (A), (B) or (A and B). For purposes of this disclosure, the term "A, B and/or C" means (A), (B), (C), (A and B), (A and C), (B and C) or (A, B and C).

In the description, the terms "in one embodiment" or "in embodiments" may be used and may refer to one or more of the same or different embodiments, respectively. Additionally, terms such as "comprising," "including," and "having" are synonymous when used in reference to embodiments of the present disclosure.

As described in the Background, with a wide range of stock units and platforms that implement a diverse mix of accelerators, it can be difficult to deploy an accelerator solution and manage the complexity of using accelerators in a portable manner. Furthermore, given the large number of operating systems (and versions, patches, etc.), deploying accelerators using a device driver model has limitations including hurdles to adoption due to developer efforts, non-portability, and strict performance requirements for big data processing. Accelerators are typically hardware devices (circuitry) that perform functions more efficiently than software running on a general-purpose processor. For example, hardware accelerators can be used to improve the execution of a particular algorithm/task (e.g., video encoding or decoding, a particular hash function, etc.) or a class of algorithms/tasks (e.g., machine learning, sparse data manipulation, cryptography, graphics, physics, regular expressions, packet processing, artificial intelligence, digital signal processing, etc.). Examples of accelerators include, but are not limited to, graphics processing units ("GPUs"), fixed function field programmable gate array ("FPGA") accelerators, and fixed function application specific integrated circuits ("ASICs"). Note that in some embodiments, the accelerator may be a general purpose central processing unit ("CPU") where the CPU is more efficient than other processors in the system.

The power budget of a given system (e.g., a system-on-chip ("SoC"), processor stock unit, rack, etc.) may be consumed by processing elements on only a portion of the available silicon area. This makes it advantageous to build various specialized hardware blocks that reduce energy consumption for specific processes, even though not all of the hardware blocks may be active at the same time.

Embodiments of systems, methods, and apparatus for selecting a processing element (e.g., a core or accelerator) to process a thread, interfacing with the processing element, and/or managing power consumption in a heterogeneous multiprocessor environment are described in detail. For example, in various embodiments, the heterogeneous multiprocessor is configured (e.g., by design or by software) to dynamically migrate threads between different types of processing elements of the heterogeneous multiprocessor based on characteristics of the threads and/or corresponding workloads of the processing elements, provide program interfaces to one or more of the processing elements, translate code for execution on a particular processing element, and select a communication protocol to use with a selected processing element based on characteristics of the workload and the selected processing element or combinations thereof.

In a first aspect, the workload dispatch interface, i.e., the heterogeneous scheduler, presents a homogeneous multiprocessor programming model to the system programmer. In particular, this aspect may enable a programmer to develop software or an equivalent abstraction that targets a specific architecture, while facilitating continuous improvements to the underlying hardware without requiring corresponding changes to the software being developed.

In a second aspect, the multi-protocol link allows a first entity (such as a heterogeneous scheduler) to communicate with multiple devices using a protocol associated with the communication. This replaces the need to have separate links for device communication. In particular, the link has three or more protocols that are dynamically multiplexed on the link. For example, a common link supports protocols consisting of: 1) a producer/consumer, discovery, configuration, interrupt (PDCI) protocol that enables device discovery, device configuration, error reporting, interrupts, DMA-style data transfers, and various services, as may be defined in one or more proprietary or industry standards (such as the PCI Express specification or equivalent alternatives); 2) a caching agent coherence (CAC) protocol that enables devices to issue coherent read and write requests to processing elements; and 3) a memory access (MA) protocol that enables a processing element to access the local memory of another processing element.

In a third aspect, thread scheduling, migration, or emulation, or portions thereof, are performed based on the phase of the thread. For example, a data-parallel phase of a thread is typically scheduled or migrated to a SIMD core, a thread-parallel phase of a thread is typically scheduled or migrated to one or more scalar cores, and a serial phase is typically scheduled or migrated to an out-of-order core. Each core type minimizes either energy or latency, both of which are considered for thread scheduling, migration, or emulation. Emulation may be used when scheduling or migration is not possible or advantageous.

In a fourth aspect, a thread or a portion thereof is opportunistically offloaded to an accelerator. In particular, the accelerator start (ABEGIN) and accelerator end (AEND) instructions of the thread, or portions thereof, the bookend instructions, may be executable on the accelerator. If the accelerator is not available, then the instructions between the ABEGIN and AEND instructions are executed normally. However, if an accelerator is available, it is preferable to use the accelerator (e.g., to use less power), then the instructions between the ABEGIN and AEND instructions are converted to execute on that accelerator and scheduled for execution on that accelerator. As a result, the use of the accelerator is opportunistic.

In a fifth aspect, a thread or a portion thereof is analyzed for (opportunistic) offload to an accelerator without using ABEGIN or AEND. A software or hardware pattern match is performed against the thread or a portion thereof for code that may be executable on an accelerator. If an accelerator is not available, or if the thread or a portion thereof does not itself lend itself to accelerator execution, the thread's instructions are executed normally. However, if an accelerator is available, it is preferable to use the accelerator (e.g., use less power), and then the instructions are translated to run on that accelerator and scheduled for execution on that accelerator. As a result, the use of the accelerator is opportunistic.

In a sixth aspect, transformations are performed on the code fragment (part of a thread) to better suit the selected destination processing element. For example, the code fragment may be 1) transformed to take advantage of a different instruction set, 2) to be more parallelized, 3) to be less parallelized (serialized), 4) to be data parallelized (e.g., vectorized), and/or 5) to be less data parallelized (e.g., non-vectorized).

In a seventh aspect, a work queue (either shared or dedicated) receives descriptors that define the scope of work to be performed by the device. A dedicated work queue stores descriptors for a single application, while a shared work queue stores descriptors submitted by multiple applications. A hardware interface/arbiter dispatches the descriptors from the work queue to the accelerator processing engines according to a particular arbitration policy (e.g., based on the processing requirements of each application and QoS/fairness policies).

In an eighth aspect, an improvement to dense matrix multiplication considers multiplication of two-dimensional matrices with a single instruction execution. Multiple packed data (SIMD, vector) sources are multiplied against a single packed data source. In some examples, a binary tree is used for the multiplication.

1 is a representation of a heterogeneous multiprocessing execution environment. In this example, a first type of code fragment (e.g., one or more instructions associated with a software thread) is received by a heterogeneous scheduler 101. The code fragment may be in any number of source code representations, including, for example, machine code, intermediate representations, bytecode, text-based code (assembly code for a high-level language, e.g., C++, source code), etc. The heterogeneous scheduler 101 presents a homogeneous multiprocessor programming model (e.g., so that all threads appear to a user and/or the operating system as if they were executing on a scalar core), determines a workload type (program phase) for a received code fragment, selects a processing element type (scalar, out-of-order (OOO), single instruction multiple data (SIMD) or accelerator) corresponding to the determined workload type to process the workload (e.g., scalar for thread-parallel code, OOO for serial code, SIMD for data parallelism, and accelerator for data parallelism), and schedules the code fragment for processing by the corresponding processing element. In the particular embodiment shown in FIG. 1, the processing element types are , a scalar core 103 (e.g., an in-order core), a single instruction multiple data (SIMD) core 105 that operates on packed data operands, in which registers have multiple data elements stored contiguously, a low-latency out-of-order core 107, and an accelerator 109. In some embodiments, the scalar core 103, the single instruction multiple data (SIMD) core 105, and the low-latency out-of-order core 107 are within the heterogeneous processor, and the accelerator 109 is external to the heterogeneous processor. However, it should be noted that various different configurations of processing elements may be utilized. In some implementations, the heterogeneous scheduler 101 converts or interprets the received code fragment, or a portion thereof, into a format corresponding to a selected type of processing element.

The processing elements 103-109 may support different instruction set architectures (ISAs). For example, an out-of-order core may support a first ISA and an in-order core may support a second ISA. This second ISA may be a (sub- or super) set of the first ISA or may be different. Additionally, the processing elements may have different microarchitectures. For example, a first out-of-order core may support a first microarchitecture and an in-order core may support a different second microarchitecture. Note that even within a particular type of processing element, the ISAs and microarchitectures may be different. For example, a first out-of-order core may support a first microarchitecture and a second out-of-order core may support a different microarchitecture. Instructions are "native" to a particular ISA in that they are part of the ISA. Native instructions execute in a particular microarchitecture without requiring external modification (e.g., translation).

In some embodiments, one or more of the processing elements are integrated on a single die, e.g., as a system on a chip (SoC). Such embodiments may benefit, for example, from improved communication latency, manufacturing/cost, reduced pin count, platform compactness, etc. In other embodiments, the processing elements are packaged together, thereby achieving one or more of the benefits of SoC referenced above, without necessarily being on a single die. These embodiments may further benefit, for example, from different processing technologies optimized for each processing element type, smaller die size for improved yield, consolidation of proprietary intellectual property blocks, etc. With some traditional multi-package limitations, it may be difficult to communicate with different devices as they are added. The multi-protocol links described herein minimize or mitigate this challenge by presenting a common interface to the user, operating system ("OS"), etc. for different types of devices.

In some embodiments, the heterogeneous scheduler 101 is implemented in software stored on a computer-readable medium (e.g., memory) for execution on a processor core (e.g., OOO core 107). In these embodiments, the heterogeneous scheduler 101 is referred to as a software heterogeneous scheduler. This software may implement a binary translator, a just-in-time ("JIT") compiler, an OS 117 that schedules execution of threads that contain code fragments, a pattern matcher, an internal module component, or a combination thereof.

In some embodiments, the heterogeneous scheduler 101 is implemented in hardware as a circuit and/or a finite state machine executed by a circuit. In these embodiments, the heterogeneous scheduler 101 is referred to as a hardware heterogeneous scheduler.

From the perspective of a program (e.g., OS 117, emulation layer, hypervisor, secure monitor, etc.), each type of processing element 103-109 utilizes a shared memory address space 115. In some embodiments, the shared memory address space 115 optionally has two types of memory, memory 211 and memory 213, as shown in FIG. 2. In such embodiments, the types of memory may be distinguished in various ways, including but not limited to differences in memory location (e.g., located on different sockets, etc.), differences in corresponding interface standards (e.g., DDR4, DDR5, etc.), differences in power requirements, and/or differences in the underlying memory technology used (e.g., high bandwidth memory (HBM), synchronous DRAM, etc.).

The shared memory address space 115 is accessible by each type of processing element. However, in some embodiments, different types of memory may be preferentially assigned to different processing elements based on, for example, workload needs. For example, in some implementations, the firmware interface (e.g., BIOS or UEFI) or memory storage of the platform includes fields that indicate the types of memory resources available in the platform and/or the commonality of processing elements with respect to a particular address range or memory type.

Heterogeneous scheduler 101 utilizes this information when analyzing threads to determine where they will execute at a given time. Typically, a thread management mechanism looks at the ensemble of information available to it to inform an informed decision depending on how it manages existing threads. This may manifest itself in a number of ways. For example, threads executing on a particular processing element that has a commonality of address ranges that are physically close to the processing element may be given priority processing over threads under normal circumstances that would be executed on the processing element.

Another example is that a thread that would benefit from a particular memory type (e.g., a faster version of DRAM) may have its data physically moved to the memory type and memory references in the code adjusted to point to a portion of the shared address space. For example, while a thread on SIMD core 105 may utilize second memory type 213, it may be moved out of this usage if accelerator 109 is active and requires that memory type 213 (or at least the portion allocated to the thread on SIMD core 105).

An example scenario is when memory is physically closer to one processing element than the other. A common example is when an accelerator is directly connected to a different memory type than the core.

In these examples, it is typically the OS that initiates the data movement. However, there is nothing to prevent a lower level (e.g., a heterogeneous scheduler) from performing this function on its own or with assistance from another component (e.g., the OS). Whether the data from the previous processing element is flushed and the page table entries invalidated depends on the implementation and the penalty for performing the data movement. If the data is not likely to be used immediately, it may be more feasible to simply copy it from storage rather than moving it from one memory type to another.

Figures 117(A)-(B) show an example of movement for threads in a shared memory. In this example, two types of memory share an address space, each with its own range of addresses within that space. In 117(A), shared memory 11715 includes a first type of memory 11701 and a second type of memory 11707. First type of memory 11701 has a first address range 11703, within a range of addresses that are dedicated to thread 1 11705. Second type of memory 11707 has a second address range 11709.

At some point during the execution of thread 1 11705, the heterogeneous scheduler makes a decision to move thread 1 11705 to use addresses in the first type of memory 11701 before the second thread 11711 is assigned to thread 1 11705. This is shown in FIG. 117B. In this example, thread 1 11705 is reassigned to the second type of memory 11707 and given a new set of addresses to use. However, this need not be the case. Note that the difference in memory types may be physical or spatial (e.g., based on distance to the PE).

Figure 118 illustrates an exemplary method for thread migration that may be performed by a heterogeneous scheduler. At 11801, a first thread is directed to run on a first processing element ("PE"), such as a core or accelerator, using a first type of memory in a shared memory space. For example, in Figure 117(A), this is thread 1.

At some later point in time, a request to execute the second thread is received at 11803. For example, an application, OS, etc. requests a hardware thread to be executed.

At 11805, a determination is made that the second thread should be executed on a second PE using a first type of memory in a shared address space. For example, the second thread may be executed on an accelerator that is directly coupled to the first type of memory, and such execution (including freeing the memory used by the first thread) is more efficient than having the second thread use the second type of memory.

In some embodiments, at 11807, data of the first thread is moved from the first type of memory to the second type of memory. This does not necessarily occur if it would be more efficient to simply stop execution of the first thread execution and start another thread in that placement.

At 11809, the translation lookaside buffer (TLB) entry associated with the first thread is invalidated. Additionally, in most embodiments, a data flush is performed.

At 11811, the second thread is directed to the second PE and assigned to the range of addresses in the first type of memory previously assigned to the first thread.

Figure 3 illustrates an exemplary embodiment of a heterogeneous scheduler 301. In some examples, scheduler 301 is part of a runtime system. As shown, a program phase detector 313 receives a code fragment and identifies one or more characteristics of the code fragment to determine whether execution of the corresponding program phase is best characterized as serial, data parallel, or thread parallel. Examples of how this is determined are described in detail below. As described in detail with respect to Figure 1, the code fragment may be in the form of any number of source code representations.

For repetitive code fragments, pattern matcher 311 identifies this "hot" code and, in some instances, also identifies corresponding characteristics that indicate that the workload associated with the code fragment may be better suited for processing on a different processing element. Further details regarding pattern matcher 311 and its operation are described below, e.g., in the context of FIG. 20.

The selector 309 selects a target processing element to execute a native representation of the received code fragment based at least in part on the characteristics of the processing element and the thermal and/or power information provided by the power manager 307. The selection of the target processing element may be as simple as selecting the best match for the code fragment (i.e., the match between the workload characteristics and the processing element capabilities), but may also take into account the current power consumption level of the system (e.g., if provided by the power manager 307), the availability of processing elements, the amount of data to be moved from one type of memory to another (and the associated penalty for doing so), etc. In some embodiments, the selector 309 is a finite state machine implemented in or executed by a hardware circuit.

In some embodiments, the selector 309 also selects a corresponding link protocol to communicate with the target processing element. For example, in some embodiments, the processing elements utilize a corresponding common link interface that can dynamically multiplex or encapsulate multiple protocols over a system fabric or point-to-point interconnect. For example, in certain embodiments, the supported protocols include: 1) a Producer/Consumer, Discover, Configure, Interrupt (PDCI) protocol that enables device discovery, device configuration, error reporting, interrupts, DMA-style data transfers, and various services, as may be defined in one or more proprietary or industry standards (e.g., PCI Express specifications or equivalent alternatives, etc.); 2) a Caching Agent Coherence (CAC) protocol that enables devices to issue coherent read and write requests to the processing elements; and 3) a Memory Access (MA) protocol that enables a processing element to access the local memory of another processing element. The selector 309 selects between these protocols based on the type of request communicated to the processing element. For example, producer/consumer, discovery, configuration or interrupt requests use the PDCI protocol, cache coherence requests use the CAC protocol, and local memory access requests use the MA protocol.

In some embodiments, the thread includes a marker indicating the phase type, and therefore a phase detector is not utilized. In some embodiments, the thread includes implicit or explicit requirements regarding the processing element type, link protocol, and/or memory type. In these embodiments, selector 309 utilizes this information in its selection process. For example, the selection by selector 309 may be overridden by the thread and/or the user.

Depending on the implementation, the heterogeneous scheduler may include one or more converters that process the received code fragments and generate corresponding native encodings for the target processing elements. For example, the heterogeneous scheduler may include a converter that converts a first type of machine code to a second type of machine code, and/or a JIT compiler that converts the intermediate representation to a format native to the target processing elements. Alternatively or in addition, the heterogeneous scheduler may include a pattern matcher that identifies repetitive code fragments (i.e., "hot" code) and caches one or more native encodings of the code fragments or corresponding micro-operations. Each of these optional components is illustrated in FIG. 3. In particular, the heterogeneous scheduler 301 includes a converter 303 and a JIT compiler 305. If the heterogeneous scheduler 301 operates on object code or intermediate representation, the JIT compiler 305 is invoked to convert the received code fragment into a format native to one or more of the target processing elements 103, 105, 107, 109. If the heterogeneous scheduler 301 operates on machine code (binary) (e.g., converting from one instruction set to another), the binary translator 303 converts the received code fragment into machine code native to one or more of the target processing elements. In alternative embodiments, the heterogeneous scheduler 301 may exclude one or more of these components.

For example, in some embodiments, a binary translator is not included. This may result in increased programming complexity as the program must take into account potentially available accelerators, cores, etc., instead of letting the scheduler handle this. For example, the program may need to include code for routines in different formats. However, in some embodiments, where there is no binary translator, there is a JIT compiler that accepts code at a higher level, and the JIT compiler performs the necessary translations. If a pattern matcher is present, hot code may be further detected to find code that should be executed on a particular processing element.

For example, in some embodiments, a JIT compiler is not included. This may also result in increased programming complexity as the program must first be compiled into machine code for a particular ISA instead of having the scheduler handle this. However, in some embodiments, if there is a binary translator and there is no JIT compiler, the scheduler may translate between ISAs, as described in more detail below. If a pattern matcher is present, hot code may be further detected to find code that should be executed on a particular processing element.

For example, in some embodiments, a pattern matcher is not included. This can also result in less efficiency, as code that may have been moved is likely to remain in a core that is less efficient for the particular task being performed.

In some embodiments, there is no binary translator, JIT compiler, or pattern matcher. In these embodiments, only phase detection or explicit requests to move threads are used for thread/processing element allocation/migration.

Referring again to FIGS. 1-3, the heterogeneous scheduler 101 may be implemented in hardware (e.g., circuitry), software (e.g., executable program code), or any combination thereof. FIG. 114 illustrates an example of a hardware heterogeneous scheduler circuit and its interaction with memory. The heterogeneous scheduler may be created in many different ways, including, but not limited to, as a field programmable gate array (FPGA)-based or application specific integrated circuit (ASIC)-based state machine, as an embedded microcontroller coupled to a memory that stores therein software that provides the functionality described in detail herein, as a logic circuit having other subcomponents (e.g., data hazard detection circuitry, etc.), and/or as software (e.g., state machine) executed by an out-of-order core, as software (e.g., state machine) executed by a scalar core, as software (e.g., state machine) executed by a SIMD core, or combinations thereof. In the illustrated example, the heterogeneous scheduler is a circuit 11401 that includes one or more components that perform various functions. In some embodiments, this circuit 11401 is part of a processor core 11419, but may also be part of a chipset.

The thread/processing element (PE) tracker 11403 maintains a status for each thread executing in the system and each PE (e.g., the availability of the PE, its current power consumption, etc.). For example, the tracker 11403 maintains an active, idle, or inactive status in a data structure such as a table.

In some embodiments, the pattern matcher 11405 identifies "hot" code, accelerator code, and/or code that requires PE allocation. More details regarding this matching are provided later.

PE information 11411 stores information about what PEs (and their types) are in the system and what can be scheduled by the OS etc.

While detailed above as separate components within the heterogeneous scheduler circuit 11401, the components may be combined and/or moved outside of the heterogeneous scheduler circuit 11401.

Memory 11413 coupled to heterogeneous scheduler circuitry 11401 may include software executed (by the cores and/or heterogeneous scheduler circuitry 11401) that provides additional functionality. For example, software pattern matcher 11417 may be used to identify "hot" code, accelerator code, and/or code that requires PE allocation. For example, software pattern matcher 11417 compares code sequences to a predetermined set of patterns stored in memory. The memory may store converters that convert code from one instruction set to another (e.g., from one instruction set to accelerator-based instructions or primitives).

These components provide information to a selector 11411 that selects the PE in which to run the thread, such as what link protocol is used and what transitions should occur if there is already a thread running on the PE. In some embodiments, the selector 11411 is a finite state machine implemented in or executed by a hardware circuit.

Memory 11413 may include, for example, in some embodiments, one or more converters 11415 (e.g., binary, JIT compilers, etc.) stored in the memory to convert thread code into a different format for a selected PE.

115 illustrates an example of a software heterogeneous scheduler. The software heterogeneous scheduler may be created in many different ways, including but not limited to, as a field programmable gate array (FPGA)-based or application specific integrated circuit (ASIC)-based state machine, as an embedded microcontroller coupled to a memory that stores therein software providing the functionality described in detail herein, as a logic circuit having other subcomponents (e.g., data hazard detection circuitry, etc.), and/or as software (e.g., state machine) executed by an out-of-order core, as software (e.g., state machine) executed by a scalar core, as software (e.g., state machine) executed by a SIMD core, or combinations thereof. In the illustrated example, the software heterogeneous scheduler is stored in memory 11413. As a result, memory 11413 coupled to processor core 11419 contains software executed (by the core) to schedule threads. In some embodiments, the software heterogeneous scheduler is part of the OS.

Depending on the implementation, a thread/processing element (PE) tracker 11403 in the core maintains a status for each thread executing in the system and each PE (e.g., the availability of the PE, its current power consumption, etc.), or this is performed in software using the thread/PE tracker 11521. For example, the tracker maintains the active, idle, or inactive status in a data structure such as a table.

In some embodiments, pattern matcher 11417 identifies "hot" code and/or code that requires PE allocation. More details regarding this matching are provided later.

PE information 11409 and/or 11509 stores information about what PEs are in the system and what can be scheduled by the OS etc.

Software pattern matcher 11417 may be used to identify "hot" code, accelerator code, and/or code that requires PE allocation.

The thread/PE tracker, processing element information, and/or pattern match are fed to a selector 11411 which selects the PE on which to run the thread, if there is already a thread running on the PE, what link protocol to use, what transition should occur, etc. In some embodiments, the selector 11411 is a finite state machine implemented and executed by the processor core 11419.

Memory 11413 may include, for example, in some embodiments, one or more converters 11415 (e.g., binary, JIT compilers, etc.) stored in the memory to convert thread code into a different format for a selected PE.

During operation, the OS utilizes a heterogeneous scheduler (e.g., heterogeneous scheduler 101, 301, etc.) that presents an abstraction of the execution environment to schedule and execute threads to be processed.

The following table summarizes potential abstraction features (i.e., what a program refers to), potential design freedoms and architectural optimizations (i.e., what is hidden from the programmer), and potential benefits or reasons for providing particular features in the abstraction.

In some exemplary implementations, the heterogeneous scheduler presents a complete programming model that runs everything and supports all programming techniques (e.g., compilers, intrinsics, assembly, libraries, JIT, offload, device) in combination with other hardware and software resources. Other exemplary implementations present alternative execution environments that are compatible with those offered by other processor developers, e.g., ARM Holdings, Inc., MIPS, IBM, or their licensees or adopters.

Figure 119 is a block diagram of a processor presenting an abstract execution environment, as described in detail above. In this example, the processor 11901 includes several different core types, such as those described in detail in Figure 1. Each (wide) SIMD core 11903 includes fused multiply-add (FMA) circuitry supporting dense arithmetic primitives, its own caches (e.g., L1 and L2), special purpose execution circuitry, and storage for thread state.

Each latency optimized (OOO) core 11913 includes fused multiply-add (FMA) circuitry, its own caches (e.g., L1 and L2) and out-of-order execution circuitry.

Each scalar core 11905 includes fused multiply-add (FMA) circuitry, its own caches (e.g., L1 and L2), special purpose execution and thread state storage. Typically, a scalar core 11905 supports enough threads to cover memory latencies. In some embodiments, the number of SIMD cores 11903 and latency-optimized cores 11913 is small compared to the number of scalar cores 11905.

In some embodiments, one or more accelerators 11917 are included. These accelerators 11917 may be fixed function or FPGA based. Alternatively or in addition to these accelerators 11917, in some embodiments, the accelerators 11917 are external to the processor.

The processor 11901 also includes a last level cache (LLC) 11907 that is shared by the cores within the processor and potentially any accelerators. In some embodiments, the LLC 11907 includes circuitry for high speed atomics.

One or more interconnects 11915 couple the cores and accelerators to each other and to external interfaces. For example, in some embodiments, a mesh-type interconnect couples the various cores.

The memory controller 11909 couples the cores and/or accelerators to memory.

Multiple input/output interfaces (e.g., PCIe, a common link, described in more detail below) 11911 connect the processor 11901 to external devices, e.g., other processors and accelerators.

FIG. 4 illustrates an embodiment of system boot and device discovery for a computer system. Knowledge about the system includes, for example, what cores are available, how much memory is available, memory locations associated with the cores, etc., to be utilized by the heterogeneous scheduler. In some embodiments, this knowledge is built using the Advanced Configuration and Power Interface (ACPI).

At 401, the computer system is booted.

At 403, configuration settings are queried. For example, in some BIOS-based systems, when booted, the BIOS performs a system check and prepares the computer for operation by querying its own memory banks for drives and other configuration settings.

At 405, a search for plug-in components occurs. For example, the BIOS searches for any plug-in components in the computer and sets up pointers (interrupt vectors) in memory to access their routines. The BIOS accepts requests from device drivers and application programs for interfacing with hardware and other peripheral devices.

At 407, data structures for system components (e.g., cores, memory, etc.) are generated. For example, the BIOS typically generates hardware device and peripheral device configuration information with which the OS interfaces with attached devices. Additionally, ACPI defines a flexible and scalable hardware interface to the system board, allowing the computer to turn its peripheral devices on and off for improved power management, especially in portable devices such as notebook computers. The ACPI specification includes hardware interfaces, software interfaces (APIs), and data structures that, if implemented, support OS-directed configuration and power management. Software designers can use ACPI to integrate power management functions for the entire computer system, including hardware, operating system, and application software. This integration allows the OS to determine which devices are active and handle all of the power management resources for computer subsystems and peripheral devices.

At 409, the operating system (OS) loads and gains control. For example, when the BIOS completes its startup routines, the BIOS hands control over to the OS. In the case of ACPI, the BIOS hands control over the computer over to the OS, which exports a data structure to the OS that contains an ACPI namespace, which may be graphically represented as a tree. The namespace acts as a directory of ACPI devices connected to the computer and contains objects that further define and provide status information for each ACPI device. Each node in the tree is associated with a device, while the nodes, subnodes, and leaves represent objects that, when evaluated by the OS, control the device and return specific information to the OS as defined in the ACPI specification. The OS or a driver accessed by the OS may include a set of functions to enumerate and evaluate namespace objects. When the OS calls a function and returns the value of an object in the ACPI namespace, the OS is said to have evaluated that object.

In some examples, available devices change. For example, accelerators, memory, etc. are added. An embodiment of a method for post-system boot device discovery is shown in FIG. 116. For example, this method embodiment may be used to discover accelerators added to a system after boot. At 11601, an indication of a connected device being powered on or reset is received. For example, an endpoint device is plugged into a PCIe slot or reset, for example, by the OS.

At 11603, link training is performed with the connected device to initialize the connected device. For example, PCIe link training is performed to establish link configuration parameters such as link width, lane polarity, and/or maximum data rate supported. In some embodiments, the capabilities of the connected device are stored (e.g., in an ACPI table).

At 11605, when the initialization of the connected device is complete, a ready message is sent from the connected device to the system.

At 11607, the connected device's ready status bit is set to indicate that the device is ready for configuration.

At 11609, the initialized, connected device is configured. In some embodiments, the device and the OS agree on an address for the device (e.g., a memory-mapped I/O (MMIO) address). The device provides a device descriptor that includes one or more of a vendor identification number (ID), a device ID, a model number, a serial number, characteristics, resource requirements, etc. The OS may determine additional operational and configuration parameters for the device based on the descriptor data and system resources. The OS may generate a configuration query. The device may respond with the device descriptor. The OS then generates configuration data and sends this data to the device (e.g., through PCI hardware). This may include setting a base address register that defines the address space associated with the device.

After system knowledge is built, the OS utilizes a heterogeneous scheduler (e.g., heterogeneous scheduler 101, 301, etc.) to schedule threads to be processed and execute the threads. The heterogeneous scheduler then dynamically and transparently (e.g., to the user and/or the OS) maps each thread's code fragment to the most suitable type of processing element, thereby potentially avoiding the need to build hardware for legacy architectural mechanisms and potentially exposing microarchitecture details to the system programmer or the OS.

In some examples, the most suitable type of processing element is determined based on the performance of the processing element and the execution characteristics of the code fragment. In general, a program and associated threads may have different execution characteristics depending on the workload being processed at a given time. Exemplary execution characteristics, or phases of execution, include, for example, a data-parallel phase, a thread-parallel phase, and a serial phase. The following table identifies these phases and summarizes their characteristics. The table also includes exemplary workloads/operations, exemplary hardware useful in processing each phase type, and typical purposes for the phases and hardware used.

In some embodiments, the heterogeneous scheduler is configured to select between thread migration and emulation. In a configuration in which each type of processing element can process any type of workload (if it requires emulation to do so), the most suitable processing element is selected for each program phase based on one or more criteria including, for example, the latency requirements of the workload, the increased execution latency associated with emulation, the power and thermal characteristics and constraints of the processing element, etc. As will be described in more detail below, the selection of the suitable processing element is achieved in some embodiments by detecting the presence of SIMD instructions or vectorizable code in the code fragment, taking into account the number of threads being executed.

Moving threads between processing elements is not without disadvantages. For example, data may need to be moved from a shared cache to a lower level cache, and both the originating and recipient processing elements will flush their pipelines to accommodate the move. Optionally, in some embodiments, the heterogeneous scheduler implements hysteresis to avoid very frequent migrations (e.g., by setting thresholds for one or more of the criteria referenced above, or a subset of the same). In some embodiments, hysteresis is implemented by limiting thread migrations to not exceed a predefined rate (e.g., one migration per millisecond). Thus, the rate of such migrations is limited to avoid excessive overload due to code generation, synchronization, and data migration.

In some embodiments, for example, if migration is not selected by the heterogeneous scheduler as being the preferred approach for a particular thread, the heterogeneous scheduler emulates missing functionality for the thread on the assigned processing element. For example, in an embodiment that maintains a constant total number of threads available to the operating system, the heterogeneous scheduler may emulate multithreading when the number of available hardware threads is oversubscribed (e.g., in a wide simultaneous multithreading core). On a scalar or latency core, one or more SIMD instructions of the thread are converted to scalar instructions, or on a SIMD core, more threads are spawned and/or instructions are converted to utilize packed data.

Figure 5 shows an example of thread migration based on a mapping of program phases to three types of processing elements. As shown, the three types of processing elements include latency optimized (e.g., out-of-order cores, accelerators, etc.), scalar (processing one data item per instruction time), and SIMD (processing multiple data elements per instruction). Typically, this mapping is performed by a heterogeneous scheduler on a per-thread or per-code fragment basis in a manner that is transparent to the programmer and the operating system.

In one embodiment, a heterogeneous scheduler is used to map each phase of a workload to the most appropriate type of processing element. Ideally, this alleviates the need to build hardware for legacy functions and avoids exposing micro-architectural details. The heterogeneous scheduler presents a complete programming model that supports multiple code types, including compiled code (machine code), intrinsics (programming language logical constructs that map directly to processor or accelerator instructions), assembly code, libraries, intermediate (JIT-based), offload (movement from one machine type to another), and device-specific.

In a particular configuration, the default selection for the target processing element is the processing element for which latency is optimized.

Referring again to FIG. 5, in the serial phase execution 501 for a workload, it is first processed on one or more latency-optimized processing elements. Upon detecting a phase shift (e.g., before execution, in a dynamic manner such that the code parallelizes more data, e.g., as seen by the type of instructions available in the code before or during execution), the workload is migrated to one or more SIMD processing elements to complete the data-parallel phase execution 503. Furthermore, execution scheduling and/or transformations are typically cached. The workload is then migrated back to one or more latency-optimized processing elements, or a second set of one or more latency-optimized processing elements, to complete the next serial phase execution 505. The workload is then migrated to one or more scalar cores to process the thread-parallel phase execution 507. The workload is then migrated back to one or more latency-optimized processing elements for completion of the next serial phase execution 509.

While this illustrated example shows a return to a latency-optimized core, the heterogeneous scheduler may continue execution of any subsequent phases of execution in one or more corresponding types of processing elements until the thread is terminated. In some embodiments, the processing elements utilize work queues to store incomplete tasks. As a result, tasks may not start immediately, but will be executed when their spot in the queue appears.

6 is an exemplary implementation flow performed by a heterogeneous scheduler, such as heterogeneous scheduler 101. The flow illustrates the selection of a processing element (e.g., a core). As shown, a code fragment is received by the heterogeneous scheduler. In some embodiments, an event occurs including, but not limited to, a thread wakeup command, a write to a page directory base register, a sleep command, a thread phase change, and one or more instructions indicating a desired reallocation.

In 601, the heterogeneous scheduler determines whether there is parallelism in a code fragment (e.g., a code fragment in a serial or parallel phase) based on, for example, detected data dependencies, instruction types, and/or control flow instructions. For example, a thread full of SIMD code would be considered parallel. If the code fragment is not suitable for parallel processing, the heterogeneous scheduler selects one or more latency-sensitive operation elements (e.g., OOO cores) to process the code fragment in the serial phase of execution 603. Typically, OOO cores have (deep) speculation and dynamic scheduling, and usually have lower performance per watt compared to simpler alternatives.

In some embodiments, there are no latency-sensitive operation elements available, as these typically consume more power and die space than scalar cores. In these embodiments, only scalar, SIMD, and accelerator cores are available.

At 605, for parallel, parallelizable, and/or vectorizable code fragments, the heterogeneous scheduler determines a type of code parallelism. At 607, for thread-parallel code fragments, the heterogeneous scheduler selects thread-parallel processing elements (e.g., multiprocessor scalar cores). Thread-parallel code fragments include independent instruction sequences that can be executed simultaneously on separate scalar cores.

Data parallel code occurs when each processing element performs the same task on a different amount of data. Data parallel code can be in the form of different data layouts: packed and random. At 609, the data layout is determined. Random data may be assigned to SIMD processing elements, but requires the use of gather instructions 613 that pull data from different memory locations, spatial computation arrays 615 (which spatially map computations onto an array of small programmable processing elements, e.g., an array of FPGAs), or arrays of scalar processing elements 617. Packed data is assigned to SIMD processing elements or processing elements that use dense arithmetic primitives at 611.

In some embodiments, transformations are performed on the code fragment to better suit the selected destination processing element. For example, the code fragment may be 1) transformed to take advantage of a different instruction set, 2) made more parallel, 3) made less parallel (serialized), 4) made data parallel (e.g., vectorized), and/or 5) made less data parallel (e.g., non-vectorized).

After a processing element is selected, the code fragment is sent to one of the determined processing elements for execution.

Figure 7 illustrates an example method for thread destination selection by a heterogeneous scheduler. In some embodiments, the method is performed by a binary translator. At 701, a thread to be evaluated or a code fragment thereof is received. In some embodiments, an event occurs including, but not limited to, a thread wakeup command, a write to a page directory base register, a sleep command, a phase change of a thread, and one or more instructions indicating a desired reallocation.

At 703, a determination is made whether the code fragment is to be offloaded to an accelerator. For example, a code fragment is sent to an accelerator. The heterogeneous scheduler may know that this is a corrective action if the code contains code that identifies a desire to use an accelerator. This desire may be an identifier indicating that the region of code may be executed on an accelerator or executed natively (e.g., ABEGIN/AEND as described herein), or an explicit command to use a particular accelerator.

In some embodiments, at 705, a transformation of the code fragment is performed to better suit the selected destination processing element. For example, the code fragment may be 1) transformed to take advantage of a different instruction set, 2) made more parallel, 3) made less parallel (serialized), 4) made data parallel (e.g., vectorized), and/or 5) made less data parallel (e.g., non-vectorized).

Typically, at 707, the translated threads are cached for later use. In some embodiments, the binary translator caches the translated threads locally so that they are available for future use by the binary translator. For example, if the code becomes "hot" (is executed repeatedly), caching provides a mechanism for future use without the penalty of translation (although there may be transmission costs).

At 709, the (translated) thread is sent (e.g., offloaded) to the destination processing element for processing. In some embodiments, the translated thread is cached by the recipient so that it is locally available for future use. Additionally, if the recipient or binary translator determines that the code is "hot," this caching allows for faster execution with less energy used.

In 711, the heterogeneous scheduler determines whether there is parallelism in the code fragment (e.g., the code fragment is in a serial or parallel phase) based on, for example, detected data dependencies, instruction type, and/or control flow instructions. For example, a thread full of SIMD code would be considered parallel. If the code fragment is not suitable for parallel processing, the heterogeneous scheduler selects one or more latency-sensitive operation elements (e.g., OOO cores) to process the code fragment in the serial phase of execution 713. Typically, OOO cores have (deep) speculation and dynamic scheduling, and therefore may have better performance per watt compared to scalar alternatives.

In some embodiments, there are no latency-sensitive operation elements available, as these typically consume more power and die space than scalar cores. In these embodiments, only scalar, SIMD, and accelerator cores are available.

At 715, for parallel, parallelizable, and/or vectorizable code fragments, the heterogeneous scheduler determines a type of code parallelism. At 717, for thread-parallel code fragments, the heterogeneous scheduler selects a thread-parallel processing element (e.g., a multiprocessor scalar core). A thread-parallel code fragment includes independent instruction sequences that can be executed simultaneously on separate scalar cores.

Data parallel code occurs when each processing element performs the same task on a different amount of data. Data parallel code can be in the form of different data layouts: packed and random. At 719, the data layout is determined. Random data may be assigned to SIMD processing elements, but requires the use of gather instructions 723, spatial computation arrays 725, or arrays of scalar processing elements 727. Packed data is assigned to SIMD processing elements or processing elements that use dense arithmetic primitives at 721.

In some embodiments, transformations of the non-offloaded code fragments are performed to better suit the determined destination processing element. For example, the code fragments are 1) transformed to utilize a different instruction set, 2) made more parallel, 3) made less parallel (serialized), 4) made data parallel (e.g., vectorized), and/or 5) made less data parallel (e.g., non-vectorized).

After a processing element is selected, the code fragment is sent to one of the determined processing elements for execution.

The OS sees the total number of potentially available threads, regardless of which cores and accelerators are accessible. In the following description, each thread is enumerated by a thread identifier (ID), referred to as the logical ID. In some embodiments, the operating system and/or heterogeneous scheduler uses the logical ID to map a thread to a particular processing element type (e.g., core type), processing element ID, and thread ID on that processing element (e.g., a tuple of core type, core ID, thread ID). For example, a scalar core has a core ID and one or more thread IDs, a SIMD core has a core ID and one or more thread IDs, an OOO core has a core ID and one or more thread IDs, and/or an accelerator has a core ID and one or more thread IDs.

Figure 8 illustrates the concept of using striped mapping for logical IDs. Striped mapping may be used by a heterogeneous scheduler. In this example, there are eight logical IDs and three core types, each with one or more threads. Typically, the mapping from logical IDs to (core ID, thread ID) is calculated using division and modulo and may be fixed to preserve commonality of software threads. The mapping from logical IDs to (core type) is performed flexibly by the heterogeneous scheduler to accommodate future new core types that become available to the OS.

Figure 9 shows an example of the use of striped mapping for logical IDs. In the example, logical IDs 1, 4, and 5 are mapped to a first core type, and all other logical IDs are mapped to a second core type. A third core type is not utilized.

In some embodiments, groupings of core types are created. For example, a "core group" tuple may consist of one OOO tuple and all scalar, SIMD, and accelerator core tuples whose logical IDs map to the same OOO tuple. Figure 10 shows an example of a core group. Typically, serial phase detection and thread migration are performed within the same core group.

Figure 11 illustrates an example method of thread execution in a system utilizing a binary translator switching mechanism. At 1101, a thread executes on a core. The core may be any of the types described in detail herein, including an accelerator.

At 1103, at some point during the execution of the thread, a potential core reallocation event occurs. Exemplary core reallocation events include, but are not limited to, a thread wakeup command, a write to the page directory base register, a sleep command, a phase change of the thread, and one or more instructions indicating a desired reallocation to a different core.

At 1105, the event is processed and a determination is made as to whether there has been a change in the core allocation. An exemplary method for processing a particular core allocation is described in detail below.

In some embodiments, the (re)allocation of cores is subject to one or more limiting factors, such as migration rate limits and power consumption limits. Migration rate limits are tracked per core type, core ID and thread ID. Once a thread is assigned to a target (core type, core ID, thread ID), a timer is started and maintained by the binary translator. No other threads are migrated to the same target until the timer expires. As a result, a thread may migrate away from its current core before the timer expires, while the reverse is not true.

As will be described in more detail, as more core types (including accelerators) are added to a computing system (either on- or off-die), limiting power consumption is likely to become an increased focus. In some embodiments, the instantaneous power consumed by all execution threads on all cores is calculated. If the calculated power consumption exceeds a threshold, new threads are only assigned to lower power cores, e.g., SIMD, scalar, and dedicated accelerator cores, and one or more threads are forced to migrate from an OOO core to a lower power core. Note that in some embodiments, power consumption limits take precedence over migration rate limits.

FIG. 12 illustrates an exemplary method for core allocation of hot code to an accelerator. At 1201, a determination is made that code is "hot." A hot portion of code may refer to a portion of code that is more suitable to run on one core over other cores based on considerations such as power, performance, thermal, other known processor criteria, or combinations thereof. This determination may be made using any number of techniques. For example, a dynamic binary optimizer may be utilized to monitor the execution of threads. Hot code may be detected based on counter values that record dynamic execution frequency of static code, such as during program execution. In an embodiment where a core is an OOO core and another core is an in-order core, then a hot portion of code may refer to a hot spot of program code that is more suitable to run on a serial core, potentially having more available resources for execution of the highly repetitive sections. Often, sections of code that have highly repetitive patterns may be optimized to run more efficiently on an in-order core. Essentially, in this example, cold code (low repetition) is distributed to native OOO cores, while hot code (high repetition) is distributed to software-managed in-order cores. Hot portions of code may be identified statically, dynamically, or a combination thereof. In the first case, a compiler or user may determine that a section of program code is hot code. Decode logic within the core, in one embodiment, is adapted to decode hot code identifier instructions from the program code, which instructions identify hot portions of the program code. Fetching or decoding such instructions may trigger translation and/or execution of the hot section of code on the core. In another example, the code execution is profiled execution, and based on characteristics of the profile - power and/or performance metrics associated with the execution - regions of the program code may be identified as hot code. Monitoring code may be executed on one core to perform monitoring/profiling of program code executing on other cores, as well as hardware operations. Note that such monitoring code may be code maintained in a storage structure within the core or maintained in a system that includes the processor. For example, the monitoring code may be microcode or other code and maintained in a storage structure of the core. As yet another example, the static identification of hot code is done as an implication. However, dynamic profiling of program code execution may ignore the static identification of a region of code as hot, and this type of static identification is often referred to as compiler or user implication that dynamic profiling may take into account when determining which cores are suitable for code distribution. Furthermore, similar to the nature of dynamic profiling, the identification of a region of code as hot is not limited to that section of code always being identified as hot. After transformation and/or optimization, a transformed version of the code section is executed.

At 1203, an appropriate accelerator is selected. The binary translator, virtual machine monitor, or operating system makes this selection based on the available accelerators and the desired performance. In many instances, an accelerator is better suited to run hot code with improved performance per watt than a larger, more general purpose core.

At 1205, the hot code is transmitted to the selected accelerator, utilizing the appropriate connection type, as described in more detail herein.

Finally, at 1207, the hot code is received and executed by the selected accelerator. During execution, the hot code may be evaluated for allocation to different cores.

Figure 13 illustrates an exemplary method for potential core allocation for wake-up or write to page directory base register events. For example, this may refer to determining the phase of a code fragment. At 1301, either a wake-up event or a page directory base register (e.g., task switch) event is detected. For example, a wake-up event occurs due to an interrupt received by a stopped thread or a wait state exit. A write to the page directory base register may indicate the start or stop of a serial phase. Typically, this detection occurs on a core running a binary translator.

At 1303, the number of cores that share the same page table base pointer as the thread that woke up or experienced a task switch is counted. In some embodiments, a table is used to map logical IDs to specific heterogeneous cores. The table is indexed by the logical ID. Each entry in the table includes a flag indicating whether the logical ID is currently active or suspended, a flag indicating whether a SIMD or scalar core is preferred, a page table base address (e.g., CR3), a value indicating the type of core the logical ID is currently mapped to, and a counter that limits the migration rate.

Threads belonging to the same process share the same address space, page table, and page directory base register value.

At 1305, a determination is made as to whether the number of counted cores is greater than one. This count determines whether the thread is in serial or parallel phase. If the count is one, then the thread experiencing the event is in serial phase 1307. As a result, a serial phase thread is one that has a unique page directory base register value among all threads in the same core group. Figure 14 shows an example of a serial phase thread. As shown, a process may have one or more threads, with each process having its own assigned address.

At 1313 or 1315, if the thread experiencing the event is not assigned to an OOO core, it is migrated to an OOO core and the existing thread on the OOO core is migrated to a SIMD or scalar core. If the thread experiencing the event is assigned to an OOO core, it will remain there in most circumstances.

If the count is greater than one, then at 1309, the thread experiencing the event is in a parallel phase and a determination of the type of parallel phase is made. When the thread experiencing the event is in a data parallel phase, if the thread is not assigned to a SIMD core, the thread is assigned to a SIMD core, otherwise at 1313, the thread is kept on the SIMD core if it is already there.

When the thread experiencing the event is in the data parallel phase, if the thread is not assigned to a SIMD core, the thread is assigned to a SIMD core, otherwise, in 1313, the thread is kept on the SIMD core if it is already there.

When the thread experiencing the event is in the thread parallel phase, if the thread is not assigned to a scalar core, the thread is assigned to a scalar core, otherwise the thread is kept on the scalar core if it is already there at 1315.

Furthermore, in some embodiments, a flag is set on the logical ID of the thread to indicate that the thread is running.

Figure 15 illustrates an exemplary method for potential core allocation for thread response to a sleep command event. For example, this may involve determining the phase of a code fragment. At 1501, a sleep event affecting a thread is detected. For example, a stop, wait entry, and timeout or pause command occurs. Typically, this detection occurs on a core running a binary translator.

In some embodiments, at 1503, a flag indicating that the thread is running is cleared for the logical ID of the thread.

At 1505, the number of threads of the core that share the same page table base pointer as the sleep thread is counted. In some embodiments, a table is used to map logical IDs to specific heterogeneous cores. The table is indexed by the logical ID. Each entry in the table includes a flag indicating whether the logical ID is currently active or suspended, a flag indicating whether a SIMD or scalar core is preferred, a page table base address (e.g., CR3), a value indicating the type of core to which the logical ID is currently mapped, and a counter to limit the migration rate. The first running thread (with a given page table base pointer) from the group is mentioned.

At 1507, a determination is made as to whether an OOO core in the system is idle. An idle OOO core has no OS threads actively running.

If the page table base pointer is shared by exactly one thread in the core group, then in 1509, the shared thread is moved from the SIMD or scalar core to the OOO core. If the page table base pointer is shared by more than one thread, then in 1511, the first running thread of the previously mentioned group is the thread that was migrated from the SIMD or scalar core to the OOO core to make room for the released thread (which executes in the place of the first running thread).

Figure 16 illustrates an example method for potential core allocation for threads in response to a phase change event. For example, this illustrates determining the phase of a code fragment. At 1601, a potential phase change event is detected. Typically, this detection occurs on a core running a binary translator.

At 1603, a determination is made as to whether the logical ID of the thread is valid on the scalar core and a SIMD instruction is present. If there is no such SIMD instruction, then the thread continues to execute normally. However, if there is a SIMD instruction present in the thread executing on the scalar core, then the thread is migrated to the SIMD core at 1605.

At 1607, a determination is made as to whether the logical ID of the thread is valid on the SIMD core and no SIMD instructions are present. If there are SIMD instructions, then the thread continues to execute normally. However, if there are no SIMD instructions present in the thread executing on the SIMD core, then the thread is migrated to the scalar core at 1609.

As noted throughout this discussion, accelerators accessible to a binary translator may provide more efficient execution (including more energy efficient execution). However, being able to write programs for each potentially available accelerator may be a difficult, if not impossible, task.

Described in detail herein are embodiments that use descriptive instructions to explicitly indicate the beginning and end of potential accelerator-based execution for a portion of a thread. If no accessors are available, the code between the descriptive instructions will execute without the use of an accelerator. In some implementations, the code between these instructions may relax some semantics of the executing core.

FIG. 17 shows an example of code describing an acceleration region. The first instruction in this region is an acceleration begin (ABEGIN) instruction 1701. In some embodiments, the ABEGIN instruction provides permission to enter a relaxed (sub) mode of execution for a non-accelerator core. For example, the ABEGIN instruction in some embodiments allows a programmer or compiler to indicate in a field of the instruction which characteristics of the sub mode differ from the standard mode. Exemplary characteristics include, but are not limited to, one or more of ignoring self-modifying code (SMC), relaxing memory consistency model restrictions (e.g., relaxing store ordering requirements), changing floating point semantics, changing performance monitoring (perfmon), changing architectural flag usage, and the like. In some embodiments, an SMC is a write to a memory location in a code segment currently cached in the processor that causes the associated cache line (or lines) to be invalidated. If the write affects a prefetch instruction, the prefetch queue is invalidated. This latter check is based on the linear address of the instruction. A write or snoop of an instruction in a code segment where the target instruction has already been decoded and is present in the trace cache invalidates the entire trace cache. SMC may be ignored by adjusting the SMC detection circuitry in the translation lookaside buffer. For example, memory consistency model restrictions may be changed by changing settings in one or more registers or tables (e.g., memory type range registers or page attribute tables). For example, when changing floating-point semantics, the way in which the floating-point execution circuits perform floating-point calculations is changed through the use of one or more control registers that control the operation of these circuits (e.g., setting a floating-point unit (FPU) control word register). Floating-point semantics that may be changed include, but are not limited to, rounding modes, how exception masks and status flags are handled, flush-to-zero, denormalization settings, and precision (e.g., single, double, extended) control. Additionally, in some embodiments, the ABEGIN instruction takes into account explicit accelerator type preferences, so that an accelerator of the preferred type is selected if it is available.

Non-accelerator code 1703 follows the ABEGIN instruction 1701. This code is native to the processor cores of the system. In the worst case, if there are no accelerators available or ABEGIN is not supported, this code will simply execute on the core. However, in some embodiments, a submode is used for the execution.

By having an accelerated end (AEND) instruction 1705, execution is gated on the processor core until the accelerator appears to have completed execution. Effectively, the use of ABEGIN and AEND allows the programmer to opt in/out of using the accelerator and/or relaxed mode of execution.

Figure 18 illustrates an embodiment of a method for execution using ABEGIN in a hardware processor core. At 1801, an ABEGIN instruction for a thread is fetched. As previously mentioned, the ABEGIN instruction typically includes one or more fields that are used to define different (sub)modes of execution.

At 1803, the fetched ABEGIN instruction is decoded using the decode circuitry. In some embodiments, the ABEGIN instruction is decoded into micro-ops.

At 1805, the decoded ABEGIN instruction is executed by the execution circuitry such that the thread enters a different mode (which may be explicitly specified by one or more fields of the ABEGIN instruction) for instructions that follow the ABEGIN instruction but precede the AEND instruction. This different mode of execution may be on an accelerator or on an existing core, depending on the availability and selection of the accelerator. In some embodiments, the accelerator selection is performed by a heterogeneous scheduler.

At 1807, the subsequent non-AEND instruction is executed in a different mode of execution. If an accelerator is used for execution, the instruction may first be translated by a binary translator into a different instruction set.

Figure 19 illustrates an embodiment of a method for execution with AEND in a hardware processor core. At 1901, an AEND instruction is fetched.

At 1903, the fetched AEND instruction is decoded using the decode circuitry. In some embodiments, the AEND is decoded into a micro-op.

At 1905, the decoded AEND instruction is executed by the execution circuitry to return from the different mode of execution previously established by the ABEGIN instruction. This different mode of execution may be on the accelerator or on the existing core, depending on the availability and selection of the accelerator.

At 1807, subsequent non-AEND instructions are executed in the original mode of execution. If an accelerator is used for execution, the instructions may first be translated by a binary translator to a different instruction set.

Figure 124 shows an example of execution when ABEGIN/AEND is not supported. At 12401, the ABEGIN instruction is fetched. At 12403, a determination is made that ABEGIN is not supported. For example, CPUID indicates no support.

In the absence of support, typically an operation (nop) is performed at 12405 that does not change the context associated with the thread. At 12407, since there is no change in execution mode, the instruction following the unsupported ABEGIN is executed normally.

In some embodiments, the equivalent usage of ABEGIN/AEND is achieved using at least pattern matching. This pattern matching may be based on hardware, software and/or both. FIG. 20 illustrates a system that provides ABEGIN/AEND equivalence using pattern matching. The illustrated system includes a scheduler 2015 (e.g., a heterogeneous scheduler as described in detail above) that includes a translator 2001 (e.g., a binary translator, JIT, etc.) stored in a memory 2005. Core circuitry 2007 executes scheduler 2015. Scheduler 2015 receives threads 2019 that may or may not have explicit ABEGIN/AEND instructions.

The scheduler 2015 manages the software-based pattern matcher 2003, performs traps and context switches during offloading, manages the user space save area (described in detail later), and generates or converts accelerator code 2011. The pattern matcher 2003 recognizes memory-stored (predefined) code sequences available in the received thread 2019 that may benefit from accelerator utilization and/or relaxed execution states, but that are not described with ABEGIN/AEND. Typically, the respective patterns are stored in the converter 2001, but are at least accessible to the pattern matcher 2003. The selector 2019 functions as previously described in detail.

Scheduler 2015 may provide performance monitoring features. For example, if the code does not have a perfect pattern match, scheduler 2015 recognizes that the code may need further relaxation of requirements to be more efficient, and adjusts the operating mode associated with the thread accordingly. The relationship of the operating modes is described in detail above.

The scheduler 2015 also performs one or more of the following: cycling through cores in the ABEGIN/AEND region, cycling through activated or stalled accelerators, counting ABEGIN calls, delaying accelerator queuing (synchronization), and monitoring memory/cache statistics. In some embodiments, the binary translator 2001 includes accelerator-specific code that is used to interpret accelerator code that may be useful in identifying bottlenecks. The accelerator executes this translated code.

In some embodiments, the core circuitry 2007 includes a hardware pattern matcher 2009 that uses stored patterns 2017 to recognize (predefined) code sequences in received threads 2019. Typically, this pattern matcher 2009 is lightweight compared to the software pattern matcher 2003 and looks for areas that are easy to express (e.g., rep movs). The recognized code sequences may be transformed for accelerator use by the scheduler 2015 and/or may result in a relaxed operating mode for the thread.

Coupled to the system are one or more accelerators 2013 that receive and execute accelerator code 2011.

FIG. 21 illustrates an embodiment of a method for non-accelerated description threads that are subjected to pattern recognition. The method is performed by a system that includes at least one type of pattern matcher.

In some embodiments, at 2101, a thread is executed. Typically, the thread is executed on a non-accelerator core. The instructions of the executing thread are provided to the pattern matcher. However, the instructions of the thread may be provided to the pattern matcher prior to any execution.

At 2103, patterns are recognized (detected) within the threads. For example, a software-based pattern matcher or a hardware pattern matcher circuit finds patterns that are typically associated with available accelerators.

At 2105, the recognized patterns are translated for the available accelerators. For example, a binary translator converts the patterns into accelerator code.

The translated code is forwarded to an available accelerator at 2107 for execution.

FIG. 22 illustrates an embodiment of a method for non-accelerated description threads that are subject to pattern recognition. The method is performed by a system that includes at least one type of pattern matcher, such as the system shown in FIG. 20.

In some embodiments, at 2201, a thread is executed. Typically, the thread is executed on a non-accelerator core. The instructions of the executing thread are provided to a pattern matcher. However, the instructions of the thread may be provided to the pattern matcher prior to any execution.

At 2203, patterns are recognized (detected) within the threads. For example, a software-based pattern matcher or a hardware pattern matcher circuit finds patterns that are typically associated with available accelerators.

At 2205, the binary translator adjusts an operating mode associated with the thread to use the mitigation request based on the recognized pattern. For example, the binary translator utilizes a setting associated with the recognized pattern.

As described in detail, in some embodiments, parallel regions of code are delimited by ABEGIN and AEND instructions. Within an ABEGIN/AEND block, isolation of certain memory load and store operations is guaranteed. Other loads and stores take potential dependencies into account. This allows an implementation to parallelize a block with little or no memory dependency checking. In any case, serial execution of the block is permitted since the serial case is included among the possible ways to execute the block. The binary translator performs static dependency analysis to create instances of parallel execution and maps these instances to hardware. Static dependency analysis may parallelize outer, middle or inner loop iterations. Slicing is implementation dependent. The ABEGIN/AEND implementation extracts parallelism in the size that is most appropriate for the implementation.

ABEGIN/AEND blocks may contain multiple levels of nested loops. Implementations are free to choose the amount of parallel execution supported or fall back to serial execution. ABEGIN/AEND provide parallelism over a much larger domain than SIMD instructions. For certain types of code, ABEGIN/AEND allow more efficient hardware implementations than multithreading.

Through the use of ABEGIN/AEND, a programmer and/or compiler can fall back to traditional serial execution by a CPU core if the parallelism criteria are not met. When running on a traditional out-of-order CPU core, ABEGIN/AEND reduces the area and power requirements of the Memory Ordering Buffer (MOB) as a result of relaxed memory ordering.

Within the ABEGIN/AEND block, the programmer specifies memory dependencies. Figure 23 shows various types of memory dependencies 2301, their semantics 2303, ordering requirements 2305, and use cases 2307. Additionally, some semantics apply to the instructions within the ABEGIN/AEND block, depending on the implementation. For example, in some embodiments, register dependencies are allowed, but modifications to registers do not persist beyond the AEND. Additionally, in some embodiments, the ABEGIN/AEND block must be entered at ABEGIN and exited at AEND (or a similar state entered based on pattern recognition) with no branches to or from the ABEGIN/AEND block. Finally, typically the instruction stream cannot be modified.

In some embodiments, the ABEGIN instruction includes a source operand that includes a pointer to a memory data block. This data memory block contains much of the information used by the runtime and core circuitry to process the code within the ABEGIN/AEND block.

Figure 24 shows an example of a memory data block pointed to by an ABEGIN instruction. As shown, depending on the implementation, the memory data block includes a field for a sequence number 2401, a field for a block class 2403, a field for an implementation identifier 2405, a field for a saved state area size 2407, and a field for a local storage area size 2409.

Sequence number 2401 indicates how much (parallel) computation the processor has gone through before an interrupt. Software initializes sequence number 2401 to zero before execution of ABEGIN. Execution of ABEGIN writes a non-zero value to sequence number 2401 to track its progress. Upon completion, execution of AEND writes zero to reinitialize sequence number 2401 for its next use.

The predefined block class identifier 2403 (i.e., GUID) defines the predefined ABEGIN/AEND block class. For example, DMULADD and DGEMM may be predefined as block classes. With predefined classes, the binary translator does not need to parse the binary to perform mapping analysis to heterogeneous hardware. Instead, the translator (e.g., binary translator) performs a pre-generated translation for this ABEGIN/AEND class by simply taking the input values. The code enclosed with ABEGIN/AEND simply serves as the code used to execute this class in the non-specialized core.

The implementation ID field 2405 indicates the type of execution hardware used. Execution of ABEGIN updates this field 2405 to indicate the type of heterogeneous hardware used. This helps implementations migrate ABEGIN/AEND code to machines with different acceleration hardware types or no accelerators at all. This field allows possible transformation of the saved context to adapt the target implementation. That is, the emulator is used to execute the code until it exits AEND after migration when ABEGIN/AEND code is interrupted and migrated to a machine that does not have the same accelerator type. This field 2405 may also allow the system to dynamically reassign ABEGIN/AEND blocks to different heterogeneous hardware in the same machine, even if they are interrupted in the middle of their execution.

The state save area field 2407 indicates the size and format of the state save area, which is implementation specific. The implementation guarantees that the implementation specific portion of the state save area does not exceed some maximum value specified in CPUID. Typically, execution of the ABEGIN instruction causes a write to the state save area of general purpose and packed data registers that are modified within the ABEGIN/AEND block, associated flags, and additional implementation specific state. Multiple instances of registers may be written to facilitate parallel execution.

Local storage area 2409 is allocated as the local storage area. The amount of storage to reserve is typically specified as an immediate operand to ABEGIN. When the ABEGIN instruction is executed, a write to a particular register (e.g., R9) is made with an address in local storage 2409. In case of a failure, this register is made to point to a sequence number.

Each instance of parallel execution receives a unique local storage area 2409. The address is different for each instance of parallel execution. Serial execution is assigned one storage area. The local storage area 2409 provides temporary storage beyond the general purpose and packed data registers of the architecture. The local storage area 2409 should not be accessed outside the ABEGIN/AEND blocks.

Figure 25 shows an example of a memory 2503 configured to use ABEGIN/AEND semantics. Not shown is the hardware (e.g., various processing elements described herein) that supports ABEGIN/AEND and utilizes this memory 2503. As will be described in more detail, the memory 2503 includes a saved state area 2507 that contains indications of registers 2501 to be used, flags 2505, and implementation specific information 2511. Additionally, local storage 2509 is stored in the memory 2503 for each parallel execution instance.

Figure 26 shows an example method of operation in different modes of execution using ABEGIN/AEND. Typically, this method is performed by a combination of entities, e.g., a transformer and an execution circuit. In some embodiments, the thread is transformed before entering this mode.

At 2601, a different mode of execution is entered, such as a relaxed mode of execution (with or without an accelerator). Typically, this mode is entered through execution of an ABEGIN instruction. However, as described in more detail above, this mode may also be entered through a pattern match. Entering this mode includes resetting the sequence numbers.

At 2603, a saved state area is written, e.g., general and packed data registers that are modified, associated flags, and additional implementation-specific information. This area allows execution to be resumed or rolled back if something goes wrong in the block (e.g., an interrupt).

At 2605, a local storage area is reserved for each parallel execution instance. The size of this area is indicated by the state save area field, as described in detail above.

At 2607, the progress of the block is tracked during execution of the block. For example, if an instruction is retired after successful execution, the sequence number of the block is updated.

At 2609, a determination is made as to whether the AEND instruction has arrived (e.g., to determine whether the block is complete). If the AEND instruction has not arrived, then at 2613, the local storage area is updated with the intermediate results. If possible, execution picks up from these results. However, in some instances, at 2615, a rollback to before the ABEGIN/AEND occurs. For example, if an exception or interrupt occurs during execution of the ABEGIN/AEND block, the instruction pointer points to the ABEGIN instruction and the R9 register points to the memory data block that is updated with the intermediate results. Upon resuming, the state saved in the memory data block is used to resume at the correction point. Furthermore, if the initial part of the memory data block, including the state save area, does not exist or is not accessible, a page fault is triggered. For loads and stores to the local storage area, a page fault is reported in the normal manner, i.e., on the first access to a page that does not exist or is not accessible. In some instances, non-accelerator processing elements are used upon resumption.

If the block completes successfully at 2611, then the discarded registers are returned to their original state along with the flags. Only the memory state is different after the block.

Figure 27 shows an example of a method of operation in different modes for execution using ABEGIN/AEND. Typically, the method is performed by a combination of entities, e.g., a binary converter and an execution circuit.

At 2701, a different mode of execution is entered, such as a relaxed mode of execution (with or without an accelerator). Typically, this mode is entered through execution of an ABEGIN instruction. However, this mode may also be entered through a pattern match, as described in detail above. Entering this mode includes resetting the sequence numbers.

At 2703, a saved state area is written, e.g., general and packed data registers that are modified, associated flags, and additional implementation-specific information. This area allows execution to be resumed or rolled back if something goes wrong in the block (e.g., an interrupt).

At 2705, a local storage area is reserved for each parallel execution instance. The size of this area is indicated by the state save area field, as described in detail above.

At 2706, the code within the block is converted for execution.

At 2707, during execution of the transformed block, the progress of the block is tracked. For example, if an instruction is retired after successful execution, the sequence number of the block is updated.

At 2709, a determination is made as to whether the AEND instruction has arrived (e.g., to determine whether the block is complete). If the AEND instruction has not arrived, then at 2713, the local storage area is updated with the intermediate results. If possible, execution picks up from these results. However, in some instances, at 2715, a rollback to before the ABEGIN/AEND occurs. For example, if an exception or interrupt occurs during execution of the ABEGIN/AEND block, the instruction pointer points to the ABEGIN instruction and the R9 register points to the memory data block that is updated with the intermediate results. Upon resuming, the state saved in the memory data block is used to resume at the correction point. Furthermore, if the initial portion of the memory data block, including the state save area, does not exist or is not accessible, a page fault is triggered. For loads and stores to the local storage area, a page fault is reported in the normal manner, i.e., on the first access to a page that does not exist or is not accessible. In some instances, non-accelerator processing elements are used upon resumption.

If the block completes successfully, then at 2711, the discarded registers are returned to their original state along with the flags. Only the memory state is different after the block.

As mentioned above, in some embodiments, a common link (called a Multi-Protocol Common Link (MCL)) is used to reach devices (e.g., the processing elements described in Figures 1 and 2). In some embodiments, these devices are seen as PCI Express (PCIe) devices. This link has three or more protocols that are dynamically multiplexed on the link. For example, the common link supports protocols consisting of: 1) a Producer/Consumer, Discover, Configure, Interrupt (PDCI) protocol that enables device discovery, device configuration, error reporting, interrupts, DMA-style data transfers, and various services as may be defined in one or more proprietary or industry standards (e.g., PCI Express specifications or equivalent alternatives); 2) a Caching Agent Coherence (CAC) protocol that allows devices to issue coherent read and write requests to processing elements; and 3) a Memory Access (MA) protocol that allows a processing element to access the local memory of another processing element. While specific examples of these protocols are provided (e.g., Intel® On-Chip System Fabric (IOSF), In-Die Interconnect (IDI), Scalable Memory Interconnect 3+ (SMI3+)), the principles underlying the invention are not limited to any particular set of protocols.

120 is a simplified block diagram 12000 illustrating an exemplary multi-chip configuration 12005 including two or more chips or dies (e.g., 12010, 12015) communicatively connected using an exemplary multi-chip link (MCL) 12020. While FIG. 120 illustrates an example of two (or more) dies interconnected using an exemplary MCL 12020, it should be understood that the principles and features described herein with respect to MCL implementations may be applied to any interconnect or link connecting a die (e.g., 12010) and other components, including connecting two or more dies (e.g., 12010, 12015), connecting a die (or chip) to another component off-die, connecting a die to another device or die-off-package (e.g., 12005), connecting a die to a BGA package, patch on interposer implementations (POINT), among other potential examples.

In some examples, the larger components (e.g., die 12010, 12015) may themselves be IC systems, such as, for example, a system on a chip (SoC), a multiprocessor chip, or other components including multiple components (12026-12030 and 12040-12045) such as cores, accelerators, etc. on a device, e.g., on a single die (e.g., 12010, 12015). MCL 12020 provides flexibility to build complex and diverse systems from potentially multiple separate components and systems. By way of example, each of die 12010, 12015 may be manufactured or otherwise provided by two different entities. Additionally, the dies and other components may themselves include interconnects or other communications fabrics (e.g., 12031, 12050) that provide an infrastructure for communications between components (e.g., 12026-12030 and 12040-12045) within a device (e.g., 12010, 12015, respectively). The various components and interconnects (e.g., 12031, 12050) may support or use multiple different protocols. Additionally, communications between dies (e.g., 12010, 12015) may potentially include transactions between various components on the die via multiple different protocols.

Embodiments of the multi-chip link (MCL) support multiple package options, multiple I/O protocols, and reliability, availability, and serviceability (RAS) features. Additionally, the physical layer (PHY) can include physical electrical and logical layers and can support longer channel lengths, including channel lengths up to and in some cases greater than about 45 mm. In some implementations, the exemplary MCL can operate at high data rates, including data rates in excess of 8-10 Gb/s.

In one exemplary embodiment of the MCL, the PHY electrical layer improves upon traditional multi-channel interconnect solutions (e.g., multi-channel DRAM I/O) to, for example, extend data rates and channel configurations with numerous features including, for example, tuned mid-rail termination, low power active crosstalk cancellation, circuit redundancy, per-bit duty cycle correction and deskew, line coding and transmitter equalization, among other potential examples.

In one exemplary embodiment of the MCL, the PHY logical layer is implemented to further support cases (e.g., electrical layer functionality) that extend data rates and channel configurations while also allowing interconnects to transport multiple protocols across the electrical layer. Such an embodiment provides and defines a modular common physical layer that is protocol independent and potentially designed to work with any existing or future interconnection protocol.

121, a simplified block diagram 12100 is shown depicting at least a portion of a system including an exemplary implementation of a multi-chip link (MCL). The MCL may be implemented using physical electrical connections (e.g., wires implemented as lanes) connecting a first device 12105 (e.g., a first die including one or more subcomponents) with a second device 12110 (e.g., a second die including one or more other subcomponents). In the specific example shown in the high-level representation of FIG. 12100, all signals (within channels 12115, 12120) may be unidirectional, and the lanes may provide data signals having both upstream and downstream data transfer. While the block diagram 12100 of FIG. 121 refers to the first component 12105 as the upstream component, the second component 12110 as the downstream component, the physical lanes of the MCL used when transmitting data as the downstream channel 12115, and the lanes used to receive data (from component 12110) as the upstream channel 12120, it should be understood that the MCL between devices 12105, 12110 can be used by each device to both transmit and receive data between the devices.

In one exemplary embodiment, the MCL can provide a physical layer including electrical MCL physical layers (PHYs) 12125a,b (or collectively 12125) and executable logical implementations MCL logical PHYs 12130a,b (or collectively 12130). The electrical or physical PHYs 12125 provide the physical connection over which data is communicated between the devices 12105, 12110. Signal conditioning components and logic can be implemented in association with the physical PHYs 12125 to establish high data rates and channel configuration capabilities of the link, with some applications relating to closely clustered physical connections with lengths of approximately 45 mm or longer. The logical PHYs 12130 include circuitry for facilitating clocking, link state management (e.g., link layers 12135a, 12135b), and protocol multiplexing between potentially multiple different protocols used to communicate over the MCL.

In one exemplary embodiment, the physical PHY 12125 includes a set of data lanes for each channel (e.g., 12115, 12120) along which in-band data is transmitted. In this example, 50 data lanes are provided for each of the upstream and downstream channels 12115, 12120, although other numbers of lanes may be used as permitted by layout and power constraints, desired application, device constraints, etc. Each channel may further include one or more dedicated lanes for strobe or clock signals for the channel, one or more dedicated lanes for enable signals for the channel, one or more dedicated lanes for stream signals, and one or more dedicated lanes for link state machine management or sideband signals. The physical PHY may further include a sideband link 12140, which in some examples may be a bidirectional low frequency control signal link used to coordinate state transitions and other attributes for the MCLs connecting the devices 12105, 12110, among other examples.

As mentioned above, multiple protocols are supported using the MCL implementation. In fact, multiple independent transaction layers 12150a, 12150b may be provided in each device 12105, 12110. By way of example, each device 12105, 12110 may support and utilize two or more protocols, such as PCI, PCIe, CAC, among others. CAC is a coherent protocol used on-die to communicate between the cores, last level cache (LLC), memory, graphics and I/O controllers. Other protocols may also be supported, including Ethernet protocols, InfiniBand protocols and other PCIe fabric-based protocols. The combination of logical PHY and physical PHY may also be used as an inter-die interconnect, connecting a SerDes PHY (PCIe, Ethernet, InfiniBand or other high-speed SerDes) on one die to its upper layer implemented on another die, among other examples.

The logical PHY 12130 supports multiplexing between these multiple protocols in the MCL. As an example, a dedicated stream lane can be used to assert an encoded stream signal that identifies which protocol applies to data transmitted substantially simultaneously on the data lanes of the channel. Additionally, the logical PHY 12130 negotiates the various types of link state transitions that the various protocols may support or require. In some examples, the LSM_SB signal transmitted over the dedicated LSM_SB lane of the channel can be used together with the sideband link 12140 to communicate and negotiate link state transitions between the devices 12105, 12110. Additionally, link training, error detection, skew detection, deskew, and other functions of conventional interconnects can be replaced or controlled in part using the logical PHY 12130. By way of example, the valid signal transmitted over one or more dedicated valid signal lanes in each channel may be used to signal link activity, detect skew and link errors, and implement other features, among other examples. In the example of FIG. 121, multiple valid lanes are provided per channel. By way of example, data lanes within a channel may be bundled or clustered (physically and/or logically) and a valid lane may be provided per cluster. Additionally, multiple strobe lanes may be provided in some cases to provide a dedicated strobe signal for each cluster within multiple data lane clusters in a channel, among other examples.

As described above, the logical PHY 12130 negotiates and manages link control signals transmitted between devices connected by the MCL. In some embodiments, the logical PHY 12130 includes a link layer packet (LLP) generation circuit 12160 that transmits link layer control messages (i.e., in-band) over the MCL. Such messages may be transmitted over the data lanes of the channel, with the stream lanes identifying the data as link layer-to-link layer messaging, such as link layer control data, among other examples. Link layer messages enabled using the LLP module 12160 support the negotiation and operation of link layer state transitions, power management, loopback, disable, recentering and scrambling, among other link layer features between the link layers 12135a, 12135b of the devices 12105, 12110, respectively.

122, a simplified block diagram 12200 is shown illustrating an example logical PHY of an example MCL. The physical PHY 12205 may be connected to a die that includes the logical PHY 12210 and additional logic to support the link layer of the MCL. The die may further include logic to support multiple different protocols over the MCL in this example. By way of example, in the example of FIG. 122, PCIe logic 12215 is provided along with CAC logic 12220, such that the die can communicate using either PCIe or CAC over the same MCL connecting the two dies, among many other potentially examples including examples where more than two protocols, or protocols other than PCIe and CAC, are supported over the MCL. The various protocols supported between the dies may provide varying levels of service and features.

The logical PHY 12210 may include link state machine management logic 12225 for negotiating link state transitions in conjunction with requests of the upper layer logic of the die (e.g., received via PCIe or CAC). In some embodiments, the logical PHY 12210 may further include link test and debug logic (e.g., 12230). As described above, the exemplary MCL may support control signals transmitted between dies via the MCL to facilitate protocol-independent, high-performance, and power-efficient functioning of the MCL (among other exemplary functions). By way of example, the logical PHY 12210 may support the generation and transmission, as well as the reception and processing of enable signals, stream signals, and LSM sideband signals in conjunction with the transmission and reception of data over dedicated data lanes, as described in the examples above.

In some embodiments, multiplexing (e.g., 12235) and demultiplexing (e.g., 12240) logic may be included in logical PHY 12210 or may be otherwise accessible to logical PHY 12210. By way of example, multiplexing logic (e.g., 12235) may be used to identify data (e.g., embodied as packets, messages, etc.) to be transmitted on the MCL. Multiplexing logic 12235 may identify the protocol governing the data and generate a stream signal encoded to identify the protocol. By way of example, in one exemplary embodiment, the stream signal may be encoded as two hexadecimal symbols of one byte (e.g., CAC: FFh; PCIe: F0h; LLP: AAh; Sideband: 55h, etc.) and transmitted during the same window (e.g., one byte time period window) for data governed by the identified protocol. Similarly, the demultiplexing logic 12240 can be used to interpret the arriving stream signal to decode the stream signal and identify the protocol that applies to the data received simultaneously with the stream signal on the data lane. The demultiplexing logic 12240 can then apply (or ensure) protocol-specific link layer processing and have the data processed by the corresponding protocol logic (e.g., PCIe logic 12215 or CAC logic 12220).

The logical PHY 12210 may further include link layer packet logic 12250 that may be used to handle various link control functions including power management tasks, loopback, disabling, recentering, scrambling, etc. The LLP logic 12250 may facilitate link layer to link layer messages via the MCLP, among other functions. Data corresponding to LLP signaling may also be identified by stream signals transmitted on dedicated stream signal lanes that are encoded to identify that data lane LLP data. Multiplexing and demultiplexing logic (e.g., 12235, 12240) may also be used to generate and interpret stream signals corresponding to LLP traffic and to have such traffic processed by the appropriate die logic (e.g., LLP logic 12250). Similarly, some embodiments of the MCLP may include dedicated sidebands (e.g., sideband 12255 and supporting logic), such as asynchronous and/or low frequency sideband channels, among other examples.

The logical PHY logic 12210 may further include link state machine management logic that can generate and receive (and use) link state management messaging via a dedicated LSM sideband lane. By way of example, the LSM sideband lane may be used to perform handshaking to proceed to a link training state and exit a power management state (e.g., an L1 state), among other potential examples. The LSM sideband signals may be asynchronous signals in that they are not aligned with the data, valid, and stream signals of the link, but instead correspond to signaling state transitions and coordinate link state machines between two dies or chips connected by the link, among other examples. Providing a dedicated LSM sideband lane may, in some examples, allow conventional squelch and receive detection circuitry in an analog front end (AFE) to be eliminated, among other exemplary benefits.

Referring to FIG. 123, a simplified block diagram 12300 is shown illustrating another representation of logic used to implement the MCL. By way of example, the logical PHY 12210 is provided with a defined logical PHY interface (LPIF) 12305 through which any one of a number of different protocols (e.g., PCIe, CAC, PDCI, MA, etc.) 12315, 12320, 12325 and signaling modes (e.g., sideband) can interface with the physical layer of the exemplary MCL. In some embodiments, the multiplexing and arbitration logic 12330 can also be provided as a layer separate from the logical PHY 12210. In one example, the LPIF 12305 can be provided as an interface on both sides of this MuxArb layer 1230. The logical PHY 12210 can interface with a physical PHY (e.g., the analog front end (AFE) 12205 of the MCL PHY) through another interface.

The LPIF can remove the PHY (logical and electrical/analog) from the higher layers (e.g., 12315, 12320, 12325) so that an entirely different PHY can be implemented under the LPIF transparent to the higher layers. This helps promote modularity and allows reuse in designs, and allows the higher layers to remain intact when the underlying signaling technology PHY is updated, among other examples. Additionally, the LPIF can define a number of signals that enable multiplexing/demultiplexing, LSM management, error detection and handling, and other functions of the logical PHY. By way of example, the following table summarizes at least some of the signals that may be defined for an exemplary LPIF:

As mentioned in the table, in some embodiments, an alignment mechanism can be provided through an AlignReq/AlignAck handshake. For example, some protocols may lose packet framing when the physical layer goes into recovery. The alignment of the packet can be corrected, for example, to ensure correct framing identification by the link layer. The physical layer can assert a StallReq signal when it goes into recovery, which causes the link layer to assert a stall signal when it is ready to forward the newly aligned packet. The physical layer logic can sample both stall and valid to determine if the packet is aligned. For example, the physical layer can continue to drive trdy to drain the link layer packet until stall and valid are sampled and asserted, among other potential implementations, including other alternative implementations that use valid to assist in packet alignment.

Various fault tolerances may be defined for signals on the MCL. By way of example, fault tolerances may be defined for active, stream, LSM sideband, low frequency sideband, link layer packets, and other types of signals. Fault tolerance for packets, messages, and other data transmitted over the dedicated data lanes of the MCL may be based on the particular protocol governing the data. In some embodiments, error detection and handling mechanisms may be provided, such as cyclic redundancy checks (CRCs), retry buffers, and the like, among other potential examples. By way of example, for PCIe packets transmitted over the MCL, a 32-bit CRC may be used for PCIe transaction layer packets (TLPs) (with guaranteed delivery (e.g., through a replay mechanism)) and a 16-bit CRC may be used for PCIe link layer packets (which may be designed to be lossy (e.g., no replays are applied)). Additionally, for PCIe framing tokens, a particular Hamming distance (e.g., a Hamming distance of four (4)), among other examples, may be defined for the token identifier, and parity and a 4-bit CRC may also be utilized. On the other hand, for CAC packets, a 16-bit CRC may be utilized.

In some embodiments, fault tolerance is defined for link layer packets (LLPs) that utilize a valid signal to transition from low to high (i.e., 0 to 1) (e.g., to aid in guarantee bit and symbol lock). Additionally, in one example, a certain number of consecutive identical LLPs may be defined to be transmitted, among other defined characteristics that may be used as a basis for determining a fault in the LLP data on the MCL, and a response may be expected for each request, with the requester retrying after a response times out. In a further example, fault tolerance may be provided for a valid signal, e.g., extending over an entire time period window or symbol through the valid signal (e.g., by holding the valid signal high for 8 UIs). Additionally, errors or faults in the stream signal may be prevented by, among other examples, maintaining a Hamming distance for the encoding values of the stream signal.

The logical PHY implementation includes error detection, error reporting, and error handling logic. In some embodiments, the logical PHY of an example MCL may include logic to detect PHY layer deframing errors (e.g., on valid and stream lanes), sideband errors (e.g., related to LSM state transitions), errors in the LLP (e.g., critical to LSM state transitions), among other examples. Some error detection/resolution may be delegated to higher layer logic, such as PCIe logic adapted to detect PCIe specific errors, among other examples.

In the case of a deframing error, in some embodiments, one or more mechanisms may be provided through the error handling logic. The deframing error may be handled based on the associated protocol. As an example, in some embodiments, the link layer may signal the error to trigger a retry. The deframing may also cause a re-realignment of the logical PHY deframing. Additionally, among other techniques, a re-centering of the logical PHY may be performed and symbol/window lock may be reacquired. Centering may include, in some examples, the PHY moving the receiver clock phase to a point that is optimal for detecting arriving data. "Optimal" in this context may refer to the most tolerant to noise and clock jitter. Among other examples, re-centering may include a simple centering function that is performed, for example, when the PHY wakes up from a low power state.

Other types of errors may be associated with other error handling techniques. As an example, an error detected in the sideband may be caught through a timeout mechanism of the corresponding state (e.g., in the LSM). The error may be logged and then the link state machine may be transitioned to reset. The LSM may be kept in the reset state until a resume command is received from the software. In another example, an LLP error, e.g., a link control packet error, may be handled using a timeout mechanism that allows the LLP sequence to be resumed if no acknowledgement for the LLP sequence is received.

In some embodiments, each of the above protocols is a variation of PCIe. PCIe devices communicate using a common address space associated with a bus. This address space is the bus address space or the PCIe address space. In some embodiments, the PCIe devices use addresses in an internal address space, which may be different from the PCIe address space.

The PCIe specification defines a mechanism by which a PCIe device can expose its local memory (or a portion of it) to the bus, thus enabling the CPU, or other devices attached to the bus, to directly access that memory. Typically, each PCIe device is assigned a dedicated region in the PCIe address space called the PCI Base Address Register (BAR). Furthermore, the addresses that the device exposes are mapped to respective addresses in the PCI BAR.

In some embodiments, a PCIe device (e.g., an HCA) uses an Input/Output Memory Mapping Unit (IOMMU) to translate between its internal addresses and PCIe bus addresses. In other embodiments, a PCIe device may use a PCI Address Translation Service (ATS) to perform address translation and resolution. In some embodiments, a tag, e.g., a Process Address Space ID (PASID) tag, is used to define addresses that are translated to belong to a particular process's virtual address space.

Figure 28 shows additional details regarding one embodiment. As with the embodiments described above, this embodiment includes an accelerator 2801 having accelerator memory 2850 coupled via a multi-protocol link 2800 to a host processor 2802 having host memory 2860. As previously mentioned, the accelerator memory 2850 may utilize a different memory technology than the host memory 2860 (e.g., the accelerator memory may be HBM or stacked DRAM, while the host memory may be SDRAM).

Multiplexers 2811 and 2812 are shown to highlight the fact that multi-protocol link 2800 is a dynamically multiplexed bus supporting PCDI, CAC and MA protocol (e.g., SMI3+) traffic, each of which may be forwarded to different functional components within accelerator 2801 and host processor 2802. By way of example and not limitation, these protocols may include IOSF, IDI and SMI3+. In one embodiment, accelerator 2801's PCIe logic 2820 includes a local TLB 2822 for caching virtual-to-physical address translations for use by one or more accelerator cores 2830 when executing commands. As previously mentioned, virtual memory space is distributed between accelerator memory 2850 and host memory 2860. Similarly, the PCIe logic on the host processor 2802 includes an I/O memory management unit (IOMMU) 2810 for managing memory access for PCIe I/O devices 2806 and, in one embodiment, the accelerator 2801. As shown, the PCIe logic on the accelerator 2820 and the PCIe logic on the host processor 2808 communicate using the PCDI protocol to perform functions such as device discovery, register access, device configuration and initialization, interrupt handling, DMA handling, and address translation services (ATS). As previously mentioned, the IOMMU 2810 on the host processor 2802 may operate primarily for the purpose of controlling and coordinating these functions.

In one embodiment, the accelerator core 2830 includes a processing engine (element) that performs the functions required by the accelerator. Additionally, the accelerator core 2830 may include a host memory cache 2834 for locally caching pages stored in the host memory 2860, and an accelerator memory cache 2832 for caching pages stored in the accelerator memory 2850. In one embodiment, the accelerator core 2830 communicates with the coherence and cache logic 2807 of the host processor 2802 via a CAC protocol to ensure that cache lines shared between the accelerator 2801 and the host processor 2802 remain coherent.

The bias/coherence logic 2840 of the accelerator 2801 implements various device/host bias techniques (e.g., page-level granularity) described herein to ensure data coherency while reducing unnecessary communication over the multi-protocol link 2800. As shown, the bias/coherence logic 2840 communicates with the coherence and cache logic 2807 of the host processor 2802 using MA memory transactions (e.g., SMI3+). The coherence and cache logic 2807 is responsible for maintaining coherency of data stored in its LLC 2809, host memory 2860, accelerator memory 2850 and caches 2832, 2834, and each of the individual caches of the core 2805.

In summary, one embodiment of the accelerator 2801 appears as a PCIe device to software running on the host processor 2802 and is accessed via the PDCI protocol (effectively reformatting the PCIe protocol for a multiplexed bus). The accelerator 2801 may participate in shared virtual memory using an accelerator device TLB and standard PCIe address translation services (ATS). The accelerator may also be treated as a coherence/memory agent. Certain functions (e.g., ENQCMD, MOVDIR, described below) are available on the PDCI (e.g., for work submission), while the accelerator may cache host data at the accelerator using the CAC and in certain bias transition flows. Accesses from the host to the accelerator memory (or host bias accesses from the accelerator) may use the MA protocol, as described.

As shown in FIG. 29, in one embodiment, the accelerator includes PCI configuration registers 2902 and MMIO registers 2906 that can be programmed to provide access to device back-end resources 2905. In one embodiment, the base address for the MMIO registers 2906 is specified by a set of base address registers (BARs) 2901 in the PCI configuration space. Unlike previous implementations, one embodiment of the Data Streaming Accelerator (DSA) described herein does not implement multiple channels or PCI functionality, so there is only one instance of each register in the device. However, there may be more than one DSA device in a single platform.

Examples may provide additional feature or debug registers not described herein. Any such registers should be considered implementation specific.

PCI configuration space accesses are performed as aligned 1-byte, 2-byte, or 4-byte accesses. Refer to the PCI Express Base specification for rules on accessing unimplemented registers and reserved bits in PCI configuration space.

MMIO space accesses to the BAR0 region (Capabilities, Configuration, and Status Registers) are performed as aligned 1-byte, 2-byte, 4-byte, or 8-byte accesses. 8-byte accesses should only be used for 8-byte registers. Software should not read or write unimplemented registers. MMIO space accesses to the BAR2 and BAR4 regions should be performed as 64-byte accesses using the ENQCMD, ENQCMDS, or MOVDIR64B instructions (described in more detail below). ENQCMD or ENQCMDS should be used to access work queues (SWQs) that are configured as shared, and MOVDIR64B must be used to access work queues (DWQs) that are configured as dedicated.

One embodiment of the DSA PCI configuration space implements three 64-bit BARs 2901. The Device Control Register (BAR0) is a 64-bit BAR that contains the physical base address of the device control registers. These registers provide information about device capabilities, controls to configure and enable the device, and device status. The size of the BAR0 area depends on the size of the interrupt message storage 2904. The size is the number of interrupt message storage entries 2904 times 16 plus 32KB rounded up to the next power of two. For example, if a device supports 1024 interrupt message storage entries 2904, then the interrupt message storage is 16KB and the size of BAR0 is 64KB.

BAR2 is a 64-bit BAR that contains the physical base addresses of the privileged and non-privileged portals. Each portal is 64 bytes in size and located on a separate 4KB page. This allows portals to be independently mapped to different address spaces using the CPU page tables. Portals are used to submit descriptors to the device. Privileged portals are used by kernel mode software and non-privileged portals are used by user mode software. The number of non-privileged portals is the same as the number of work queues supported. The number of privileged portals is the number of work queues (WQs) x (size of MSI-X table - 1). The address of the portal used to submit the descriptor allows the device to determine which WQ to place the descriptor in, whether the portal is privileged or non-privileged, and which MSI-X table entry can be used for completion interrupts. For example, if a device supports 8 WQs, the WQ for a given descriptor is (portal address >> 12) and 0x7. If Portal Address >> 15 is 0, the portal is not privileged. Otherwise, the portal is privileged and the MSI-X2903 table index used for the completion interrupt is Portal Address >> 15. Bits 5:0 must be 0. Bits 11:6 are ignored. Thus, any 64-byte aligned address on the page can be used with the same effect.

Descriptor submissions using a non-privileged portal are subject to the WQ occupancy threshold if configured using the Work Queue Configuration (WQCFG) register. Descriptor submissions using a privileged portal are not subject to the threshold. Descriptor submissions for SWQ must be submitted using ENQCMD or ENQCMDS. Other write operations to SWQ portals are ignored. Descriptor submissions for DWQ must be submitted using 64-byte write operations. Software uses MOVDIR64B to ensure contiguous 64-byte writes. ENQCMD or ENQCMDS to a disabled or dedicated WQ portal returns a retry. Other write operations to DWQ portals are ignored. Any read operation to the BAR2 address space returns all ones. Kernel mode descriptors should be submitted using a privileged portal to receive a completion interrupt. If a kernel mode descriptor is submitted using a non-privileged portal, no completion interrupt can be requested. A user mode descriptor may be submitted using either a privileged or non-privileged portal.

The number of portals in the BAR2 area is the number of WQs supported by the device x the size of the MSI-X2903 table. The size of the MSI-X table is typically the number of WQs plus one. So, for example, if a device supports 8 WQs, the useful size of BAR2 would be 8 x 9 x 4KB = 288KB. The total BAR2 size would be rounded up to the next power of 2, or 512KB.

BAR4 is a 64-bit BAR that contains the physical base address of the guest portal. Each guest portal is 64 bytes in size and is located on a separate 4KB page. This allows portals to be independently mapped into different address spaces using the CPU Extended Page Tables (EPT). If the Interrupt Message Storage Support field in GENCAP is 0, then this BAR is not implemented.

The guest portal may be used by guest kernel mode software to submit descriptors to the device. The number of guest portals is the number of entries in the interrupt message storage times the number of WQs supported. The address of the guest portal used to submit the descriptor allows the device to determine the WQ for the descriptor, and the interrupt message storage entry can be used to generate a completion interrupt for the descriptor completion (if it is a kernel mode descriptor and the request completion interrupt flag is set in the descriptor). For example, if a device supports 8 WQs, the WQs for a given descriptor are (guest portal address >> 12) and 0x7, and the interrupt table entry index used for the completion interrupt is guest portal address >> 15.

In one embodiment, MSI-X is the only PCIe interrupt function that the DSA provides and where the DSA does not implement legacy PCI interrupts or MSI. Details of this register structure are in accordance with the PCI Express specification.

In one embodiment, three PCI Express functions control address translation. Only certain combinations of values of these functions can be supported, as shown in Table A. The values are checked when the enable bit in the general control register (GENCTRL) is set to 1.

If any of these functions are changed by software while the device is enabled, the device may be halted and an error reported to the software error register.

In one embodiment, software configures the PASID function to control whether the device performs address translation using PASID. If PASID is disabled, only physical addresses may be used. If PASID is enabled, virtual or physical addresses may be used depending on the IOMMU configuration. If PASID is enabled, both address translation services (ATS) and page request services (PRS) should be enabled.

In one embodiment, software configures the ATS function to control whether the device should translate addresses before performing a memory access. If address translation is enabled in IOMMU 2810, ATS must be enabled in the device to obtain acceptable system performance. If address translation is not enabled in IOMMU 2810, ATS must be disabled. If ATS is disabled, only physical addresses may be used and all memory accesses are performed using untranslated accesses. If PASID is enabled, ATS must be enabled.

In one embodiment, software configures the PRS feature to control whether a device can request a page if address translation fails. If PASID is enabled, PRS must be enabled, and if PASID is disabled, PRS must be disabled.

Some embodiments utilize a virtual memory space that is seamlessly shared between one or more processor cores, accelerator devices, and/or other types of processing devices (e.g., I/O devices). In particular, one embodiment utilizes a shared virtual memory (SVM) architecture in which the same virtual memory space is shared between cores, accelerator devices, and/or other processing devices. Additionally, some embodiments include a heterogeneous form of physical system memory that is addressed using a common virtual memory space. The heterogeneous form of physical system memory may use different physical interfaces to interface with the DSA architecture. For example, the accelerator device may be directly coupled to a local accelerator memory, such as a high bandwidth memory (HBM), and each core may be directly coupled to a host physical memory, such as a dynamic random access memory (DRAM). In this example, the shared virtual memory (SVM) is mapped to the combined physical memory of the HBM and DRAM such that the accelerator, processor cores, and/or other processing devices can access the HBM and DRAM using a consistent set of virtual memory addresses.

These and other features of the accelerator are described in detail below. By way of overview, different implementations may include one or more of the following infrastructure features:

Shared Virtual Memory (SVM): Some embodiments support SVM, which allows user-level applications to submit commands directly to the DSA using virtual addresses in the descriptors. The DSA may support translating virtual addresses to physical addresses using an Input/Output Memory Management Unit (IOMMU), including handling page faults. The virtual address range referenced by a descriptor may span multiple pages distributed across multiple heterogeneous memory types. Additionally, one embodiment also supports the use of physical addresses as long as the data buffer is contiguous in physical memory.

Partial Descriptor Completion: With SVM support, an operation may encounter a page fault during address translation. In some cases, the device may terminate processing of the corresponding descriptor upon encountering the fault and provide a completion record to the software indicating partial completion and fault information, allowing the software to take remedial action and retry the operation after the fault is resolved.

Batch Processing: Some embodiments support submitting descriptors in "batches". A batch descriptor points to a set of substantially contiguous work descriptors (i.e., descriptors that contain the actual data processing). When processing a batch descriptor, the DSA fetches the work descriptors from a specified memory and processes them.

Stateless Device: Descriptors in one embodiment are designed such that all information needed to process the descriptor is in the descriptor payload itself. This allows the device to store little client-specific state which improves its scalability. One exception, if used, that is configured by trusted software is the completion interrupt message.

Cache Allocation Control: This allows the application to specify whether to write to the cache or to bypass the cache and write directly to memory. In one embodiment, completion records are always written to the cache.

Shared Work Queue (SWQ) Support: As described in more detail below, some embodiments support scalable work submission through a shared work queue (SWQ) using the enqueue command (ENQCMD) and enqueue command (ENQCMDS) instructions. In this embodiment, the SWQ is shared by multiple applications.

Dedicated Work Queue (DWQ) Support: In some embodiments, there is support for high throughput work submission through a Dedicated Work Queue (DWQ) using the MOVDIR64B instruction. In this embodiment, the DWQ is dedicated to a particular application.

QoS Support: Some embodiments allow a quality of service (QoS) level to be specified per work queue (e.g., by a kernel driver). Different work queues may then be assigned to different applications, allowing work from different applications to be dispatched from the work queue with different priorities. Work queues can be programmed to use specific channels for fabric QoS.

Biased cache coherence mechanism

In one embodiment, direct attached memory, such as stacked DRAM or HBM, is used to improve accelerator performance and simplify application development for applications that use accelerators with directly attached memory. In this embodiment, accessor attached memory is mapped as part of the system memory, allowing it to be accessed using shared virtual memory (SVM) techniques (e.g., as used in current IOMMU implementations), but without incurring the typical performance penalties associated with full system cache coherence.

The ability to access accessor-attached memory as part of the system memory without onerous cache coherence overhead provides a beneficial operating environment for accelerator offload. The ability to access memory as part of the system address mapping allows host software to set up operands and access computation results without the overhead of traditional I/O DMA data copies. Such traditional copies involve driver calls, interrupts, and memory-mapped I/O (MMIO) accesses, which are all inefficient compared to simple memory accesses. At the same time, the ability to access accessor-attached memory without cache coherence overhead can be critical to the execution time of the offloaded computation. In cases involving substantial streaming write memory traffic, for example, cache coherence overhead can cut the effective write bandwidth seen by the accelerator in half. The efficiency of operand setup, the efficiency of the resulting accesses, and the efficiency of the accelerator computation all play a role in determining how well the accelerator offload works. The cost of the offload function (e.g., setting up the operands and getting the result) may be so high that offloading may not be effective at all, or may limit the accelerator to only very large jobs. The efficiency with which the accelerator performs the computation may have the same effect.

In one embodiment, different memory access and coherence techniques are applied depending on the entity initiating the memory access (e.g., accelerator, core, etc.) and the memory being accessed (e.g., host memory or accelerator memory). These techniques are generally referred to as "coherence bias" mechanisms that provide accessor-attached memory with two sets of cache coherence flows, one optimizing efficient accelerator accesses to its attached memory and the second optimizing host accesses to accessor-attached memory and shared accelerator/host accesses to accessor-attached memory. It further includes two techniques for switching between these flows, one driven by application software and the other driven by independent hardware hints. In both sets of coherence flows, the hardware maintains full cache coherence.

As generally shown in FIG. 30, one embodiment applies a computer system including an accelerator 3001 and one or more computer processor chips having a processor core and I/O circuitry 3003, the accelerator 3001 being coupled to the processor via a multi-protocol link 2800. In one embodiment, the multi-protocol link 3010 is a dynamically multiplexed link supporting multiple different protocols, including but not limited to those detailed above. However, it should be noted that the principles underlying the present invention are not limited to any particular set of protocols. It should also be noted that the accelerator 3001 and core I/O 3003 may be integrated on the same semiconductor chip or different semiconductor chips, depending on the implementation.

In the illustrated embodiment, an accelerator memory bus 3012 couples the accelerator 3001 to the accelerator memory 3005, and a separate host memory bus 3011 couples the core I/O 3003 to the host memory 3007. As previously mentioned, the accelerator memory 3005 may comprise high bandwidth memory (HBM) or stacked DRAM (some examples of which are described herein), and the host memory 3007 may comprise DRAM, for example, double data rate synchronous dynamic random access memory (e.g., DDR3 SDRAM, DDR4 SDRAM, etc.). However, the principles underlying the present invention are not limited to any particular type of memory or memory protocol.

In one embodiment, both the accelerator 3001 and the "host" software executing on the processing cores within the processor chip 3003 access the accelerator memory 3005 using two separate sets of protocol flows, referred to as "host biased" flows and "device biased" flows. As described below, one embodiment supports multiple options for modulating and/or selecting the protocol flows for a particular memory access.

The coherence bias flow is implemented in part on two protocol layers of the multi-protocol link 3010 between the accelerator 3001 and one of the processor chips 3003: the CAC protocol layer and the MA protocol layer. In one embodiment, the coherence bias flow is enabled by (a) using existing opcodes in the CAC protocol in a new way, (b) adding new opcodes to the existing MA standard, and (c) adding support for the MA protocol to the multi-protocol link 3001 (before the link included only CAC and PCDI). Note that the multi-protocol link is not limited to just supporting CAC and MA. In one embodiment, it is merely required to support at least those protocols.

As used herein, the "host biased" flows shown in FIG. 30 are a set of flows that funnel all requests, including requests from the accelerator itself, to the accelerator memory 3005 through a standard coherence controller 3009 in the processor chip 3003 on which the accelerator 3001 is attached. This forces the accelerator 3001 to take a detour to access its own memory, but allows accesses from both the accelerator 3001 and the processor core I/O 3003 to be kept coherent using the processor's standard coherence controller 3009. In one embodiment, the flows issue requests to the processor's coherence controller 3009 over a multi-protocol link using a CAC opcode in the same or similar manner that the processor core 3009 issues requests to the coherence controller 3009. For example, the processor chip's coherence controller 3009 may issue UPI and CAC coherence messages (e.g., snoops) resulting from requests from the accelerator 3001 to all peer processor core chips (e.g., 3003) and internal processor agents on behalf of the accelerator, to the same extent as would be the case for requests from the processor core 3003. In this manner, coherency is maintained between data accessed by the accelerator 3001 and the processor core I/O 3003.

In one embodiment, the coherence controller 3009 also conditionally issues memory access messages to the accelerator's memory controller 3006 over the multiprotocol link 2800. These messages also force the data to be returned to the processor's coherence controller 3009 over the multiprotocol link 2800, as well as messages containing new opcodes that the coherence controller 3009 sends to the memory controller local to these processor dies, allowing the data to be returned directly to an agent internal to the accelerator 3001, instead of being returned to the accelerator 3001 as a CAC response over the multiprotocol link 2800.

In one embodiment of the "host bias" mode shown in FIG. 30, all requests from the processor core 3003 targeting the accessor attached memory 3005 are sent directly to the processor coherency controller 3009, just as they would have been for normal host memory 3007. The coherence controller 3009 may apply these standard cache coherence algorithms and send these standard cache coherence messages just as they would for accesses from the accelerator 3001 and just as they would for accesses to normal host memory 3007. The coherence controller 3009 also conditionally sends MA commands over the multiprotocol link 2800 for this class of request, but in this case the MA flow returns data across the multiprotocol link 2800.

The "device bias" flows shown in FIG. 31 are flows that allow the accelerator 3001 to access its local attached memory 3005 without consulting the host processor's cache coherence controller 3007. More specifically, these flows allow the accelerator 3001 to access its local attached memory through the memory controller 3006 without sending a request over the multiprotocol link 2800.

In "device bias" mode, requests from the processor core I/O 3003 are issued as described for "host bias" above, but are completed differently in the MA portion of these flows. With "device bias", processor requests to the accessor attached memory 3005 are completed as if they had been issued as "uncached" requests. This "uncached" convention is adopted so that data that is the subject of a device bias flow is never cached in the processor's cache hierarchy. This is in order to allow the accelerator 3001 to access device bias data in its memory 3005 without asking the cache coherence controller 3009 on the processor.

In one embodiment, support for "uncached" processor core 3003 access flows is implemented with a globally monitored use-once ("GO-UO") response on the processor's CAC bus. This response returns a portion of the data to the processor core 3003 and instructs the processor to use only the value of the data once. This prevents caching of the data and satisfies the requirements of the "uncached" flow. In systems with cores that do not support GO-UO responses, the "uncached" flow may be implemented using a multi-message response sequence on the MA layer of the multiprotocol link 2800 and on the processor core 3003's CAC bus.

Specifically, when a processor core is obtained to target a "device bias" page in the accelerator 3001, the accelerator sets up some state to block future requests from the accelerator for the target cache line and sends a special "device bias hit" response on the MA layer of the multi-protocol link 2800. In response to this MA message, the processor's cache coherence controller 3009 returns the data to the requesting processor core 3003, followed immediately by a snoop invalidate message. When the processor core 3003 recognizes the snoop invalidate as complete, the cache coherence controller 3009 sends another special MA "device bias back complete" message back to the accelerator 3001 on the MA layer of the multi-protocol link 2800. This completion message causes the accelerator 3001 to clear the aforementioned blocked state.

Figure 107 illustrates an embodiment using bias. In one embodiment, the selection of device and host bias flows is driven by a bias tracker data structure that may be maintained as a bias table 10707 in the accelerator memory 3005. This bias table 10707 may be a page-granular structure (i.e., controlled at the granularity of a memory page) that includes one or two bits per accelerator-attached memory page. The bias table 10707 may be implemented in stolen memory ranges of the accessor-attached memory 3005, with or without a bias cache 10703 in the accelerator (e.g., caching frequently/recently used entries of the bias table 10707). Alternatively, the entire bias table 10707 may be maintained in the accelerator 3001.

In one embodiment, the bias table entry associated with each access to the accelerator associated memory 3005 is accessed prior to the actual access to the accelerator memory to perform the following operations:
Local requests from the accelerator 3001 finding these pages in the device memory are forwarded directly to the accelerator memory 3005.
Local requests from the accelerator 3001 to find these pages in the host database are forwarded to the processor 3003 as CAC requests on the multi-protocol link 2800.
MA requests from processor 3003 that find these pages in the device bias will complete the request using the "not cached" flow described above.
MA requests from the processor 3003 that find these pages in the host bias complete the request like a normal memory read.

The bias state of a page can be changed by either software-based mechanisms, hardware-assisted software-based mechanisms, or, in a limited set of cases, purely hardware-based mechanisms.

One mechanism for changing the bias state employs an API call (e.g., OpenCL) that in turn invokes the accelerator's device driver, which in turn sends a message (or enqueues a command descriptor) to the accelerator 3001 instructing it to change the bias state, and for some transitions, performs a cache flush operation in the host. A cache flush operation is required for transitions from host bias to device bias, but is not required for the reverse transition.

In some cases, it is difficult for software to determine when to make a bias transition API call and when to identify a page request bias transition. In such cases, the accelerator may implement a bias transition (implied) mechanism to detect the need for a bias transition and send a message to its driver indicating so. The implied mechanism can be as simple as a mechanism that corresponds to a bias table lookup that triggers an accelerator access to the host bias page or a host access to the device bias page and signals the event to the accelerator's driver via an interrupt.

Note that in some embodiments, a second bias state bit may be required to enable the bias transition state value. This allows the system to continue accessing memory pages while those pages are in the process of a bias change (i.e., the cache must be partially flushed and incremental cache pollution due to subsequent requests must be suppressed).

An exemplary process according to one embodiment is shown in FIG. 32. The process may be implemented on the systems and processor architectures described herein, but is not limited to any particular system or processor architecture.

At 3201, a particular set of pages are placed in the device bias. As previously mentioned, this may be accomplished by updating the entries for those pages in the bias table to indicate that the pages are in the device bias (e.g., by setting a bit associated with each page). In one embodiment, once placed in the device bias, the pages are guaranteed not to be cached in the host cache memory. At 3202, the pages are allocated from the device memory (e.g., software allocates the pages by initiating a driver/API call).

At 3203, the operands are pushed from the processor core to the allocated page. In one embodiment, this is accomplished by software using an API call to flip the host biased n-operand page (e.g., via an OpenCL API call). There are no data copies or cache flushes required, and the operand data may end up at this stage in some arbitrary location within the host cache hierarchy.

At 3204, the accelerator device uses the operands to generate results. For example, it may execute commands to process data directly from its local memory (e.g., 3005, described above). In one embodiment, the software flips the operand pages back to the device bias (e.g., updates the bias table) using the OpenCL API. As a result of the API call, a work descriptor is submitted to the device (e.g., via sharing on a dedicated work queue, as described below). The work descriptor may instruct the device to flush the operand pages from the host cache, resulting in a cache flush (e.g., performed using CLFLUSH in the CAC protocol). In one embodiment, the accelerator executes with no host-related coherence overhead and dumps the data into the result pages.

At 3205, the results are pulled from the allocated page. For example, in one embodiment, software makes one or more API calls (e.g., via the OpenCL API) to flip the result page to the host bias. This operation may change some bias state, but does not cause any coherence or cache flush operations. The host processor core can then access, cache, and share the result data as needed. Finally, at 3206, the allocated page is freed (e.g., via software).

A similar process in which operands are released from one or more I/O devices is shown in FIG. 33. At 3301, a particular set of pages are placed in the device bias. As previously mentioned, this may be accomplished by updating the entries for those pages in the bias table to indicate that the pages are in the device bias (e.g., by setting a bit associated with each page). In one embodiment, once placed in the device bias, the pages are guaranteed not to be cached in the host cache memory. At 3302, the pages are allocated from the device memory (e.g., software allocates the pages by initiating a driver/API call).

At 3303, the operand is pushed from the I/O agent to the allocated page. In one embodiment, this is accomplished by software using an unallocated store to the I/O agent and posting a DMA request to the I/O agent to write the data. In one embodiment, the data is never allocated in the host cache hierarchy and the target page remains in the device cache.

At 3304, the accelerator device uses the operands to generate a result. For example, software may submit work to the accelerator device and no page transition is required (i.e., the page remains within the device bias). In one embodiment, the accelerator device executes with no host-related coherence overhead and the accelerator dumps the data into a result page.

At 3305, the I/O agent (e.g., under direction from software) pulls the results from the allocated page. For example, software may post a DMA request to the I/O agent. If the source page remains on the device bias, no page transition is required. In one embodiment, the I/O bridge grabs a non-cacheable copy of the data from the result page with an RdCurr (read current) request.

In some embodiments, a Work Queue (WQ) holds "descriptors" submitted by software, an arbiter used to implement quality of service (QoS) and fairness policies, processing engines for processing the descriptors, an address translation and caching interface, and a memory read/write interface. The descriptors define the scope of work to be done. As shown in FIG. 34, in one embodiment, there are two different types of Work Queues: Dedicated Work Queue 3400 and Shared Work Queue 3401. Dedicated Work Queue 3400 stores descriptors for a single application 3413, while Shared Work Queue 3401 stores descriptors submitted by multiple applications 3410-3412. A Hardware Interface/Arbiter 3402 dispatches the descriptors from the Work Queues 3400-3401 to the accelerator processing engines 3405 according to a particular arbitration policy (e.g., based on the processing requirements of each application 3410-3413 and QoS/fairness policies).

Figures 108a-108b show memory mapped I/O (MMIO) space registers for use with a work queue based implementation. Version register 10807 reports the version of this architecture specification supported by the device.

General Capabilities Register (GENCAP) 10808 defines the general capabilities of the device, e.g., maximum transfer size, maximum batch size, etc. Table B lists the various parameters and values that may be specified in the GENCAP register.

In one embodiment, Work Queue Capability Register (WQCAP) 10810 defines the capabilities of the Work Queue, such as support for dedicated and/or shared modes of operation, number of engines, number of Work Queues, etc. Table C below lists various parameters and values that may be configured.

In one embodiment, Operation Capabilities Register (OPCAP) 20811 is a bitmask that defines the operation types supported by the device. Each bit corresponds to an operation type with the same code as the bit position. For example, bit 0 of this register corresponds to a No-op operation (code 0). The bit is set if the operation is supported and cleared if the operation is not supported.

In one embodiment, General Configuration Register (GENCFG) 10812 defines Virtual Channel (VC) Steering Tags, see Table E below.

In one embodiment, General Control Register (GENCTRL) 10813 indicates whether an interrupt was generated for a hardware or software error. See Table F below.

In one embodiment, the device enable register (ENABLE) stores an error code, an indicator as to whether the device is enabled, and a device reset value. See Table G below for further details.

In one embodiment, the interrupt cause register (INTCAUSE) stores a value indicating the cause of the interrupt, see Table H below.

In one embodiment, the command register (CMD) 10814 is used to submit the drain WQ, drain PASID, and drain all commands. The abort field indicates whether the requested operation is drain or abort. Before writing to this register, software may ensure that any command has completed before being submitted via this register. Before writing to this register, software may also configure the command configuration register and, if completion recording is requested, the command completion record address register.

The drain all command drains or aborts all outstanding descriptors in all WQs and all engines. The drain PASID command drains or aborts descriptors with a specific PASID in all WQs and all engines. The drain WQ drains or aborts all descriptors in a specific WQ. Depending on the implementation, any drain command may wait for other descriptors to complete in addition to the one it needs to wait for.

If the abort field is 1, software has requested that the affected descriptors be discarded. However, the hardware may still complete some or all of them. If a descriptor is discarded, no completion record is written and no completion interrupt is generated for that descriptor. Some or all other memory accesses may occur.

Completion of a command is indicated by generating a completion interrupt (if requested) and by clearing the status field of this register. When completion is signaled, all affected descriptors are either completed or discarded, and there are no further address translations, memory reads, memory writes or interrupts generated due to any affected descriptors. See Table I below.

In one embodiment, software error status register (SWERROR) 10815 stores several different types of errors, such as an error when submitting a descriptor, an error when converting the completion record address in the descriptor, an error when validating the descriptor if the completion record address valid flag in the descriptor is 0, and an error while processing a descriptor, such as a page fault if the completion record address valid flag in the descriptor is 0. See Table J below.

In one embodiment, the hardware error status register (HWERROR) 10816 is similar in format to the software error status register (see above).

In one embodiment, the Group Configuration Register (GRPCFG) 10817 stores configuration data for each Work Queue/Engine Group (see Figures 36-37). In particular, the Group Configuration Table is an array of registers in BAR0 that control the mapping of Work Queues to engines. There are as many groups as there are engines, but software may configure the number of groups required. Each active group contains one or more Work Queues and one or more engines. Any unused group must have both the WQ and Engine fields equal to 0. Descriptors submitted to any WQ in a group may be processed by any engine in the group. Each active Work Queue must be in a single group. An active Work Queue is one whose corresponding WQCFG register has a non-zero WQ Size field. Any engine not in a group is inactive.

Each GRPCFG register 10817 may be divided into three sub-registers, each of which is one or more 32-bit words (see Tables K-M). These registers may be read-only while the device is enabled. They are also read-only if the Work Queue Configuration Supported field in the WQCAP is 0.

The offsets of the sub-registers in BAR0, for each group G, 0≦G<number of engines, are as follows in one embodiment:

In one embodiment, the Work Queue Configuration Register (WQCFG) 10818 stores data that defines the operation of each Work Queue. The WQ Configuration Table is an array of 16-byte registers in BAR0. The number of WQ Configuration Registers matches the number of WQ fields in the WQCAP.

Each 16-byte WQCFG register is divided into four 32-bit sub-registers and may be read or written using aligned 64-bit read or write operations.

While the device is enabled or if the Work Queue Configuration Supported field in the WQCAP is 0, each WQCFG-A subregister is read-only.

Each WQCFG-B is writable at any time unless the Work Queue Configuration Support field in the WQCAP is 0. If the WQ Threshold field contains a value greater than the WQ size when WQ is enabled, WQ is not enabled and the WQ Error Code is set to 4. If the WQ Threshold field is written with a value greater than the WQ size while WQ is enabled, WQ is disabled and the WQ Error Code is set to 4.

Each WQCFG-C subregister is read-only while WQ is enabled. It may be written before or at the same time as setting WQ Enable to 1. If the Work Queue Configuration Support field of WQCAP is 0, the following fields are always read-only: WQ Mode, WQ Fault on Block Enable, and WQ Priority. Even if the Work Queue Configuration Support field of WQCAP is 0, the following fields of WQCFG-C are writeable if WQ is not enabled: WQ PASID and WQ U/S.

Each WQCFG-D subregister is writable at any time. However, if the device is not enabled, it is an error to set WQ Enable to 1.

If WQ Enable is set to 1, then both the WQ Enable and WQ Error Code fields are cleared. Then, either WQ Enable or WQ Error Code is set to a non-zero value that indicates whether the WQ was successfully enabled.

The sum of the WQ size fields of all WQCFG registers cannot be greater than the sum of the WQ size fields in GENCAP. This constraint is checked when the device is enabled. WQs with a WQ size field of 0 cannot be enabled and all other fields in such WQCFG registers are ignored. The WQ size field is read-only while the device is enabled. See Table N for data regarding each of the sub-registers.

In one embodiment, Work Queue Occupancy Interrupt Control Registers 10819 (one per Work Queue (WQ)) allow software to request an interrupt if the occupancy of a Work Queue drops to a certain threshold. If the WQ Occupancy Interrupt Enable for a WQ is 1 and the current WQ occupancy is at or below the WQ occupancy limit, the following actions may be performed: 1. The WQ Occupancy Interrupt Enable field is cleared. 2. Bit 3 of the Interrupt Reason Register is set to 1. 3. If bit 3 of the Interrupt Reason Register was 0 before step 2, an interrupt is generated using MSI-X table entry 0. 4. If the register is written with Enable=1 and Limit>=Current WQ Occupancy, an interrupt is generated immediately. As a result, an interrupt is always generated immediately if the register is written with Enable=1 and Limit>=WQ Size.

In one embodiment, the Work Queue Status Registers (one per WQ) 10820 define the current number of entries in each WQ. This number can change any time a descriptor is submitted to or dispatched from the queue, so it cannot be relied upon to determine whether a WQ has room.

In one embodiment, MSI-X entry 10821 stores the MSI-X table data. The offset and number of entries are in the MSI-X feature. The suggested number of entries is the number of WQs plus 2.

In one embodiment, the MSI-X raw bit array 10822 stores the offset and number of entries in the MSI-X function.

In one embodiment, the interrupt message storage entry 10823 stores interrupt messages in a table structure. The format of this table is similar to the format of the MSI-X table defined in PCIe, but the size is not limited to 2048 entries. However, in some embodiments, the size of this table may vary between different DSA implementations and may be less than 2048 entries. In one embodiment, the number of entries is in the interrupt message storage size field of the general capabilities register. If the interrupt message storage support capability is 0, this table is not present. In order for the DSA to support a large number of virtual machines or containers, the size of the supported table needs to be quite large.

In one embodiment, the format of each entry in the IMS is described in Table P below.

FIG. 35 illustrates one embodiment of a Data Streaming Accelerator (DSA) device having multiple work queues 3511-3512 that receive descriptors submitted via an I/O fabric interface 3501 (such as the multi-protocol link 2800 described above). The DSA uses the I/O fabric interface 3501 to receive downstream work requests from clients (such as processor cores, peer input/output (IO) agents (such as network interface controllers (NICs)) and/or software chain offload requests) and for upstream read, write, and address translation operations. The illustrated embodiment includes an arbiter 3513 that arbitrates between the work queues and dispatches work descriptors to one of multiple engines 3550. The operation of the arbiter 3513 and the work queues 3511-3512 may be configured through work queue configuration registers 3500. For example, the arbiter 3513 may be configured to implement various QoS and/or fairness policies for dispatching descriptors from each of the work queues 3511-1012 to each of the engines 3550.

In one embodiment, some of the descriptors queued in the work queues 3511-3512 are batch descriptors 3515 that contain/identify a batch of work descriptors. The arbiter 3513 forwards the batch descriptors to a batch processing unit 3516 which processes the batch descriptors by reading an array of descriptors 3518 from memory using addresses translated through a translation cache 3520 (and potentially other address translation services on the processor). Once the physical addresses have been identified, the data read/write circuitry 3540 reads the batch of descriptors from memory.

The second arbiter 3519 arbitrates between batches of work descriptors 3518 provided by the batch processing unit 3516 and individual work descriptors 3514 obtained from the work queues 3511-3512, and outputs the work descriptors to the work descriptor processing unit 3530. In one embodiment, the work descriptor processing unit 3530 has stages for reading memory (via data R/W unit 3540), performing requested operations on the data, generating output data, and writing (via data R/W unit 3540) the output data, completion records, and interrupt messages.

In one embodiment, the work queue configuration allows software to configure each WQ (via the WQ configuration registers 3500) as either a shared work queue (SWQ) that receives descriptors using non-posted ENQCMD/S instructions, or as a dedicated work queue (DWQ) that receives descriptors using posted MOVDIR64B instructions. As already discussed above with respect to FIG. 34, a DWQ may process work and batch descriptors submitted from a single application, while a SWQ may be shared among multiple applications. The WQ configuration registers 3500 also allow software to control which WQs 3511-3512 feed which accelerator engines 3550, and the priorities associated with the WQs 3511-3512 that feed each engine. For example, an ordered set of priorities may be defined (e.g., high, medium, low: 1, 2, 3, etc.), and descriptors may generally be dispatched from a higher priority work queue before or more frequently than a lower priority work queue. For example, with two work queues identified as high and low priority, for every 10 descriptors dispatched, 8 of the 10 descriptors may be dispatched from the high priority work queue, while 2 of the 10 descriptors are dispatched from the low priority work queue. Various other techniques may be used to achieve different priority levels between work queues 3511-3512.

In one embodiment, the Data Streaming Accelerator (DSA) is software compatible with PCI Express configuration mechanisms and implements the PCI header and extension space in its configuration mapping register set. The configuration registers can be programmed through CFC/CF8 or MMCFG from the root complex. Similarly, all internal registers may also be accessible through the JTAG or SM bus interface.

In one embodiment, the DSA device uses memory-mapped registers to control its operation. Capability, configuration and work submission registers (portals) are accessible through the MMIO regions defined by the BAR0, BAR2 and BAR4 registers (described below). Each portal may be on a separate 4K page so that they can be independently mapped to different address spaces (clients) using the processor page tables.

As mentioned above, software specifies work to the DSA through descriptors. The descriptors specify the type of operation to the DSA to perform, such as addresses of data and status buffers, immediate operands, completion attributes, etc. (Additional details and details regarding the format of the descriptors are described below). The completion attributes specify the address to write the completion record to and the information needed to generate an optional completion interrupt.

In one embodiment, the DSA avoids maintaining client-specific state on the device. All information to process the descriptor is in the descriptor itself. This improves its sharing capabilities between user mode applications as well as between different virtual machines (machine containers) in a virtualized system.

A descriptor may contain an operation and associated parameters (called a work descriptor), or the descriptor may contain the address of an array of work descriptors (called a batch descriptor). Software prepares the descriptors in memory and submits them to the device's work queue (WQ) 3511-3512. Descriptors are submitted to the device using the MOVDIR64B, ENQCMD, or ENQCMDS instructions, depending on the mode of the WQ and the privilege level of the client.

Each WQ 3511-3512 has a fixed number of slots, which can fill up under heavy load. In one embodiment, the device provides the necessary feedback to help the software implement flow control. The device dispatches descriptors from the work queues 3511-3512 and submits them to the engine for further processing. When the engine 3550 completes a descriptor and encounters a particular failure or error that results in an abort, it notifies the host software by either writing a completion record in host memory and/or issuing an interrupt.

In one embodiment, each work queue is accessible through multiple registers, each in a separate 4KB page in the device MMIO space. For each WQ, one work submission register is called the "non-privileged portal" and is mapped into user space for use by user mode clients. Another work submission register is called the "privileged portal" and is used by kernel mode drivers. The remaining is the guest portal and is used by kernel mode clients within the virtual machine.

As mentioned above, each Work Queue 3511-3512 can be configured to run in one of two modes: dedicated or shared. The DSA exposes capability bits in the Work Queue Capability Register to indicate support for dedicated and shared modes. The DSA also exposes control to the Work Queue Configuration Register 3500 to configure each WQ to operate in one of the modes. The mode of a WQ can only be changed while the WQ is disabled, i.e., (WQCFG Enable = 0). Additional details of the WQ Capability Register and WQ Configuration Register are described below.

In one embodiment, in shared mode, a DSA client submits descriptors to a work queue using the ENQCMD or ENQCMDS command. ENQCMD and ENQCMDS use 64-byte non-posted writes and wait for a response from the device before completing. The DSA returns (e.g., to the requesting client/application) "success" if there is space in the work queue, or "retry" if the work queue is full. The ENQCMD and ENQCMDS commands may return the status of the command submission with a zero flag (0 indicates success, 1 indicates retry). Using the ENQCMD and ENQCMDS commands, multiple clients may submit descriptors directly and simultaneously to the same work queue. The device provides this feedback so that clients can tell whether their descriptors were received or not.

In shared mode, the DSA may reserve some SWQ capacity for submissions through privileged portals for kernel mode clients. Work submissions through non-privileged portals are accepted until the number of descriptors in the SWQ reaches the threshold configured for the SWQ. Work submissions through privileged portals are accepted until the SWQ is full. Work submissions through guest portals are limited by the threshold in the same manner as non-privileged portals.

If the ENQCMD or ENQCMDS command returns "success", the descriptor is received by the device and queued for processing. If the command returns "retry", the software can either try to resubmit the descriptor to the SWQ, or, if it is a user mode client using a non-privileged portal, can request the kernel mode driver to submit the descriptor on behalf of the user mode client using the privileged portal. This helps to avoid denial of service and provides forward progress assurance. Alternatively, the software may use other methods (e.g., using the CPU to perform work) if the SWQ becomes full.

Clients/applications are identified by the device using a 20-bit ID called the Process Address Space ID (PASID). The PASID is used by the device to look up addresses in the device TLB 1722 and send address translation or page requests to the IOMMU 1710 (e.g., over the multi-protocol link 2800). In shared mode, the PASID to be used with each descriptor is included in the PASID field of the descriptor. In one embodiment, ENQCMD copies the PASID of the current thread from a specific register (e.g., the PASID MSR) into the descriptor, while ENQCMDS allows supervisor mode software to copy the PASID into the descriptor.

In "dedicated" mode, the DSA client may submit descriptors to the device's work queue using the MOVDIR64B instruction. MOVDIR64B uses 64-byte posted writes, which complete faster due to the posted nature of the write operation. With a dedicated work queue, the DSA may expose the total number of slots in the work queue and rely on software to provide flow control. Software is responsible for tracking the number of descriptors submitted and completed to detect work queue full conditions. If software erroneously submits a descriptor to a dedicated WQ when there is no room in the work queue, the descriptor may be dropped and an error may be logged (e.g., in a software error register).

The MOVDIR64B instruction does not write PASID as the ENQCMD or ENQCMDS instructions do, so the PASID field in the descriptor cannot be used in dedicated mode. The DSA may ignore the PASID field in descriptors submitted to a dedicated work queue, and instead uses the WQ PASID field of the WQ configuration register 3500 for address translation. In one embodiment, the WQ PASID field is set by the DSA driver when configuring a work queue in dedicated mode.

Dedicated mode does not share a single DWQ by multiple clients/applications, but the DSA device can be configured to have multiple DWQs, each of which can be independently assigned to a client. Furthermore, the DWQs can be configured to have the same or different QoS levels with different performance levels provided for different clients/applications.

In one embodiment, the Data Streaming Accelerator (DSA) includes two or more engines 3550 that process descriptors submitted to the work queues 3511-1012. One embodiment of the DSA architecture includes four engines, numbered 0 through 3. Engines 0 and 1 can each utilize up to the full bandwidth of the device (e.g., 30 GB/s for reads and 30 GB/s for writes). Of course, the combined bandwidth for all engines is also limited to the maximum bandwidth available to the device.

In one embodiment, software configures WQs 3511-3512 and engines 3550 into groups using group configuration registers. Each group contains one or more WQs and one or more engines. The DSA may use any engine in a group to process a descriptor posted to any WQ in the group, and each WQ and each engine may be in only one group. The number of groups may be the same as the number of engines, so each engine may be in a separate group, but not all groups need to be used if any group contains more than one engine.

The DSA architecture allows great flexibility when configuring work queues, groups, and engines, but the hardware may be narrowly designed for use with a particular configuration. Engines 0 and 1 may be configured in one of two different ways, depending on the requirements of the software. One recommended configuration is to place both engines 0 and 1 in the same group. The hardware will use either one of the engines to process descriptors from any work queue in the group. In this configuration, if one engine has a stall due to high latency memory address translation or a page fault, the other engine can continue to operate, maximizing overall device throughput.

Figure 36 shows two work queues 3621-3622 and 3623-3624 in each group 3611 and 3612, respectively, but there may be any number up to the maximum number of WQs supported. The WQs in a group may be shared WQs with different priorities, one shared WQ and another dedicated WQ, or multiple dedicated WQs with the same or different priorities. In the illustrated example, group 3611 is served by engines 0 and 1 3601, and group 3612 is served by engines 2 and 3 3602.

As shown in FIG. 37, another configuration using engine 0 3700 and engine 1 3701 places them in separate groups 3710 and 3711, respectively. Similarly, group 2 3712 is assigned to engine 2 3702 and group 3 is assigned to engine 3 3703. Furthermore, group 0 3710 is composed of two work queues 3721 and 3722, group 1 3711 is composed of work queue 3723, work queue 2 3712 is composed of work queue 3724, and group 3 3713 is composed of work queue 3725.

Software may select this configuration if it wants to reduce the chance that latency-sensitive operations become blocked behind other operations. In this configuration, the software submits latency-sensitive operations to work queue 3723 connected to engine 1 3702, and other operations to work queues 3721-3722 connected to engine 0 3700.

Engine 2 3702 and Engine 3 3703 may be used to write to high bandwidth non-volatile memory, such as, for example, phase change memory. The bandwidth capabilities of these engines may be sized to match the expected write bandwidth of this type of memory. For this usage, bits 2 and 3 of the Engine Configuration Register should be set to 1, indicating that Virtual Channel 1 (VC1) should be used for traffic from these engines.

In platforms that do not have high bandwidth non-volatile memory (e.g., phase change memory), or if DSA devices are not used to write to this type of memory, engines 2 and 3 may be unused. However, software can make use of them as additional low latency paths, provided the submitted operations can tolerate the limited bandwidth.

As each descriptor arrives at the head of the work queue, it may be removed by the scheduler/arbiter 3513 and forwarded to one of the engines in the group. For a batch descriptor 3515 that points to a work descriptor 3518 in memory, the engine fetches the array of work descriptors from memory (i.e., using the batch processing unit 3516).

In one embodiment, for each work descriptor 3514, the engine 3550 prefetches the translation for the completion record address and passes the operation to the work descriptor processing unit 3530. The work descriptor processing unit 3530 uses the device TLB 1722 and IOMMU 1710 for source and destination address translation, reads the source data, performs the specific operation, and writes the destination data back to memory. When the operation is completed, the engine writes the completion record to the pre-translated completion address and generates an interrupt, if requested by the work descriptor.

In one embodiment, multiple work queues in the DSA may be used to provide multiple levels of quality of service (QoS). The priority of each WQ may be specified in the WQ configuration registers 3500. The priority of a WQ is relative to other WQs in the same group (e.g., priority levels for a WQ that exists alone in a group are meaningless). Work queues in a group may have the same or different priorities. However, since a single SWQ will service the same application, there is no point in configuring multiple shared WQs with the same priority in the same group. The scheduler/arbiter 3513 dispatches work descriptors from the work queues 3511-3512 to the engine 3550 according to these priorities.

Figure 38 shows one embodiment of a descriptor 1300 that includes an operation field 3801 that specifies the operation to be performed, a number of flags 3802, a processing address space identifier (PASID) field 3803, a completion record address field 3804, a source address field 3805, a destination address field 3806, a completion interrupt field 3807, a transfer size field 3808, and (potentially) one or more operation specific fields 3809. In one embodiment, there are three flags: completion record address valid, requested completion record, and requested completion interrupt.

Common fields include both trusted and non-trusted fields. Trusted fields are always trusted by the DSA device because they are entered by the CPU on the host or by privileged (ring 0 or VMM) software. Non-trusted fields are supplied directly by the DSA client.

In one embodiment, the trusted field includes a PASID field 3803, a reserved field 3811, and a U/S (user/supervisor) field 3810 (i.e., 4 bytes starting at an offset of 0). When the descriptor is submitted using an ENQCMD command, these fields in the source descriptor may be ignored. The value contained in the MSR (e.g., the PASID MSR) may be placed in these fields before the descriptor is sent to the device.

In one embodiment, these fields in the source descriptor are initialized by software when the descriptor is submitted using the ENQCMDS instruction. If the PCI Express PASID feature is not enabled, the U/S field 3810 is set to 1 and the PASID field 3803 is set to 0.

If the descriptor is submitted using the MOVDIR64B instruction, these fields in the descriptor may be ignored. The device uses the WQ U/S and WQ PASID fields of the WQ Config Register 3500 instead.

These fields may be ignored for any descriptor in the batch. The corresponding fields in the batch descriptor 3515 are used for each descriptor 3518 in the batch. Table Q provides a description and bit position for each of these trusted fields.

Table R below lists the operations performed in one embodiment according to the operation field 3801 of the descriptor.

Table S below lists the flags used in one embodiment of the descriptor.

In one embodiment, the completion record address 3804 specifies the address of the completion record. The completion record may be 32 bytes and the completion record address is aligned on a 32-byte boundary. If the completion record address valid flag is 0, this field is reserved. If the requested completion record flag is 1, the completion record is written to this address upon completion of the operation. If the requested completion record is 0, the completion record is written to this address only if there is a page fault or error.

For any operation that produces a result such as a comparison, the completion record address valid and requested completion record flags should both be 1 and the completion record address should be valid.

For any operation that uses a virtual address, the completion record address should be valid whether or not the requested completion record flag is set, so that the completion record can be written in the event of a page fault or error.

For best results, this field should be valid in all descriptors, as it allows the device to report errors to the software that submitted the descriptor. If this flag is 0 and an unexpected error occurs, the error is reported in the SWERROR register and the software that submitted the request may not be notified of the error.

If the completion queue enable flag is set in the batch descriptor, the completion record address field 3804 ignores the descriptor in the batch and the completion queue address in the batch descriptor is used instead.

In one embodiment, for operations that read data from memory, the source address field 3805 specifies the address of the source data. There are no alignment requirements for the source address. For operations that write data to memory, the destination address field 3806 specifies the address of the destination buffer. There are no alignment requirements for the destination address. For some operation types, this field is used as the address of a second source buffer.

In one embodiment, the transfer size field 3808 indicates the number of bytes to be read from the source address to perform the operation. The maximum value of this field may be 232-1, but the maximum possible transfer size may be less and must be determined from the maximum transfer size field in the general capability register. The transfer size should not be zero. For many operation types, there are no alignment requirements for the transfer size. Exceptions are noted in the operation description.

In one embodiment, if the use interrupt message storage flag is 1, the completion interrupt handling field 3807 specifies the interrupt message storage entry used to generate the completion interrupt. The value of this field should be less than the value of the interrupt message storage size field in GENCAP. In one embodiment, the completion interrupt handling field 3807 is reserved under any of the following conditions: the use interrupt message storage flag is 0, the request completion interrupt flag is 0, the U/S bit is 0, the interrupt message storage support field of the general capabilities register is 0, or the descriptor is being submitted through a guest portal.

As shown in FIG. 39, one embodiment of a completion record 3900 is a 32-byte structure in memory that the DSA writes to when an operation completes or encounters an error. The completion record address should be 32-byte aligned.

This section describes the fields of the completion record that are common to many operation types. Each operation type description includes a completion record diagram if the format differs from this. Additional operation-specific fields are described further below. The completion record 3900 will always be 32 bytes, even if none of the fields are required. The completion record 3900 contains enough information to continue the operation if it is partially completed due to a page fault.

The completion record may be implemented as a 32-byte aligned structure in memory (identified by completion record address 3804 of descriptor 3800). The completion record 3900 includes a completion status field 3904 that indicates whether the operation completed. If the operation completed successfully, the completion record may include the results of the operation, if any, depending on the type of operation. If the operation did not complete successfully, the completion record includes failure or error information.

In one embodiment, status field 3904 reports the completion status of the descriptor. Software should initialize this field to zero so that it can detect when the completion record has been written.

Table T above provides various status codes and related explanations for one embodiment.

Table U below shows the fault codes 3903 available in one embodiment, which include a first bit indicating whether the faulting address was read or written, and a second bit indicating whether the faulting access was a user mode or supervisor mode access.

In one embodiment, if this completion record 3900 was for a descriptor submitted as part of a batch, index field 3902 contains the index within the batch of the descriptor that generated this completion record. For batch descriptors, this field may be 0xff. For other descriptors that are not part of a batch, this field may be reserved.

In one embodiment, if an operation is partially completed due to a page fault, the bytes completed field 3901 contains the number of source bytes that were processed before the fault occurred. All of the source bytes represented by this count have been fully processed, and the results are written to the destination address as appropriate according to the operation type. For some operation types, this field may be used if the operation stopped before completion for some reason other than a fault. If the operation completed completely, this field may be set to 0.

For operation types where the output size is not readily determinable from this value, the completion record also includes the number of bytes written to the destination address.

If the operation was partially completed due to a page fault, this field contains the address that caused the fault. As a general rule, all descriptors should have a valid completion record address 3804 and the completion record address valid flag should be 1. Some exceptions to this rule are described below.

In one embodiment, the first byte of the completion record is the status byte. All status values written by the device are non-zero. Software should initialize the status field of the completion record to 0 before submitting the descriptor to allow the device to indicate when it has written to the completion record. Initializing the completion record also ensures that it is mapped, so the device does not encounter a page fault when accessing it.

The requested completion record flag indicates to the device that it should write a completion record even if the operation completed successfully. If this flag is not set, the device will only write a completion record if there is an error.

Descriptor completion can be detected by software using any of the following methods:

1. Poll the completion record and wait for the status field to become non-zero.

2. Use the UMONITOR/UMWAIT instruction (as described herein) to the completed record address to block until written or until a timeout occurs. Software should then check if the status field is non-zero to determine if the operation is complete.

3. For kernel mode descriptors, request an interrupt when the operation is completed.

4. If the descriptor is in a batch, set the fence flag in subsequent descriptors in the same batch. Completion of the descriptor with the fence or any subsequent descriptor in the same batch indicates completion of all descriptors preceding the fence.

5. If the descriptor is in a batch, completion of the batch descriptor that initialized the batch indicates completion of all the descriptors in the batch.

6. Issue a drain descriptor or drain command and wait for it to complete.

If the completion status indicates partial completion due to a page fault, the completion record indicates how much (if any) of the processing was completed before the fault was triggered, and the virtual address at which the fault was triggered. Software may choose to reverse the fault (by touching the faulting address from the processor) and resubmit the remainder of the work in a new descriptor, or complete the remainder of the work in software. Faults on the descriptor list and completion record addresses are handled differently, and are explained in more detail below.

One embodiment of the DSA supports only message signaling interrupts. The DSA provides two types of interrupt message storage: (a) MSI-X tables, enumerated through MSI-X functions, that store interrupt messages used by host drivers, and (b) device-specific interrupt message storage (IMS) tables that store interrupt messages used by guest drivers.

In one embodiment, interrupts may be generated for three types of events: (1) completion of a kernel mode descriptor, (2) completion of a drain or abort command, and (3) an error posted in the software or hardware error register. There is a separate interrupt enable for each type of event. Interrupts due to errors and completion of abort/drain commands are generated using entry 0 in the MSI-X table. The interrupt reason register may be read by software to determine the reason for the interrupt.

For kernel mode descriptor completions (e.g., descriptors with a U/S field of 1), the interrupt message used depends on how the descriptor was submitted and the use interrupt message storage flag in the descriptor.

The completed interrupt message for a kernel mode descriptor submitted through a privileged portal is typically an entry in the MSI-X table, determined by the portal address. However, if the interrupt message storage support field in GENCAP is 1, a descriptor submitted through a privileged portal may override this behavior by setting the use interrupt message storage flag in the descriptor. In this case, the completed interrupt handling field in the descriptor is used as an index into the interrupt message storage.

The completion interrupt message for a kernel mode descriptor submitted through a guest portal is an entry in the interrupt message storage, determined by the portal address.

Interrupts generated by the DSA are handled through interrupt remapping and posting hardware, as configured by the kernel or VMM software.

As mentioned above, the DSA supports submitting multiple descriptors at once. A batch descriptor contains the address of an array of work descriptors in host memory and the number of elements in that array. The array of work descriptors is called a "batch." The use of batch descriptors allows a DSA client to submit multiple work descriptors with a single ENQCMD, ENQCMDS, or MOVDIR64B instruction, potentially improving overall throughput. The DSA enforces a limit on the number of work descriptors in a batch. The limit is indicated in the maximum batch size field in the general capabilities register.

Batch descriptors are submitted to a work queue in the same way as other work descriptors. When a batch descriptor is processed by a device, the device reads the array of work descriptors from memory and then processes each of the work descriptors. The work descriptors are not necessarily processed in order.

The PASID 3803 and U/S flag in the batch descriptor are used for all the descriptors in the batch. The PASID and U/S fields 3810 in the descriptors in the batch are ignored. Each work descriptor in the batch can specify a completion record address 3804, just like a directly submitted work descriptor. Alternatively, the batch descriptor can specify a "completion queue" address where the completion records for all the work descriptors from the batch are written by the device. In this case, the completion record address field 3804 in the descriptors in the batch is ignored. The completion queue should be one entry larger than the total number of descriptors, so there is room for the batch descriptor and completion records for all the descriptors in the batch. Completion records are generated in the order that the descriptors are completed, which may not be the same order that they appear in the descriptor array. Each completion record contains the index of the descriptor in the batch that generated it. An index of 0xff is used for the batch descriptor itself. An index of 0 is used for directly submitted descriptors other than the batch descriptor. Some descriptors in a batch may not generate completion records if they did not request them and they complete successfully. In this case the number of completion records written to the completion queue may be less than the number of descriptors in the batch. The completion record for the batch descriptor (if requested) is written to the completion queue after the completion records for all descriptors in the batch.

A batch descriptor does not specify a completion queue, and the completion record for a batch descriptor (if requested) is written to its own completion record address after all descriptors in the batch have completed. The completion record for a batch descriptor includes an indication as to whether any of the descriptors in the batch completed with a status other than successful. This allows software to only examine the completion record for a batch descriptor in the normal case where all descriptors in the batch have completed successfully.

Completion interrupts may be requested by one or more work descriptors in a batch, as required. The completion record for the batch descriptor (if requested) is written after the completion records and completion interrupts for all descriptors in the batch. The completion interrupt for the batch descriptor (if requested) is generated after the completion record for the batch descriptor, just like any other descriptor.

Batch descriptors do not have to be contained in a batch. Nested or chained descriptor arrays are not supported.

By default, the DSA does not guarantee any ordering while executing work descriptors. Descriptors can be dispatched and completed in any order the device sees fit to maximize throughput. Thus, if ordering is required, software must explicitly order it. For example, software can submit a descriptor, wait for a completion record or interrupt from the descriptor to ensure completion, and then submit the next descriptor.

Software can also specify ordering for descriptors within a batch defined by a batch descriptor. Each work descriptor has a fence flag. If set, the fence ensures that processing of a descriptor does not begin until the previous descriptor in the same batch has completed. This allows a descriptor with a fence to consume data produced by a previous descriptor in the same batch.

A descriptor is completed after all writes generated by the operation are globally observable, after the destination has read back if requested, after the writes to the completion record are globally observable if required, and after a completion interrupt has been generated if requested.

If any descriptor in the batch completes with a status other than successful, for example if it partially completed due to a page fault, the subsequent descriptor with the fence flag equal to 1, and any subsequent descriptors in the batch, are discarded. The completion record for the batch descriptor used to submit the batch indicates how many descriptors have completed. Any descriptors that partially completed and generated a completion record are counted as completed. Only discarded descriptors are considered to be incomplete.

Fences also ensure ordering for completion records and interrupts. For example, a no-op descriptor with a set fence and a requesting completion interrupt will cause the interrupt generated after all preceding descriptors in the batch have completed (and their completion records written, if required). Completion record writes are always ordered behind data writes generated by the same work descriptor, and completion interrupts (if requested) are always ordered behind completion record writes for the same work descriptor.

A drain is a descriptor that allows a client to wait for all descriptors belonging to its own PASID to complete. It can be used as a fence operation for an entire PASID. A drain operation completes when all previous descriptors with that PASID have completed. A drain descriptor can be used by software to request a single completion record or interrupt for the completion of all its descriptors. A drain is a normal descriptor submitted to the normal work queue. A drain descriptor may be included in a batch. (The fence flag may be used in a batch to wait for the previous descriptor in the batch to complete.)

The software must ensure that after a drain descriptor is submitted and before it is completed, no other descriptors with the particular PASID are submitted to the device. If additional descriptors are submitted, it is unspecified whether the drain operation also waits for the additional descriptors to complete. This could cause the drain operation to take a long time. Even if the device does not wait for the additional descriptors to complete, some of the additional descriptors may complete before the drain operation completes. Thus, draining differs from fencing because fencing ensures that no subsequent operations start until all previous operations have completed.

In one embodiment, abort/drain commands are submitted by privileged software (OS kernel or VMM) by writing to the abort/drain register. Upon receiving one of these commands, the DSA waits for the completion of a particular descriptor (described below). When the command completes, the software can be sure that there are no more descriptors in the particular category outstanding on the device.

In one embodiment, there are three types of drain commands: DRAIN ALL, DRAIN PASID, and DRAIN WQ. Each command has an abort flag that indicates the device may discard any outstanding descriptors rather than processing them to completion.

The drain-all command waits for all descriptors submitted before the drain-all command to complete. Descriptors submitted after the drain-all command may be in progress when the drain-all command completes. The device may start work on new descriptors while the drain-all command is waiting for previous descriptors to complete.

The drain PASID command waits for all descriptors associated with the particular PASID. When the drain PASID command completes, there are no more descriptors for the PASID in the device. Software may ensure that after the drain PASID command is submitted and before it completes, there are no more descriptors with the particular PASID submitted to the device. Otherwise, the behavior is undefined.

The drain WQ command waits for all descriptors submitted to a particular work queue. Software can ensure that no descriptors are submitted to the WQ after the drain WQ command is submitted and before it completes.

When an application or VM using the DSA is suspended, it may have outstanding descriptors submitted to the DSA. This work must be completed so the clients are in a coherent state that can be resumed later. The drain PASID and drain all commands are used by the OS or VMM to wait for any outstanding descriptors. The drain PASID command is used for applications or VMs that were using a single PASID. The drain all command is used for VMs that use multiple PASIDs.

When an application using the DSA exits or is terminated by the Operating System (OS), the OS must ensure that there are no outstanding descriptors before it can free or reclaim the address space, allocated memory, and PASID. To dispose of any outstanding descriptors, the OS uses the DRAIN PASID command with the PASID of the client being terminated, and the abort flag is set to 1. Upon receiving this command, the DSA discards all descriptors belonging to the particular PASID without further processing.

One embodiment of the DSA provides a mechanism to specify quality of service for dispatching work from multiple WQs. DSA allows software to divide the total WQ space into multiple WQs. Each WQ can be assigned a different priority for dispatching work. In one embodiment, the DSA scheduler/arbiter 3513 dispatches work from WQs such that higher priority WQs are serviced more than lower priority WQs. However, the DSA ensures that higher priority WQs do not starve lower priority WQs. As already mentioned, various prioritization schemes may be adopted based on implementation requirements.

In one embodiment, a WQ configuration register table is used to configure the WQs. Software can configure the number of active WQs to match the number of desired QoS levels. Software configures each WQ by programming the WQ size and some additional parameters into the WQ configuration register table. This effectively divides the entire WQ space into the desired number of WQs. Unused WQs have a size of 0.

Errors are broadly divided into two categories: 1) related errors that occur when processing a particular PASID's descriptors, and 2) unrelated errors that are broad in nature and not specific to a PASID. Whenever possible, the DSA attempts to prevent errors from one PASID from stopping or affecting other PASIDs. PASID-specific errors are reported in the completion record for each descriptor, except when the error is in the completion record itself (e.g., a page fault on the completion record address).

Errors in the descriptor submission or on recording the completion of the descriptor may be reported to the host driver through the software error register (SWERROR). Hardware errors may be reported through the hardware error register (HWERROR).

In one embodiment of the DSA, when the enable bit in the device enable register is set to one, the following checks are performed:
Bus master enable is 1.
- A combination of PASID, ATS and PRS features is valid (see Table 6-3, Section 6.1.3).
The sum of the WQ size fields of all WQCFG registers is not greater than the total WQ size.
For each GRPCFG register, the WQ and engine fields are either both zero or both non-zero.
Each WQ with a non-zero size field in the WQCFG register is in one group.
Each WQ whose size field in the WQCFG register is zero is not in any group.
- Each engine is in only one group.

If any of these checks fail, the device is not enabled and an error code is recorded in the error code field of the device enable register. These checks may be performed in any order; therefore, an indication of one type of error does not imply that other errors exist. The same configuration error may result in different error codes at different times, or in different versions of the device. If none of the checks fail, the device is enabled and the enable field is set to 1.

The device performs the following checks when the WQ Enable bit in the WQCFG register is set to 1:
The device is enabled (i.e., the enable field in the device enable register is 1).
- The WQ size field is non-zero.
- The WQ threshold is not greater than the WQ size field.
The WQMode field selects a supported mode, i.e. if the Shared Mode Supported field in the WQCAP is 0 then WQMode is 1, or if the Dedicated Mode Supported field in the WQCAP is 0 then WQMode is 0. If both the Shared Mode Supported and Dedicated Mode Supported fields are 1 then either one of the values of WQMode is allowed.
If the block support bit of the failure in GENCAP is 0, then the block enable field of the WQ failure is 0.

If any of these checks fail, the WQ is not enabled and an error code is recorded in the WQ Error Code field of the WQ Config Register 3500. These checks may be performed in any order; therefore, an indication of one type of error does not imply that other errors exist. The same configuration error may result in different error codes at different times or in different versions of the device. If none of the checks fail, the device is enabled and the WQ Enable field is set to 1.

In one embodiment, the DSA performs the following checks when a descriptor is received:
The WQ identified by the register address used to submit the descriptor is an active WQ (the size field in the WQCFG register is non-zero). If this check fails, an error is logged in the Software Error Register (SWERROR).
If the descriptor was submitted to a shared WQ,
- It was submitted with ENQCMD or ENQCMDS. If this check fails, an error is logged in SWERROR.
If the descriptor was submitted via a non-privileged or guest portal, the current queue occupancy is not greater than the WQ threshold. If this check fails, a retry response is returned.
If the descriptor was submitted through a privileged portal, the current queue occupancy is less than the WQ size. If this check fails, a retry response is returned.
If the descriptor was submitted to a dedicated WQ,
- It was submitted using MOVDIR64B.
- The queue occupancy is smaller than the WQ size.

If any of these checks fail, an error is logged in SWERROR.

In one embodiment, the device performs the following checks on each descriptor as it is processed:
The value in the operation code field corresponds to a supported operation. This includes checking that the operation is valid in the context in which it was submitted. For example, a batch descriptor in a batch would be treated as an invalid operation code.
No reserved flags are set. This includes flags whose corresponding feature bits in the GENCAP register are 0.
No unsupported flags are set. This includes flags that are reserved for use with specific operations, e.g., fence bits are reserved in descriptors that are enqueued directly rather than as part of a batch. It also includes flags that are disabled in the configuration, e.g., the fault block flag, which is reserved if the fault block enable field in the WQCFG register is 0.
The requested flags are set. For example, the request completion record flag must be 1 in the descriptor for the compare operation.
Reserved fields are 0. This includes any field that is not defined to mean a specific action. In some embodiments, it is not necessary to check all reserved fields, but software should take care to clear all unused fields for maximum compatibility. In a batch descriptor, the total number of descriptors field is no larger than the maximum batch size field in the GENCAP register.
- The transfer size, source size, maximum delta record size, delta record size and maximum destination size (as applicable to the descriptor type) are not greater than the maximum transfer size field in the GENCAP register.
In a memory copy descriptor using dual cast, bits 11:0 of the two destination addresses are the same.
If the Used Interrupt Message Storage flag is set, the completed interrupt process is less than the interrupt message storage size.

In one embodiment, if the completion record address 3804 cannot be translated, the descriptor 3800 is discarded and an error is recorded in a software error register. Otherwise, if any of these checks fail, a completion record is written with a status field indicating the type of check that failed and the byte completion is set to 0. A completion interrupt is generated if requested.

These checks may be performed in any order. Thus, an indication of one type of error in the completion record does not imply that other errors exist. The same invalid descriptor may report different error codes at different times or on different versions of the device.

The reserved fields 3811 in a descriptor may be divided into three categories: fields that are always reserved, fields that are reserved under some conditions (e.g., based on the functionality, configuration fields, how the descriptor was submitted, or the values of other fields in the descriptor itself), and fields that are reserved based on the operation type. The following table lists the conditions under which fields are reserved.

As mentioned above, the DSA supports the use of either physical or virtual addresses. The use of virtual addresses that are shared with processes running on a processor core is called Shared Virtual Memory (SVM). To support SVM, the device provides a PASID when performing address translation, which handles page faults that occur when no translation exists for an address. However, the device itself does not distinguish between virtual and physical addresses. This distinction is controlled by programming of the IOMMU 1710.

In one embodiment, the DSA supports address translation services (ATS) and page request services (PRS) PCI Express features as shown in FIG. 28, which shows PCIe logic 2820 communicating with PCIe logic 2808 using PCDI to utilize the ATS. The ATS describes the behavior of the device during address translation. When a descriptor enters the descriptor processing unit, the device 2801 may request a translation for the address in the descriptor. If there is a hit in the device TLB 2822, the device uses the corresponding host physical address (HPA). If there is a failure or permission failure, one embodiment of the DSA 2801 sends an address translation request to the IOMMU 2810 (i.e., across the multi-protocol link 2800) for translation. The IOMMU 2810 may then look for the translation by walking its respective page table and return an address translation response that includes the translated address and valid permissions. The device 2801 then stores the translation in the device TLB 2822 and uses the corresponding HPA for the operation. If the IOMMU 2810 is unable to find a translation in the page tables, it may return an address translation response indicating that no translation is available. If the IOMMU 2810 response indicates that no translation is available, or indicates valid permissions that do not include the permissions requested by the operation, it is considered a page fault.

The DSA device 2801 may encounter a page fault at one of: 1) the completion record address 3804, 2) the descriptor list address in the batch descriptor, or 3) the source or destination buffer address. The DSA device 2801 may block until the page fault is resolved, or may complete the descriptor early and return a partial completion to the client. In one embodiment, the DSA device 2801 always blocks on page faults on the completion record address 3804 and the descriptor list addresses.

If the DSA blocks a page fault, it reports the page fault as a Page Request Service (PRS) request to the IOMMU 2810 for servicing by the OS page fault handler. The IOMMU 2810 may notify the OS via an interrupt. The OS validates the address and, if the check is successful, creates a mapping in the page table and returns the PRS response through the IOMMU 2810.

In one embodiment, each descriptor 3800 has a fault blocking flag that indicates whether DSA 2801 should return partial completion or block if a page fault occurs on the source or destination buffer address. If the fault blocking flag is 1 and a fault is encountered, the faulted descriptor is blocked until a PRS response is received. Other operations behind the faulting descriptor may also be blocked.

If the fault blocks 0 and a page fault occurs on the source or destination buffer address, the device stops the operation and writes a partial completion status to the completion record along with the fault address and progress information. When the client software receives a completion record indicating partial completion, it has the option to revert the fault on the processor (e.g., by touching the page) and submit a new work descriptor with the remaining work.

Alternatively, software can complete the remaining work on the processor. The Block on Fault Support field in the General Capabilities Register (GENCAP) may indicate device support for this feature, and the Block on Fault Enable field in the Work Queue Configuration Register allows the VMM or kernel driver to control whether applications are allowed to use the feature.

Device page faults can be relatively expensive. In fact, the cost of performing work on a device page fault may be higher than the cost of performing work on a processor page fault. Even if the device performs partial work completion on the fault instead of blocking the fault, additional overhead is incurred because software intervention is required to perform work on the page fault and resubmit the work. Thus, for best performance, it is preferable for software to minimize device page faults without incurring the overhead of pinning and unpinning.

The batch descriptor list and source data buffers are typically created by software just prior to submitting them to the device. Thus, their addresses are unlikely to cause failures due to temporal locality. However, the completion descriptors and destination data buffers are likely to cause failures if they are not touched by software before submitting them to the device. Such failures can be minimized by software explicitly "write touching" these pages before submission.

During a device TLB invalidation request, if the address being invalidated is in use in the descriptor processing unit, the device waits for the engine to be done with the address before completing the invalidation request.

Additional Descriptor Types

In some embodiments, one or more of the following additional descriptor types may be used:

No-op

Figure 40 shows an example no-op descriptor 4000 and a no-op completion record 4001. A no-op operation 4005 does not perform a DMA operation. It may require a completion record and/or a completion interrupt. If it is in a batch, a fence flag may be specified that ensures that the completion of the no-op descriptor occurs after the completion of all previous descriptors in the batch.

Batch

Figure 41 shows an example batch descriptor 4100 and no-op completion record 4101. Batch operation 4108 queues multiple descriptors at a time. Descriptor list address 4102 is the address for a contiguous array of work descriptors to be processed. In one embodiment, each descriptor in the array is 64 bytes. Descriptor list address 4102 is 64-byte aligned. Total number of descriptors 4103 is the number of descriptors in the array. The set of descriptors in the array is called a "batch." The maximum number of descriptors allowed in a batch is given in the maximum batch size field in GENCAP.

The PASID 4104 and U/S flag 4105 in the batch descriptor are used for all descriptors in the batch. The PASID 4104 and U/S flag fields 4105 in the descriptors in the batch are ignored. If the Completion Queue Enable flag in the batch descriptor 4100 is set, the Completion Record Address Valid flag must be 1, and the Completion Queue Address field 4106 contains the address of the completion queue to be used for all descriptors in the batch. In this case, the Completion Record Address field 4106 in the descriptors in the batch is ignored. If the Completion Queue Support field in the General Capabilities Register is 0, the Completion Queue Enable flag is reserved.

If the completion queue enable flag in the batch descriptor is 0, the completion records for each descriptor in the batch are written to the completion record address 4106 in each descriptor. In this case, if the requested completion record flag in the batch descriptor is 1, the completion queue address field is used exclusively as the completion record address 4106 for the batch descriptor.

The status field 4110 of the batch completion record 4101 indicates success if all of the descriptors in the batch completed, otherwise it indicates that one or more descriptors completed with a status other than success. The descriptors completed field 4111 of the completion record contains the total number of descriptors in the batch that were being processed, whether they were successful or not. The descriptors completed 4111 may be less than the total number of descriptors 4103 if there is a fence in the batch or if a page fault occurred while reading the batch.

Drain

Figure 42 shows an example drain descriptor 4200 and drain completion record 4201. The drain operation 4208 waits for the completion of all outstanding descriptors associated with PASID 4202 in the work queue to which the drain descriptor 4200 was submitted. This descriptor may be used during normal shutdown by processes using the device. To wait for all descriptors associated with PASID 4202, the software should submit a separate drain operation to each work queue in which PASID 4202 was used. The software should ensure that after the drain descriptor 4201 is submitted and before it completes, there are no descriptors with the particular PASID 4202 submitted to the work queue.

The drain descriptor 4201 may not be included in the batch and is treated as an unsupported operation type. The drain should specify a request completion record or a request completion interrupt. Completion notification occurs after other descriptors have completed.

Memory transfer

Figure 43 shows an example memory move descriptor 4300 and memory move completion record 4301. A memory move operation 4308 copies memory from a source address 4302 to a destination address 4303. The number of bytes copied is given by the transfer size 4304. There are no alignment requirements for the memory addresses or the transfer size. If the source and destination regions overlap, the memory copy is performed as if the entire source buffer was copied to a temporary space and then copied to the destination buffer. This can be accomplished by reversing the direction of the copy if the start of the destination buffer overlaps with the end of the source buffer.

If the operation is partially completed due to a page fault, the direction field 4310 of the completion record is 0 if a copy was performed starting from the beginning of the source and destination buffers, and the direction field is 1 if the direction of the copy was reversed.

To resume an operation after partial completion, if direction is 0, the source and destination address fields 4302-4303 in successive descriptors should be increased by byte completions and the transfer size should be decreased by byte completions 4311. If direction is 1, the transfer size 4304 should be decreased by byte completions 4311, but the source and destination address fields 4302-4303 should be the same as in the original descriptor. Note that when subsequent partial completions occur, the direction field 4310 does not have to be the same as for the first partial completion.

Full (Fill)

Figure 44 shows an example fill descriptor 4400. A memory fill operation 4408 fills memory at destination address 4406 with the value in pattern field 4405. The pattern size may be 8 bytes. To use a smaller pattern, software must replicate the pattern in the descriptor. The number of bytes written is given by transfer size 4407. The transfer size does not need to be a multiple of the pattern size. There are no alignment requirements for the destination address or transfer size. If the operation is partially completed due to a page fault, the bytes completed field of the completion record contains the number of bytes written to the destination before the fault occurred.

Compare

Figure 45 shows an exemplary compare descriptor 4500 and compare completion record 4501. A compare operation 4508 compares memory at address 4504 of source 1 with memory at address 4505 of source 2. The number of bytes compared is given by the transfer size 4506. There are no alignment requirements for the memory addresses or the transfer size 4506. The completion record address valid and requested completion record flags must be 1 and the completion record address must be valid. The result of the comparison is written to the result field 4510 of the completion record 4501. A value of 0 indicates that the two memory regions match and a value of 1 indicates that they do not match. If the result 4510 is 1, the bytes completed 4511 field of the completion record indicates the byte offset of the first difference. If the operation is partially completed due to a page fault, the result is 0. If a difference is detected, the difference will be reported in place of the page fault.

If the operation is successful and the check result flag is 1, the status field 4512 of the completion record is set according to the result and expected result, as shown in the table below. This allows subsequent descriptors in the same batch as the fence flag to continue or stop execution of the batch based on the outcome of the comparison.

Comparison intermediate

Figure 46 shows an exemplary compare intermediate descriptor 4600. The compare intermediate process 4608 compares the memory at source address 4601 with the value in pattern field 4602. The pattern size is 8 bytes. To use a smaller pattern, the software must duplicate the pattern in the descriptor. The number of bytes compared is given by the transfer size 4603. The transfer size does not have to be a multiple of the pattern size. The completion record address valid and requested completion record flags must be 1. The completion record address 4604 must be valid. The result of the comparison is written to the result field of the completion record. A value of 0 indicates that the memory region matches the pattern, and a value of 1 indicates that it does not match. If the result is 1, the byte done field of the completion record indicates the location of the first difference. It may not be the exact byte location, but it is guaranteed to be no greater than the first difference. If the operation is partially completed due to a page fault, the result is 0. If a difference was detected, the difference will be reported instead of the page fault. In one embodiment, the completion record format for the comparison intermediate and the behavior of the check results and predicted results are the same as the comparison.

Created difference record

Figure 47 shows an exemplary create delta record descriptor 4700 and create delta record completion record 4701. A create delta record operation 4708 compares memory at source 1 address 4705 with memory at source 2 address 4702 to generate a delta record containing the information needed to update source 1 to match source 2. The number of bytes compared is given by the transfer size 4703. As explained below, the transfer size is limited by the maximum offset that can be stored in the delta record. There are no alignment requirements for the memory address or transfer size. The completion record address valid and requested completion record flags must be 1, and the completion record address 4704 must be valid.

The maximum size of a differential record is given by the maximum differential record size 4709. The maximum differential record size 4709 should be a multiple of the differential size (10 bytes) and must not be larger than the maximum transfer size in GENCAP. The actual size of the differential record depends on the number of differences found between source 1 and source 2, and is written to the differential record size field 4710 of the completed record. If the space required for the differential record exceeds the maximum differential record size 4709 specified in the descriptor, the operation completes with a partial differential record.

The result of the comparison is written to the result field 4711 of the completion record 4701. If the two regions match exactly, result is 0, diff record size is 0, and bytes done is 0. If the two regions do not match, the complete set of differences is written to the diff record, result is 1, diff record size contains the total size of all differences obtained, and bytes done is 0. If the two regions do not match and the space required to record all differences exceeds the maximum diff record size, result is 2, diff record size 4710 contains the size of the set of differences written to the diff record (typically equal or nearly equal to the diff record size specified in the descriptor), and bytes done 4712 contains the number of bytes compared before the space in the diff record was exceeded.

If the operation is partially completed due to a page fault, then result 4711 will be either 0 or 1, bytes completed 4712 will contain the number of bytes compared before the page fault occurred, and differential record size will contain the space used in the differential record before the page fault occurred, as described in the previous paragraph.

The format of a delta record is shown in Figure 48. A delta record contains an array of deltas. Each delta contains a 2-byte offset 4801 and an 8-byte block of data 4802 from source 2 that differs from the corresponding 8 bytes in source 1. The total size of the delta record is a multiple of 10. Because offset 4801 is a 16-bit field that represents a multiple of 8 bytes, the maximum offset that can be represented is 0x7FFF8, and therefore the maximum transfer size is 0x80000 bytes (512KB).

If the operation is successful and the check result flag is 1, the status field of the completion record is set according to the result and expected result as shown in the table below. This allows subsequent descriptors in the same batch as the fence flag to continue or stop execution of the batch based on the result of the delta record creation. Bits 7:2 of the expected result are ignored.

Matching difference record

Figure 49 illustrates an exemplary conforming difference record descriptor 4901. Conforming difference record operation 4902 applies a difference record to the contents of memory at destination address 4903. Difference record address 4904 is the address of the difference record created by create difference record operation 4902 completing with result equal to 1. Difference record size 4905 is the size of the difference record as reported in the completion record of create difference record operation 4902. Destination address 4903 is the address of a buffer that contains the same contents as the memory at source 1's address when the difference record was created. Transfer size 4906 is the same as the transfer size used when the difference record was created. After conforming difference record operation 4902 completes, the memory at destination address 4903 matches the contents that were in the memory at source 2's address when the difference record was created. There are no alignment requirements for the memory address or transfer size.

If a page fault occurs during a conforming delta record operation 4902, the bytes done field of the completion record contains the number of bytes of the delta record that were successfully applied to the destination. If the software chooses to submit another descriptor to resume the operation, the successive descriptor contains the same destination address 4903 as the original. The delta record address 4904 should be increased by the bytes done (thus pointing to the first unapplied delta) and the delta record size 4905 should be reduced by the bytes done.

Figure 50 shows one embodiment of the use of create difference record and adaptive difference record operations. First, a create difference record operation 5001 is performed. The create difference record operation 5001 reads two source buffers - source 1 and 2 - and writes a difference record 5010 in its completion record 5003 that records the actual difference record size 5004. An adaptive difference record operation 5005 takes the contents of the difference record written by the create difference record operation 5001, along with its size and a copy of the source 1 data, and updates the destination buffer 5015 to be a replica of the original source 2 buffer. The create difference record operation includes a maximum difference record size 5002.

Memory copy using dual cast

Figure 51 shows an exemplary memory copy with dual cast descriptor 5100 and a memory copy with dual cast completion record 5102. A memory copy with dual cast operation 5104 copies memory from a source address 5105 to both a destination 1 address 5106 and a destination 2 address 5107. The number of bytes copied is given by the transfer size 5108. There are no alignment requirements for the source address or transfer size. Bits 11:0 of the two destination addresses 5106-5107 should be the same.

If the source area overlaps with either of the destination areas, the memory copy is performed as if the entire source buffer was copied to a temporary space and then copied to the destination buffer. This can be done by reversing the direction of the copy if the beginning of the destination buffer overlaps the end of the source buffer. It is an error if the source area overlaps both destination areas, or if the two destination areas overlap. If the operation is partially completed due to a page fault, the copy operation stops after writing the same number of bytes to both destination areas, and the direction field 5110 of the completion record is 0 if the copy is performed starting from the beginning of the source and destination buffers, and the direction field is 1 if the direction of the copy is reversed.

To resume an operation after partial completion, if direction 5110 is 0, the source 5105 and both destination address fields 5106-5107 in the successive descriptor should be increased by byte completion 5111, and the transfer size 5108 should be decreased by byte completion 5111. If direction is 1, the transfer size 5108 should be decreased by byte completion 5111, but the source 5105 and destination 5106-5107 address fields should be the same as in the original descriptor. Note that when a subsequent partial completion occurs, the direction field 5110 does not have to be the same as for the first partial completion.

Cyclic redundancy check (CRC) generation

Figure 52 shows an example CRC generation descriptor 5200 and CRC generation 5201. The CRC generation process 5204 calculates the CRC in memory at the source address. The number of bytes used in the CRC calculation is given by the transfer size 5205. There is no alignment requirement for the memory address or transfer size 5205. The completion record address valid and requested completion record flags must be 1 and the completion record address 5206 must be valid. The calculated CRC value is written to the completion record.

If an operation is partially completed due to a page fault, the partial CRC result is written to the completion record along with the page fault information. When software corrects the fault and resumes the operation, it must copy this partial result into the CRC seed field of the successive descriptor. Otherwise, the CRC seed field should be 0.

Copying using CRC generation

Figure 53 shows an example copy with CRC generation descriptor 5300. The copy with CRC generation process 5305 copies memory from source address 5302 to destination address 5303 and calculates a CRC for the copied data. The number of bytes copied is given by transfer size 5304. There are no alignment requirements for memory addresses or transfer size. It is an error if the source and destination regions overlap. The completion record address valid and requested completion record flags must be 1 and the completion record address must be valid. The calculated CRC value is written to the completion record.

If an operation is partially completed due to a page fault, the partial CRC result is written to the completion record along with the page fault information. When software corrects the fault and resumes the operation, it must copy this partial result into the CRC seed field of the successive descriptor. Otherwise, the CRC seed field should be 0. In one embodiment, the completion record format for copying with CRC generation is the same as the format for CRC generation.

Data Integrity Field (DIF) Insertion

Figure 54 shows an exemplary DIF insertion descriptor 5400 and DIF insertion completion record 5401. A DIF insertion operation 5405 copies memory from a source address 5402 to a destination address 5403, calculates a data integrity field (DIF) on the source data, and inserts the DIF into the output data. The number of source bytes copied is given by the transfer size 5406. The DIF calculation is performed on each block of source data, for example, 512, 520, 4096, or 4104 bytes. The transfer size should be a multiple of the source block size. The number of bytes written to the destination is the transfer size plus 8 bytes per source block. There is no alignment requirement for the memory address. It is an error if the source and destination regions overlap. If the operation is partially completed due to a page fault, the updated values of the reference tag and application tag are written to the completion record along with the page fault information. When the software corrects the failure and resumes operation, it can copy these fields into the contiguous descriptor.

DIF strip

Figure 55 shows an exemplary DIF strip descriptor 5500 and DIF strip completion record 5501. A DIF strip operation 5505 copies memory from a source address 5502 to a destination address 5503, calculates a data integrity field (DIF) on the source data, and compares the calculated DIF to the DIF contained in the data. The number of source bytes read is given by the transfer size 5506. The DIF calculation is performed on each block of source data, which can be 512, 520, 4096, or 4104 bytes. The transfer size should be a multiple of the source block size plus 8 bytes per source block. The number of bytes written to the destination is the transfer size minus 8 bytes per source block. There is no alignment requirement on the memory address. It is an error if the source and destination regions overlap. If the operation is partially completed due to a page fault, the updated values of the reference tag and application tag are written to the completion record along with the page fault information. When the software corrects the fault and resumes operation, it may copy these fields into the successive descriptors.

DIF Update

Figure 56 shows an exemplary DIF update descriptor 5600 and DIF update completion record 5601. A memory move using DIF update operation 5605 copies memory from source address 5602 to destination address 5603, calculates the data integrity field (DIF) on the source data, and compares the calculated DIF with the DIF contained in the data. It simultaneously calculates the DIF on the source data using the destination DIF field in the descriptor, and inserts the calculated DIF into the output data. The number of source bytes read is given by the transfer size 5606. The DIF calculation is performed on each block of source data, which can be 512, 520, 4096, or 4104 bytes. The transfer size 5606 should be a multiple of the source block size plus 8 bytes per source block. The number of bytes written to the destination is equal to the transfer size 5606. There is no alignment requirement for memory addresses. It is an error if the source and destination regions overlap. If an operation is partially completed due to a page fault, the updated values of the source and destination reference tags and the application tag are written to the completion record along with the page fault information. When software corrects the fault and resumes the operation, it may copy these fields into the contiguous descriptor.

Table AA below shows the DIF flags used in one embodiment, Table BB shows the source DIF flags used in one embodiment, and Table CC shows the destination DIF flags in one embodiment.

Source DIF Flag

Destination DIF Flag

In one embodiment, the DIF result field reports the status of the DIF operation. This field may be defined only for DIF strip and update operations, and only if the status field of the completion record is success or success with incorrect predicate. Table DD below shows exemplary DIF result field codes.

The F detection condition is detected if one of the following in Table EE is true:

If the operation is successful and the check result flag is 1, the status field of the completion record is set according to the DIF result, as shown in Table FF below. This allows subsequent descriptors in the same batch as the fence flag to continue or stop execution of the batch based on the outcome of the operation.

Cache flush

Figure 57 shows an example cache flush descriptor 5700. A cache flush operation 5705 flushes the processor cache at a destination address. The number of bytes flushed is given by the transfer size 5702. The transfer size does not need to be a multiple of the cache line size. There are no alignment requirements for the destination address or transfer size. Any cache lines that are partially covered by the destination region are flushed.

If the destination cache filter is 0, the affected cache line may be invalidated from each level of the cache hierarchy. If the cache line contains data that was modified at any level of the cache hierarchy, the data is written back to memory. This is similar to the behavior of the CLFLUSH instruction implemented in some processors.

If the destination cache filter is 1, the modified cache line is written to main memory but is not evicted from the cache. This is similar to the behavior of the CLWB instruction in some processors.

Term accelerator is sometimes used herein to refer to a loosely coupled agent that can be used by software running on a host processor to offload or perform any kind of computational or I/O task. Depending on the type of accelerator and usage model, these can be tasks that perform data movement to or from memory or storage, computation, communication, or any combination of these.

"Loosely coupled" refers to how these accelerators are exposed and accessed by the host software. Specifically, they are not exposed as processor ISA extensions, but instead as PCI Express countable endpoint devices on the platform. Loose coupling allows these agents to accept work requests from the host software and operate asynchronously to the host processor.

An "accelerator" can be a programmable agent (e.g., GPU/GPGPU), a fixed function agent (e.g., compression or encryption engine), or a reconfigurable agent such as a field programmable gate array (FPGA). Some of these may be used for computation offload, while others (e.g., RDMA or host fabric interfaces) may be used for packet processing, communication, storage, or message passing operations.

Accelerator devices may be physically integrated at different levels, including on-die (i.e., on the same die as the processor), on-package, chipset, motherboard, or may be separate PCIe-attached devices. For integrated accelerators, even if they are listed as PCI Express endpoint devices, some of these accelerators may be coherently attached (to the on-die coherent fabric or to an external coherent interface), while others may be attached to an internal non-coherent interface or to an external PCI Express interface.

At a conceptual level, "accelerators" and high-performance I/O device controllers are similar. What distinguishes them are features such as unified/shared virtual memory, the ability to operate on pageable memory, support for user-mode work submission, task scheduling/preemption, and low-latency synchronization. As a result, accelerators can be considered a new and improved category of high-performance I/O devices.

Off-road operation model

Accelerator offload operation models can be broadly categorized into three usage categories:

1. Streaming: In the streaming offload model, small units of work are streamed to the accelerator at high rates. A typical example of this use case is a network data plane performing various types of packet processing at high rates.

2. Low Latency: For some offload use cases, the latency of the offload operation (both the dispatch of the task to the accelerator and the accelerator executing it) is important. An example of this use case is a low latency message passing composition model involving remote get, put and atomic operations over the host fabric.

3. Scalable: Scalable offloading refers to applications where the services of a computational accelerator are directly accessible (e.g., from the highest ring in a hierarchical protection domain, such as ring 3) to a large number (unlimited number) of client applications (within and through virtual machines) without constraints imposed by accelerator devices such as multiple work queues or multiple doorbells supported on the device. Some of the accelerator devices and processor interconnects described in this specification fall into this category. Such scalability applies to GPUs, GPGPUs, FPGAs, or compression accelerators, or computational offload devices that support time-sharing/scheduling of work, such as message passing, applications such as enterprise databases that have large scalability requirements for lock-free operations.

Work dispatch across off-road models

Each of the above offload operation models presents its own work dispatch challenges, which are described below.

1. Work dispatch for using streaming offload

For streaming usage, the typical work dispatch model is to use a work queue that resides in memory. Specifically, the device is configured with the location and size of the work queue in memory. The hardware implements a doorbell (tail pointer) register that is updated by software when adding a new work element to the work queue. The hardware reports the current head pointer to the software to enforce producer-consumer flow control on the work queue elements. For streaming usage, the typical model is for the software to check if there is space in the work queue by looking at the head and tail pointers cached in software (often maintained in host memory by the hardware to avoid the overhead of a UC MMIO read by the software), add the new work element to the work queue that resides in memory, and update the tail pointer using a doorbell register write to the device.

Doorbell writes are typically 4-byte or 8-byte uncacheable (UC) writes to MMIO. In some processors, UC writes ensure that older stores are globally monitored before issuing the UC write (required for producer-consumer usage), but are serialized operations that block all younger stores in the processor pipeline from being issued until the UC write is posted by the platform. Typical latencies for UC write operations on Xeon server processors are on the order of 80-100 nanoseconds, limiting streaming offload performance during the time that all younger store operations are blocked by the core.

One approach to address serialization of younger stores following UC doorbell writes is to use write combining (WC) store operations for doorbell writes (due to WC's weak ordering), but using WC stores for doorbell writes incurs several challenges. The doorbell write size (typically a DWORD or QWORD) is smaller than the cache line size. These partial writes incur additional latency due to the processor keeping the partial writes in its write combining buffer (WCB) for potential write combining opportunities, and incur latency for doorbell writes issued by the processor. Software can issue these through explicit store fences, resulting in the same serialization for younger stores as for UC doorbells.

Another issue with WC-mapped MMIO is that mispredictions (using MOVNTDQA) and speculative reads are exposed to WC-mapped MMIO (which has registers that can affect the read side). This is cumbersome for devices to deal with, as they would need to host the WC-mapped doorbell registers in a separate page than the rest of the UC-mapped MMIO registers. This also poses challenges for virtualization usage, as the VMM software can no longer ignore the guest memory type and enforce UC mapping for any device MMIO exposed to the guest using the EPT page table.

The MOVDIRI instruction described herein addresses the above limitations of using UC or WC storage for doorbell writes in these streaming offload applications.

2. Work dispatch for low latency offloading

Some types of accelerator devices are highly optimized to complete requested operations with minimal latency. Unlike (throughput-optimized) streaming accelerators, these accelerators typically implement device-hosted work queues (exposed through device MMIO) to avoid DMA read latencies to fetch work elements (and in some cases, the same data buffer) from a memory-hosted work queue. Instead, host software submits work to device-hosted work queues exposed through device MMIO by directly writing work descriptors (and in some cases, data as well). Examples of such devices include host fabric controllers, remote DMA (RDMA) devices, and new storage controllers such as non-volatile memory (NVM) Express. The use of device-hosted work queues poses few challenges with respect to existing ISAs.

To avoid serialization overhead of UC writes, the MMIO address of a device-hosted work queue is typically mapped as a WC. This exposes the same challenges as a doorbell mapped to a WC for streaming accelerators.

Furthermore, using WC stores to device-hosted work queues requires the device to adhere to the write atomicity behavior of some processors. For example, the atomicity of a write operation of up to 8 bytes size only guarantees that some processors write within cache line boundaries (and due to locking operations), but does not define any guaranteed atomicity of write completion. The atomicity of a write operation is the granularity at which a processor store operation is monitored by other agents and is a property of the processor instruction set architecture and coherency protocol. The atomicity of write completion is the granularity at which a non-cacheable store operation is monitored by the receiver (the memory controller in the case of memory, or the device in the case of MMIO). The atomicity of write completion is stronger than the atomicity of a write operation and is a function of the platform as well as the processor instruction set architecture. Without the atomicity of write completion, a processor instruction that performs a non-cacheable store operation of N bytes may be received as multiple (tone) write transactions by the device-hosted work queue. Currently, the device hardware must guard against such tonal writes by tracking each word or piece of work descriptor data that is written to the device-hosted work queue.

The MOVDIR64B instruction described herein addresses the above limitations by supporting 64-byte writes with guaranteed 64-byte write completion atomicity. MOVDIR64B is also useful for other uses, such as writing to persistent memory (NVM attached to a memory controller) and replicating data through the system through a non-transparent bridge (NTB).

3. Work dispatch for scalable offload utilization

The traditional approach for submitting work from an application to an I/O device involves making a system call to the kernel I/O stack, which forwards the request through a kernel device driver to the device I/O controller. While this approach is scalable (any number of applications can share the services of the device), it introduces the latency and overhead of a serialized kernel I/O stack that often becomes a performance bottleneck for high performance devices and accelerators.

To support low overhead work dispatch, some high performance devices support direct ring 3 access that allows direct work dispatch to the device and checks for work completion. In this model, the device allocates and maps some resources (doorbell, work queue, completion queue, etc.) to the application's virtual address space. Once mapped, ring 3 software (user mode drivers or libraries) can dispatch work directly to the accelerator. For devices that support shared virtual memory (SVM) functionality, the doorbell and work queue are set up by the kernel mode driver to identify the process address space identifier (PASID) of the application process to which they are mapped. When processing work items dispatched through a particular work queue, the device uses the respective PASID configured for that work queue for virtual to physical address translation through the I/O memory management unit (IOMMU).

One of the challenges with Direct Ring 3 work submission is the scalability issue. The number of application clients that can submit work directly to an accelerator device depends on the number of queues/doorbells (or device-hosted work queues) supported by the accelerator device. This is because doorbells or device-hosted work queues are statically assigned/mapped to application clients, and the number of these resources supported by the accelerator device design is fixed. Some accelerator devices try to "address" this scalability challenge by over-containment the doorbell resources they have (by dynamically detaching and re-attaching doorbells on demand for applications), but this is often cumbersome and difficult to scale. For devices that support I/O virtualization (e.g., single root I/O virtualization (SR-IOV)), the limited doorbell/work queue resources are further constrained, as they need to be partitioned across various virtual functions (VFs) that are assigned to different virtual machines.

The scaling issue is most important for high performance message passing accelerators used by enterprise applications such as databases for lock-free operations (with some RDMA devices supporting 64K-1M queue pairs) and for computational accelerators that support sharing of accelerator resources across tasks submitted by multiple clients.

The ENQCMD/S instructions described herein address the above scaling limitations to enable an unlimited number of clients to subscribe and share the resources of a work queue on an accelerator.

In one embodiment, new types of store operations by the processor core include direct stores and enqueue stores.

In one embodiment, direct stores are generated by the MOVDIRI and MOVDIR64B instructions described herein.

Cacheability: Like UC and WC stores, direct stores are not cacheable. When a direct store is issued to an address that is cached, the line is written back (if modified) and invalidated from the cache before the direct store.

Memory ordering: Like WC stores, direct stores are weakly ordered. Specifically, they are not ordered with respect to older WB/WC/NT stores, CLFLUSHOPT, and CLWB to different addresses. A younger WB/WC/NT store, CLFLUSHOPT, or CLWB to a different address can pass an older direct store. Direct stores to the same address are always ordered with older stores (including direct stores) to the same address. Direct stores are fenced by any operation that enforces store fencing (e.g., SFENCE, MFENCE, UC/WP/WT stores, lock, IN/OUT instructions, etc.).

Write combining: Direct stores have different write combining behavior than normal WC stores. Specifically, direct stores are subject to immediate eviction from the write combining buffer and thus cannot be combined with younger stores (including direct stores) to the same address. Older WC/NT stores held in the write combining buffer may be combined with younger direct stores to the same address, and applications that need to avoid such combinations must explicitly store fenced WC/NT stores before performing a direct store to the same address.

Atomicity: Direct stores support atomicity of write completion with respect to the write size of the instruction issuing the direct store. In the case of MOVDIRI, if the destination is 4-byte aligned (respectively 8-byte aligned), the atomicity of the write completion is 4-byte (respectively 8-byte). For MOVDIR64B, the destination is forced to be 64-byte aligned and the atomicity of the write completion is 64-byte. The atomicity of the write completion ensures that the direct store is not torn into multiple write transactions as processed by the memory controller or root complex. The root complex implementation on processors that support direct stores ensures that the direct store is forwarded on the external PCI Express fabric (and the internal I/O fabric within the SoC which follows PCI Express ordering) as a single non-torn posted write transaction. A read operation from any agent (processor or non-processor agent) to a memory location sees either all or none of the data written by instructions issuing direct store operations.

Ignoring destination memory type: Direct stores ignore the destination address memory type (including UC/WP types) and always follow weak ordering. This allows software to map device MMIO as UC and access certain registers (e.g. doorbell or device hosted work queue registers) using direct store instructions (MOVDIRI or MOVDIR64B) while continuing to access other registers that may have strict serialization requirements using normal MOV operations that follow the UC ordering per the memory type of the mapped UC. This also allows direct store instructions to work from within guest software, while virtual machine monitor (VMM) software (which has no device specific knowledge) maps guest exposed MMIO as UC in the processor extended page table (EPT) and ignores the guest memory type.

SoCs that support direct store must ensure atomicity of write completion for direct store as follows:

Direct Stores to Main Memory: For direct stores to main memory, the coherent fabric and system agents ensure that all data bytes in a direct store are issued to the home agent or other global observability (GO) point for requests to memory as a single (non-ton) write transaction. For platforms that support persistent memory, the home agent, memory controller, memory-side cache, in-line memory encryption engine, memory bus that attaches persistent memory (e.g., DDR-T), and persistent memory controller must each support the same or higher granularity of write completion atomicity for direct stores. Thus, software can perform a 64-byte direct store to memory (volatile or persistent) using MOVDIR64B and be guaranteed that all 64-byte writes are processed atomically by all agents. As with normal writes to persistent memory, if software needs to explicitly commit to persistence, the software performs the direct store in a fence/commit/fence sequence.

Direct Stores to Memory Mapped I/O: For direct stores to memory mapped I/O (MMIO), the coherent fabric and system agents must ensure that all data bytes in a direct store are issued to the root complex (the global observability point for requests to MMIO) as a single (non-ton) write transaction. The root complex implementation must ensure that each direct store is processed and forwarded as a single (non-ton) posted write transaction on the internal I/O fabric that attaches the PCI Express root complex integrated endpoint (RCIEP) and root port (RP). PCI Express root ports and switch ports must forward each direct store as a single posted write transaction. Atomicity of write completion is not specified or guaranteed for direct stores targeting devices on or behind secondary bridges (e.g., legacy PCI, PCI-X bridges) or secondary buses (e.g., USB, LPC, etc.).

Note that some SoC implementations already guarantee atomicity of write completion for WC write requests. In particular, partial line WC writes (WCiL) and full line WC writes (WCiLF) are already handled with atomicity of write completion by the system agent, memory controller, root complex, and I/O fabric. For such embodiments, there is no need for the processor to distinguish direct writes from WC writes; the behavioral difference between direct and WC stores is internal to the processor core. Therefore, no changes are proposed to the internal or external fabric specifications for direct writes.

Handling of direct writes received by a PCI Express endpoint or RCIEP is device implementation specific. Depending on the device's programming interface, the device and its drivers may require some of its registers (e.g., doorbell registers or device-hosted work queue registers) to always be written with a direct store instruction (e.g., MOVDIR64B) and handle these atomically within the device. Writes to other registers on the device may be handled by the device without any consideration or expectation of atomicity. With respect to the RCIEP, if registers with write atomicity requirements are implemented for access through a sideband or private wire interface, such embodiments must ensure the write atomicity property through implementation specific means.

In one embodiment, an enqueue store is generated by the ENQCMD and ENQCMDS instructions described herein. The specified target of the enqueue store is a shared work queue (SWQ) on the accelerator device. In one embodiment, the enqueue store has the following characteristics:

Non-Posted: The enqueue store generates a 64-byte non-posted write transaction to the target address and receives a completion response indicating success or retry status. The success/retry status returned in the completion response may be returned to software via the ENQCMD/S instruction (e.g., in a zero flag).

Cacheability: In one embodiment, enqueue stores are not cacheable. Platforms that support enqueue stores enforce that enqueue non-posted writes are only forwarded to address (MMIO) ranges that are explicitly enabled to accept these stores.

Memory ordering: An enqueue store may update architectural state (e.g., the zero flag) with non-posted write completion status. Thus, at most one enqueue store may be outstanding from a given logical processor. In that sense, an enqueue store from a logical processor cannot pass another enqueue store issued from the same logical processor. Enqueue stores do not order to different addresses relative to older WB/WC/NT stores, CLFLUSHOPT, or CLWB. Software that needs to enforce such ordering may use explicit store fencing after such stores and before the enqueue store. Enqueue stores are always ordered to the same address with older stores.

Alignment: The ENQCMD/S instruction forces the enqueue store destination address to be 64-byte aligned.

Atomicity: Enqueue stores generated by the ENQCMD/S instructions support atomicity of 64-byte write completion. Atomicity of write completion ensures that enqueue stores are not torn into multiple transactions as processed by the root complex. Root complex implementations on processors that support enqueue stores ensure that each enqueue store is forwarded to the endpoint device as a single (non-tonn) 64-byte non-posted write transaction.

Ignore destination memory type: Like direct stores, enqueue stores ignore the destination address memory type (including UC/WP type) and always follow the ordering as described above. This allows software to continue to map device MMIO as UC and access shared work queue (SWQ) registers with ENQCMD/S instructions while continuing to access other registers with normal MOV instructions or through direct store (MOVDIRI or MOVDIR64B) instructions. This also allows enqueue store instructions to operate from within guest software, while the VMM software (which has no device-specific knowledge) maps guest exposed MMIO as UC in the processor extended page table (EPT) and ignores the guest memory type.

Platform considerations for enqueue storage

For some embodiments, a specific set of platform integration devices support shared work queue (SWQ) functionality. These devices may be attached to the root complex through the internal I/O fabric. These devices may be exposed to the host software as either PCI Express root complex integrated endpoints (RCIEPs) or PCI Express endpoint devices behind a virtual root port (VRP).

Platforms that support integrated devices with SWQs should restrict the transmission of enqueue non-posted write requests on the internal I/O fabric to only such devices. This is to ensure that the new transaction type (enqueue non-posted write) is not treated as a malformed transaction layer packet (TLP) by enqueue-unaware endpoint devices.

Enqueues to all other addresses (including the main memory address range and all other memory mapped address ranges) are terminated by the platform and a normal (non-error) response is returned to the issuing processor along with a retry completion status. Because unprivileged software (ring 3 software in VMX non-root mode or ring 0 software) can generate enqueue non-posted write transactions by executing the ENQCMD/S instruction, no platform error is generated upon termination of such an enqueue.

The root complex implementation should ensure that enqueue stores are processed and forwarded as a single (non-ton), non-posted write transaction on the internal I/O fabric for integrated devices that support SWQ.

Examination of platform performance

This section describes some performance considerations for the system agent and the processing of enqueue storage by the system agent.

Relaxed ordering for System Agent Tracker (TOR) entry allocation for enqueue storage.

To maintain memory consistency, system agent implementations typically enforce strict ordering on requests to cache line addresses (when allocating TOR entries) for coherent memory and MMIO. While this is required to support overall ordering for coherent memory accesses, this strict ordering on enqueue stores incurs performance issues because it is common for enqueue stores to target a shared work queue (SWQ) on the device, thereby causing enqueue store requests to be issued from multiple logical processors with the same destination SWQ address. Also, unlike regular stores posted to the system agent, enqueue stores are non-posted and incur similar latency to reads. To negate the requirement of allowing only one outstanding enqueue store to the shared work queue, a system agent implementation is required to relax the strict ordering on enqueue store requests to the same address, and instead allow TOR allocation for multiple in-flight enqueue stores to the same address. A logical processor can only issue at most one enqueue store at a time, so the system agent/platform can process each enqueue store independently without worrying about ordering.

Support for multiple outstanding enqueued non-posted writes in I/O bridge agents.

I/O bridge implementations typically limit the number of non-posted (read) requests supported in the downstream path to a small number (often to a single request). This is because reads from the processor to the MMIO (usually UC reads) are not performance critical for most uses, and support a large queue depth due to the buffers required to read the returned data, which increases the hardware cost. Since it is expected that enqueue stores will typically be used for work dispatch to accelerator devices, applying this limited queuing for enqueue non-posted writes can be detrimental to performance. I/O bridge implementations are encouraged to support increased queue depths (some actual percentage of the number of logical processors, since a logical processor can only have one outstanding enqueue store request at a time) for improved enqueue non-posted write bandwidth. Unlike read requests, enqueue stores do not incur the hardware cost of data buffers, since enqueue non-posted write completions simply return a completion status (success vs. retry) and do not return data.

Virtual channel support for enqueue non-posted writes

Unlike typical memory read and write requests on the I/O bus, which have producer-consumer ordering requirements (e.g., as specified by PCI Express transaction ordering), enqueue non-posted writes do not enforce ordering requirements on the I/O bus. This allows the use of a non-VC0 virtual channel to issue enqueue non-posted writes and return their respective completions. The benefit of using a non-VC0 channel is that enqueue non-posted write completions can have better latency (fewer cycles delaying the core) by avoiding being ordered behind upstream posted writes on VC0 from the device to the host. Implementations are encouraged to carefully consider their use of the integrated device to minimize enqueue non-posted completion latency.

Intermediate stop of enqueue non-posted light

To handle certain flow control in high latency situations (e.g. waking up an internal link or power management on a locked flow), an intermediate agent (system agent, I/O bridge, etc.) may drop the normal enqueue store request and return completion to the issuing core with a retry response. The software issuing the enqueue store has no direct visibility into when the retry response is from the intermediate agent or target, or when the software will retry normally (potentially with some backoff).

Implementations that perform such intermediate stops must be very careful to ensure that such behavior cannot expose any denial of service attacks across software clients that share the SWQ.

Support for shared work queues on endpoint devices

Figure 34 illustrates the concept of a shared work queue (SWQ), allowing multiple non-cooperative software agents (applications 3410-3412) to submit work through a shared work queue 3401 using the ENQCMD/S commands described herein.

The following considerations are applicable to endpoint devices that implement a shared work queue (SWQ).

SWQs and their enumeration: A device physical function (PF) may support one or more SWQs. Each SWQ is 64-byte aligned within the device MMIO address range and is accessible for enqueue non-posted writes through a size register (referred to herein as SWQ_REG). It is recommended that each such SWQ_REG on a device be located in a unique system page size (4KB) region. The device driver for the device is responsible for reporting/enumerating the SWQ capabilities, number of SWQs supported, and corresponding SWQ_REG addresses to software through an appropriate software interface. The driver may selectively report the depth of the SWQs supported for software tuning or informational purposes (although this is not required for functional correctness). For devices that support multiple physical functions, it is recommended that each physical function support SWQs independently.

SWQ support on Single Root I/O Virtualization (SR-IOV) devices: Devices that support SR-IOV may support independent SWQs for each Virtual Function (VF) exposed through a SWQ_REG in each VF Base Address Register (BAR). This design allows for maximum performance isolation for work submission across VFs and may be appropriate for a small to moderate number of VFs. For devices that support a large number of VFs (where a separate SWQ per VF is not practical), a single SWQ may be shared across multiple VFs. Even in this case, each VF has its own private SWQ_REG in its VF BAR, but is backed by a common SWQ across the VFs that share the SWQ. For such device designs, the VFs that share the SWQ may be statically determined by the hardware design, or the mapping between SWQ_REGs of a given VF to SWQ instances may be dynamically set up/toned down through the physical function and its driver. Device designs that share SWQs across VFs need to pay special attention to QoS and protection against denial of service attacks, as described later in this section. When sharing SWQs across VFs, care must be taken in the device design to identify which VF received an enqueue request that was accepted into the SWQ. When dispatching work requests from the SWQ, the device should ensure that upstream requests are properly tagged with the requester ID (bus/device/function #) of the respective VF (in addition to the PASID carried in the enqueue request payload).

Enqueue Non-Posted Write Addresses: Endpoint devices that support SWQ are required to accept enqueue non-posted writes to any address forwarded through their PF or VF memory BARs. For any enqueue non-posted write request received by an endpoint device to an address that is not a SWQ_REG address, the device may be required not to treat this as an error (e.g., malformed TLP, etc.) but instead to return a completion with retry completion status (MRS). This may be done to ensure that non-privileged (ring 3 or ring 0 VMX guest) software use of the ENQCMD/S instruction that mistakenly or maliciously issues an enqueue store to a non-SWQ_REG address on a SWQ-capable device cannot result in reporting a non-fatal or fatal error with platform-specific error handling consequences.

Non-enqueue request handling for SWQ_REG: An endpoint device that supports SWQ may silently drop non-enqueue requests (normal memory writes and reads) to the SWQ_REG address without treating them as fatal or non-fatal errors. Read requests to the SWQ_REG address may return a successful completion response (as opposed to UR or CA) with all-ones values for the requested data bytes. Normal memory (posted) write requests to the SWQ_REG address are simply dropped without action by the endpoint device. This may be done to ensure that unprivileged software cannot generate normal read and write requests to the SWQ_REG address in a way that causes them to erroneously or maliciously report non-fatal or fatal errors with platform-specific error handling results.

SWQ queue depth and storage: SWQ queue depth and storage are device implementation specific. The device design should ensure that sufficient queue depth is supported for the SWQ to achieve maximum utilization of the device. Storage for the SWQ may be implemented on the device. An integrated device on a SoC may use stolen main memory (non-OS visible private memory reserved for device use) as buffer overflow for the SWQ, allowing a larger SWQ queue depth than can be achieved with on-device storage. For such designs, the use of buffer overflow is transparent to the software since the device hardware decides when to overflow (versus dropping the enqueue request and sending a retry completion status) and fetch from the buffer overflow for command execution, maintaining any command-specific ordering requirements. For all purposes, such buffer overflow utilization is equivalent to a separate device using local device attached DRAM for SWQ storage. Device designs involving buffer overflows in stolen memory must be very careful to ensure that such stolen memory is protected from any access other than reading and writing the buffer overflows by the device to which it is allocated.

Non-blocking SWQ behavior: For performance reasons, device implementations should respond promptly to enqueue non-posted write requests with success or retry completion status and should not block enqueue completion of SWQ capacity that would be freed up to accept the request. The decision to accept or reject an enqueue request to a SWQ may be based on capacity, QoS/occupancy or other policies. Some example QoS considerations are described next.

SWQ QoS considerations: For enqueue non-posted writes targeted to SWQ_REG addresses, the endpoint device may apply admission control to decide to accept the request for the respective SWQ (and send a successful completion status) or drop it (and send a retry completion status). Admission control may be device and usage specific, and the specific policies supported/enforced by the hardware may be exposed to software through a physical function (PF) driver interface. Because the SWQ is a shared resource with multiple producer clients, the device implementation must ensure adequate protection against denial of service attacks across producers. The QoS for the SWQ refers solely to the acceptance of work requests (through enqueue requests) to the SWQ and is orthogonal to any QoS applied by the device hardware on how the QoS is applied to share the device's execution resources when processing work requests submitted by different producers. Several exemplary approaches for configuring endpoint devices to enforce admission policies for accepting enqueue requests to the SWQ are described below. These are listed for illustrative purposes only, and the exact implementation choices are device specific.

In one embodiment, the MOVDIRI instruction moves a doubleword integer in a source operand (second operand) to a destination operand (first operand) using a direct store operation. The source operand may be a general-purpose register. The destination operand may be a 32-bit memory location. In 64-bit mode, the default operation size of the instruction is 32 bits. MOVDIRI defines the destination to be doubleword or quadword aligned.

Direct stores may be implemented by using a write combining (WC) memory type protocol for writing data. With this protocol, the processor neither writes the data to the cache hierarchy nor fetches the corresponding cache line from memory to the cache hierarchy. If the destination address is cached, the line is written back (if modified) and invalidated from the cache before the direct store. Unlike stores with non-transient implications, which allow Uncached (UC) and Write Protected (WP) memory types for the destination to override the non-transient implications, direct stores always follow the WC memory type protocol regardless of the destination address memory type (including UC and WP types).

Unlike WC stores and stores with non-transient hinting, direct stores are subject to immediate eviction from the write combining buffer and thus cannot be combined with younger stores (including direct stores) to the same address. Older WC and non-transient stores held in the write combining buffer may be combined with younger direct stores to the same address.

The WC protocol used by direct stores follows a weakly ordered memory consistency model, so the fencing process should follow the MOVDIRI instruction to enforce ordering when necessary.

Direct stores issued by MOVDIRI to the destination are aligned on 4-byte boundaries to guarantee atomicity of 4-byte write completion. This means that the data arrives at the destination in a single non-tone 4-byte (or 8-byte) write transaction. If the destination is not aligned to the write size, a direct store issued by MOVDIRI arrives at the destination split in two parts. The parts of such a split direct store do not merge with the younger store, but can arrive at the destination in any order.

Figure 59 illustrates an embodiment of a method performed by a processor to process the MOVDIRI instruction. For example, hardware details are used here.

At 5901, an instruction is fetched. For example, MOVDIRI is fetched. The MOVDIRI instruction includes an opcode (and, in some embodiments, a prefix), a destination field representing a destination operand, and a source field representing a source register operand.

At 5903, the fetched instruction is decoded. For example, the MOVDIRI instruction is decoded by a decode circuit as described in more detail herein.

At 5905, data values associated with the source operands of the decoded instruction are obtained. Further, in some embodiments, the instruction is scheduled.

At 5907, the decoded instruction is executed by execution circuitry (hardware) as described in detail herein, which moves doubleword-sized data from source register operands to destination register operands without caching the data.

In some embodiments, at 5909, the instruction is committed or retired.

Moves 64 bytes from a source memory address to a destination memory address as a direct store with 64-byte write atomicity. The source operand is a regular memory operand. The destination operand is a memory location specified in a general-purpose register. The register contents are interpreted as an offset into the ES segment without using any segment overrides. In 64-bit mode, the register operand width is 64 bits (or 32 bits). Outside 64-bit mode, the register width is 32 bits or 16 bits. MOVDIR64B requires that the destination address is 64-byte aligned. No alignment restrictions are enforced on the source operand.

MOVDIR64B reads 64 bytes from the source memory address and performs a 64-byte direct store operation to the destination address. The load operation follows normal read ordering based on the memory type of the source address. The direct store is implemented by using the Write Combining (WC) memory type protocol for writing the data. Using this protocol, the processor does not need to write the data to the cache hierarchy and does not need to fetch the corresponding cache line from memory to the cache hierarchy. If the destination address is cached, the line is written back (if modified) and invalidated from the cache before the direct store.

Unlike stores with non-transient implications, which allow the UC/WP memory type for the destination to override the non-transient implications, direct stores may follow the WC memory type protocol regardless of the destination address memory type (including UC/WP types).

Unlike WC stores and stores with non-transient hinting, direct stores are subject to immediate eviction from the write combining buffer and thus cannot be combined with younger stores (including direct stores) to the same address. Older WC and non-transient stores held in the write combining buffer may be combined with younger direct stores to the same address.

The WC protocol used by direct stores follows a weakly ordered memory consistency model, so the fencing process should follow the MOVDIR64B instruction to enforce ordering when necessary.

There are no atomicity guarantees provided for the 64-byte load operation from the source address, and processor implementations may use multiple load operations to read the 64 bytes. The 64-byte direct store issued by MOVDIR64B guarantees atomicity of the 64-byte write completion, meaning that the data arrives at the destination in a single non-tone 64-byte write transaction.

Figure 60 illustrates an embodiment of a method performed by a processor to process the MOVDIRI64B instruction. For example, hardware details are used here.

At 6001, an instruction is fetched. For example, MOVDIRI64B is fetched. The MOVDIRI64B instruction includes an opcode (and, in some embodiments, a prefix), a destination field representing the destination operand, and a source field representing the source register operand.

At 6003, the fetched instruction is decoded. For example, the MOVDIRI64B instruction is decoded by a decode circuit as described in detail herein.

At 6005, data values associated with the source operands of the decoded instruction are obtained. Further, in some embodiments, the instruction is scheduled.

At 6007, the decoded instruction is executed by execution circuitry (hardware) as described in detail herein, which moves 64 bytes of data from the source register operand to the destination register operand without caching the data.

In some embodiments, at 6009, the instruction is committed or retired.

In one embodiment, the ENQCMD command enqueues a 64-byte command using a non-posted write with 64-byte write atomicity from a source memory address (second operand) to a device shared work queue (SWQ) memory address in the destination operand. The source operand is a regular memory operand. The destination operand is a memory address specified in a general purpose register. The register contents are interpreted as an offset into the ES segment without using any segment overrides. In 64-bit mode, the register operand width is 64-bits or 32-bits. Outside 64-bit mode, the register width is 32-bits or 16-bits. ENQCMD requires that the destination address is 64-byte aligned. There are no alignment restrictions enforced on the source operand.

In one embodiment, ENQCMD reads a 64 byte command from a source memory address, formats the 64 byte enqueue store data, and performs an enqueue store operation of the 64 byte store data to the destination address. The load operation follows normal read ordering based on the memory type of the source address. A general protection error can be caused if the lower 4 bytes of the 64 byte command data read from the source memory address have a non-zero value or if the PASID valid field bit is 0. Otherwise, the 64 byte enqueue store data is formatted as follows:
Enqueue stored data [511:32] = command data [511:32]
Enqueue storage data [31] = 0
Enqueue stored data [30:20] = 0
Enqueue store data [19:0] = PASID MSR [19:0]

In one embodiment, the 64 bytes of enqueue store data generated by ENQCMD has the format shown in FIG. 58. The upper 60 bytes in the command descriptor specify a command 5801 specific to the target device. The PRIV field 5802 (bit 31) may be forced to 0 to convey user privileges for the enqueue store generated by the ENQCMD instruction. The PASID field (bits 19:0) 5804 conveys the process address space identification (as programmed in the PASID MSR) assigned by the system software for the software thread executing ENQCMD1.

The enqueue store operation uses a non-posted write protocol to write the 64 bytes of data. The non-posted write protocol does not require writing the data to the cache hierarchy and does not require fetching the corresponding cache line in the cache hierarchy. The enqueue store always follows the non-posted write protocol regardless of the destination address memory type (including UC/WP types).

The Non-Posted Write protocol may return a completion response to indicate success or retry status for the Non-Posted Write. The ENQCMD command may return this completion status with zero flags (0 indicates success, 1 indicates retry). A success status indicates that the Non-Posted Write data (64 bytes) was accepted (but not necessarily acted upon) by the target shared work queue. A retry status indicates that the Non-Posted Write was not accepted by the destination address due to capacity or other temporary reasons (or because the destination address is not a valid shared work queue address).

In one embodiment, at most one enqueue store can be outstanding from a given logical processor. In that sense, an enqueue store cannot pass another enqueue store. Enqueue stores do not order to different addresses with respect to older WB stores, WC and non-temporal stores, CLFLUSHOPT, or CLWB. Software that needs to enforce such ordering must use explicit store fencing after such stores and before the enqueue store. ENQCMD only affects shared work queue (SWQ) addresses that are not affected by other stores.

There are no atomicity guarantees provided for the 64-byte load operation from the source address, and processor implementations may read the 64 bytes using multiple load operations. The 64-byte enqueue store issued with ENQCMD guarantees the atomicity of the 64-byte write completion. The data may arrive at the destination as a single non-ton 64-byte non-posted write transaction.

In some embodiments, the MSR of the PASID architecture is used by the ENQCMD instruction.

Figure 61 illustrates an embodiment of a method performed by a processor to process the ENCQMD instruction. For example, hardware details are used here.

At 6101, an instruction is fetched. For example, an ENCQMD is fetched. The ENCQMD instruction includes an opcode (and, in some embodiments, a prefix), a destination field representing a destination memory address operand, and a source field representing a source memory operand.

At 6103, the fetched instruction is decoded. For example, the ENCQMD instruction is decoded by a decode circuit as described in detail herein.

At 6105, data values associated with the source operands of the decoded instruction are obtained. Further, in some embodiments, the instruction is scheduled.

At 6107, the decoded instructions are executed by execution circuitry (hardware) as described in detail herein, which writes the command (the retrieved data) to a destination memory address. In some embodiments, the destination memory address is a shared work queue.

In some embodiments, at 6109, the instruction is committed or retired.

In one embodiment, the ENQCMDS instruction enqueues a 64-byte command using a non-posted write with 64-byte write atomicity from a source memory address (second operand) to a device shared work queue (SWQ) memory address in the destination operand. The source operand is a regular memory operand. The destination operand is a memory address specified in a general-purpose register. The register contents may be interpreted as an offset into the ES segment without using any segment overrides. In 64-bit mode, the register operand width is 64-bits or 32-bits. Outside 64-bit mode, the register width is 32-bits or 16-bits. ENQCMD requires that the destination address is 64-byte aligned. There are no alignment restrictions enforced on the source operand.

Unlike ENQCMD (which may be executed from any privilege level), ENQCMDS is a privileged instruction. If the processor is running in protected mode, the CPL must be 0 to execute this instruction. ENQCMDS reads a 64-byte command from a source memory address and performs a 64-byte enqueue store operation with this data to the destination address. The load operation follows normal read ordering based on the memory type of the source address. The 64 bytes of enqueue store data are formatted as follows:
Enqueue stored data [511:32] = command data [511:32]
Enqueue stored data [31] = command data [31]
Enqueue stored data [30:20] = 0
Enqueue store data [19:0] = command data [19:0]

The 64-byte enqueue store data generated by ENQCMDS may have the same format as ENQCMD. In one embodiment, ENQCMDS has the format shown in FIG. 62.

The upper 60 bytes in the command descriptor specify a command 6201 specific to the target device. The PRIV field (bit 31) 6202 is specified by bit 31 in the command data at the source operand address to convey either user (0) or supervisor (1) privilege for the enqueue store generated by the ENQCMDS instruction. The PASID field (bits 19:0) 6204 conveys the process address space identification as specified by bits 19:0 in the command data at source operand address 1.

In one embodiment, the enqueue store operation uses a non-posted write protocol to write 64 bytes of data. The non-posted write protocol does not write the data to the cache hierarchy, nor does it fetch the corresponding cache line from the cache hierarchy. The enqueue store always follows the non-posted write protocol, regardless of the destination address memory type (including UC/WP types).

The Non-Posted Write protocol returns a completion response to indicate success or retry status for a Non-Posted Write. The ENQCMD command returns this completion status with zero flags (0 indicates success, 1 indicates retry). A success status indicates that the Non-Posted Write data (64 bytes) was accepted (but not necessarily acted upon) by the target shared work queue. A retry status indicates that the Non-Posted Write was not accepted by the destination address due to capacity or other temporary reasons (or because the destination address is not a valid shared work queue address).

At most one enqueue store (ENQCMD or ENQCMDS) may be outstanding from a given logical processor. In that sense, an enqueue store cannot pass another enqueue store. Enqueue stores may not be ordered to different addresses with respect to older WB stores, WC and non-temporal stores, CLFLUSHOPT, or CLWB. Software that needs to enforce such ordering may use explicit store fencing after such stores and before the enqueue stores.

ENQCMDS only affects shared work queue (SWQ) addresses that are not affected by other stores.

There are no atomicity guarantees provided for the 64-byte load operation from the source address, and a processor implementation may read the 64 bytes using multiple load operations. The 64-byte enqueue store issued by ENQCMDS guarantees the atomicity of the 64-byte write completion (i.e., it reaches the destination as a single non-ton 64-byte non-posted write transaction).

Figure 63 illustrates an embodiment of a method performed by a processor to process the ENCQMD instruction. For example, hardware details are used here.

At 6301, an instruction is fetched. For example, an ENCQMD is fetched. The ENCQMD instruction includes an opcode (and, in some embodiments, a prefix), a destination field representing a destination memory address operand, and a source field representing a source memory operand.

At 6303, the fetched instruction is decoded. For example, the ENCQMD instruction is decoded by a decode circuit as described in detail herein.

At 6305, data values associated with the source operands of the decoded instruction are obtained. Further, in some embodiments, the instruction is scheduled.

At 6307, the decoded instruction is executed in privileged mode by execution circuitry (hardware) as described in detail herein, which writes the command (the retrieved data) to a destination memory address. In some embodiments, the destination memory address is a shared work queue.

In some embodiments, at 6309, the instruction is committed or retired.

In one embodiment, two instructions are utilized to ensure efficient synchronization between the accelerator and the host processor, namely UMONITOR and UMWAIT. Briefly, the UMONITOR instruction arms the address monitoring hardware with the address specified in the source register, and the UMWAIT instruction instructs the processor to enter an implementation-dependent optimized state while monitoring a range of addresses.

The UMONITOR instruction arms the address monitoring hardware with the address specified in the r32/r64 source register (the address range that the monitoring hardware checks for store operations can be determined by using the CPUID monitor leaf function). Stores to addresses within the specified address range trigger the monitoring hardware. The state of the monitor hardware is used by UMWAIT.

The following operand encodings are used in one embodiment of the UMONITOR instruction:

The contents of the r32/r64 source register is a valid address (in 64-bit mode, r64 is used). By default, the DS segment is used to create the linear address to be monitored. Segment overrides may be used. The address range must use write-back type memory. Only write-back memory is guaranteed to correctly trigger the monitoring hardware.

The UMONITOR instruction is ordered as a load operation with respect to other memory transactions. The instruction is subject to the permission checks and faults associated with byte loads. Like loads, UMONITOR sets the A bit in the page table, but not the D bit.

UMONITOR and UMWAIT may be executed at any privilege level. The operation of the instructions is the same in non-64-bit and 64-bit modes.

UMONITOR is not used simultaneously with the legacy MWAIT instruction. If MWAIT is executed followed by UMONITOR before the most recent execution of the legacy MONITOR instruction, MWAIT does not have to enter an optimized state. Execution resumes at the instruction following MWAIT.

The UMONITOR instruction aborts a transaction when used within a transactional region.

UMONITOR sets up the monitor hardware address range using the contents of the source register as a valid address and places the monitor hardware in an armed state. Stores to the specified address range trigger the monitor hardware.

Figure 64 illustrates an embodiment of a method performed by a processor to process a UMONITOR instruction. For example, hardware details are used here.

At 6401, an instruction is fetched. For example, UMONITOR is fetched. The UMONITOR instruction includes an opcode (and, in some embodiments, a prefix) and an explicit source register operand.

At 6403, the fetched instruction is decoded. For example, the UMONITOR instruction is decoded by a decode circuit as described in detail herein.

At 6405, data values associated with the source operands of the decoded instruction are obtained. Further, in some embodiments, the instruction is scheduled.

At 6407, the decoded instruction is executed by execution circuitry (hardware) as described in detail herein, which enables monitoring hardware for storage at the address specified by the obtained source register data.

In some embodiments, at 6409, the instruction is committed or retired.

UMWAIT instructs the processor to enter an implementation-dependent optimized state while monitoring a range of addresses. The optimized state may be either a lightweight power/performance optimized state or an improved power/performance optimized state. The selection between the two states is controlled by an explicit input register bit[0] source operand.

UMWAIT may be executed at any privilege level. The operation of this instruction is the same in non-64-bit and 64-bit modes.

The input register may contain information such as the preferred and optimized state the processor should be in, as described in the table below. Bits other than bit 0 are reserved and result in #GP if non-zero.

The instruction wakes up when the timestamp counter reaches or exceeds an implicit 64-bit input value (if the monitoring hardware has not been previously triggered).

Before executing the UMWAIT instruction, the operating system may specify the maximum delay that the processor is allowed to suspend its operations, which may include one of two power/performance optimized states. It can do so by writing the TSC quanta value to the following 32-bit MSR:
UMWAIT_CONTROL[31:2] - Determines the maximum time in a TSC quantum that a processor can be in either C0.1 or C0.2. A value of zero indicates that there is no OS imposed limit on the processor. The maximum time value is a 32-b value where the most significant 30-b comes from this field and the least significant 2 bits are assumed to be zero.
UMWAIT_CONTROL[1] - Reserved.
UMWAIT_CONTROL[0] - C0.2 is not allowed by the OS. A value of 1 means that all C0.2 requests fall back to C0.1.

In one embodiment, if the processor that executed the UMWAIT instruction woke up due to the expiration of an operating system time limit, the instruction sets the carry flag; otherwise, the flag is cleared.

The UMWAIT instruction aborts a transaction when used within a transactional region. In one embodiment, the UMWAIT instruction works in conjunction with the UMONITOR instruction. The two instructions allow the definition of an address to wait at (UMONITOR) and allow an implementation-dependent optimized operation to begin at the wait address (UMWAIT). Execution of UMWAIT is an indication to the processor that it can enter an implementation-dependent optimized state while waiting for an event or store operation to the address range actionable by UMONITOR.

The following events may cause the processor to exit the optimized state in an implementation-dependent manner: a store in an address range that can be acted upon by the UMONITOR instruction, a non-maskable interrupt (NMI) or system management interrupt (SMI), a debug exception, a machine check exception, the BINIT# signal, the INIT# signal, and the RESET# signal. Other implementation-dependent events may cause the processor to exit the optimized state in an implementation-dependent manner.

Furthermore, an external interrupt may cause the processor to exit a state that is optimized depending on the implementation, regardless of whether maskable interrupts are blocked.

Following an exit from the state, which may be optimized depending on the implementation, control is passed to the instruction following the UMWAIT instruction. Any unmasked pending interrupts (including NMI or SMI) may be delivered before the instruction is executed.

Unlike the HLT instruction, the UMWAIT instruction does not support resuming at a UMWAIT instruction following the handling of an SMI. If a preceding UMONITOR instruction did not enable the address range correctly, or if UMONITOR was not executed prior to the execution of UMWAIT followed by the most recent execution of a legacy MONITOR instruction (UMWAIT was not used simultaneously with MONITOR), the processor will not enter an optimized state. Execution resumes at the instruction following UMWAIT.

Note that UMWAIT is used to enter the C0-substate, which is numerically lower than C1, and therefore a store to an address range operable by the UMONITOR instruction will cause the processor to exit UMWAIT either if the store is issued by another processor agent or if the store is issued by a non-processor agent.

Figure 65 illustrates an embodiment of a method performed by a processor to process the UMWAIT instruction. For example, hardware details are used here.

At 6501, an instruction is fetched. For example, UMWAIT is fetched. The UMWAIT instruction includes an opcode (and, in some embodiments, a prefix) and an explicit source register operand.

At 6503, the fetched instruction is decoded. For example, the UMWAIT instruction is decoded by a decode circuit as described in detail herein.

At 6505, data values associated with the source operands of the decoded instruction are obtained. Further, in some embodiments, the instruction is scheduled.

At 6507, the decoded instruction is executed by execution circuitry (hardware), as described in detail herein, which places the processor (or core) in an implementation-dependent state defined by the data for the explicit source register operands while monitoring the address range.

In some embodiments, at 6509, the instruction is committed or retired.

TPAUSE commands the processor to enter an implementation dependent optimized state. There are two such optimized states to choose between: a light power/performance optimized state and an improved power/performance optimized state. The selection between the two is governed by an explicit input register bit[0] source operand.

TPAUSE may be executed at any privilege level. The operation of this instruction is the same in non-64-bit and 64-bit modes.

Unlike PAUSE, the TPAUSE instruction does not cause an abort when used inside a transactional region. The input registers contain information such as the preferred and optimized state the processor will be in, as described in the table below. Bits other than bit 0 are reserved and, if non-zero, yield #GP.

The instruction wakes up when the timestamp counter reaches or exceeds an implicit 64-bit input value (unless the monitoring hardware has been previously triggered). Before executing the TPAUSE instruction, the operating system may specify the maximum delay that the processor is allowed to suspend its operation in either of two power/performance optimized states. It can do so by writing the TSC quanta value to the following 32-bit MSR:
UMWAIT_CONTROL[31:2] - Determines the maximum time in a TSC quantum that a processor can be in either C0.1 or C0.2. A value of zero indicates that there is no OS imposed limit on the processor. The maximum time value is a 32-b value where the most significant 30-b comes from this field and the least significant 2 bits are assumed to be zero.
UMWAIT_CONTROL[1] - Reserved.
UMWAIT_CONTROL[0]--C0.2 is not allowed by the OS. A value of '1 means that all C0.2 requests fall back to C0.1.

A wakeup reason due to expiration of an OS time limit may be indicated by setting the carry flag.

If the processor that executed the TPAUSE instruction woke up due to the expiration of an operating system time limit, the instruction sets the carry flag; otherwise, the flag is cleared.

For monitoring multiple address ranges, the TPAUSE instruction may be placed within a transactional domain consisting of a set of addresses to monitor and a subsequent TPAUSE instruction. The transactional domain allows for the definition of a set of addresses to wait for, allowing an implementation-dependent optimized operation to begin upon execution of the TPAUSE instruction. In one embodiment, execution of TPAUSE directs the processor to enter an implementation-dependent optimized state while waiting for an event or store operation for an address within the range defined by the read set.

The use of TPAUSE in transactional memory regions may be restricted to C0.1 (lightweight power/performance optimized state). The processor may enter C0.1 even if software sets bit [0] = 0 to indicate its preference over C0.2 (improved power/performance optimized state).

The following may cause the processor to exit the optimized state depending on the implementation: a store to a read set range within the transactional domain, an NMI or SMI, a debug exception, a machine check exception, the BINIT# signal, the INIT# signal, and the RESET# signal. All these events also abort the transaction.

Other implementation-dependent events may cause the processor to leave the implementation-dependent optimized state and proceed to the instruction following TPAUSE, resulting in a transactional region that is not aborted. Additionally, an external interrupt may, in some embodiments, cause the processor to leave the implementation-dependent optimized state regardless of whether maskable interrupts are inhibited. Note that if maskable interrupts are inhibited, execution proceeds to the instruction following TPAUSE, whereas if the interrupt enabled flag is set, the transactional region is aborted.

Figure 66 illustrates an embodiment of a method performed by a processor to process a TPAUSE instruction. For example, hardware details are used here.

At 6601, an instruction is fetched. For example, TPAUSE is fetched. The TPAUSE instruction includes an opcode (and, in some embodiments, a prefix) and an explicit source register operand.

At 6603, the fetched instruction is decoded. For example, the TPAUSE instruction is decoded by a decode circuit as described in detail herein.

At 6605, data values associated with the source operands of the decoded instruction are obtained. Further, in some embodiments, the instruction is scheduled.

At 6607, the decoded instruction is executed by execution circuitry (hardware), as described in detail herein, which places the processor (or core) in an implementation-specific state defined by data for the explicit source register operands.

In some embodiments, at 6609, the instruction is committed or retired.

Figure 67 shows an example of execution using the UMWAIT and UMONITOR instructions.

At 6701, the UMWAIT command is executed to set the range of addresses to monitor.

At 6703, the UMONITOR instruction is executed to put the core executing the instruction into an implementation-dependent state defined by data for the instruction's explicit source register operands for the range of addresses being monitored.

At 6705, the implementation-dependent state is exited in response to one of a store to a monitored address, an NMI, an SMI, a debug exception, a machine check exception, an init signal, or a reset signal.

Figure 68 shows an example of execution using the TPAUSE and UMONITOR instructions.

At 6801, the TPAUSE instruction is executed to set the range of addresses to monitor.

At 6803, the UMONITOR instruction is executed to put the core executing the instruction into an implementation-dependent state defined by data for the instruction's explicit source register operands for the range of addresses being monitored.

At 6805, the implementation-dependent state is exited in response to one of a store to a monitored address, an NMI, an SMI, a debug exception, a machine check exception, an init signal, or a reset signal.

At 6807, the transaction associated with the thread is aborted when the implementation-dependent state is exited.

In some embodiments, an accelerator is coupled to a processor core or other processing element to accelerate certain types of operations, such as graphics operations, machine learning operations, pattern analysis operations, and sparse matrix multiplication operations (as described in more detail below), to name a few. The accelerator may be communicatively coupled to the processor/core via a bus or other interconnect (e.g., a point-to-point interconnect), or may be integrated on the same chip as the processor and communicatively coupled to the core via an internal processor bus/interconnect. Regardless of how the accelerator is connected, the processor core may assign specific processing tasks (e.g., in the form of a sequence of instructions or UOPs) to the accelerator, which contains dedicated circuitry/logic to efficiently process these tasks.

Figure 69 illustrates an exemplary implementation in which an accelerator 6900 is communicatively coupled to multiple cores 6910-6911 through a cache coherent interface 6930. Each of the cores 6910-6911 includes a translation lookaside buffer 6912-6913 for storing virtual-to-physical address translations and one or more caches 6914-6915 (e.g., L1 cache, L2 cache, etc.) for caching data and instructions. A memory management unit 6920 manages access by the cores 6910-6911 to a system memory 6950, which may be dynamic random access memory DRAM. A shared cache 6926, such as an L3 cache, may be shared between the processor cores 6910-6911 and with the accelerator 6900 through the cache coherent interface 6930. In one embodiment, the cores 6910-1011, MMU 6920 and cache coherent interface 6930 are integrated onto a single processor chip.

The illustrated accelerator 6900 includes a data management unit 6905 having a cache 6907 and a scheduler 6906 for scheduling operations for a number of processing elements 6901-6902, N. In the illustrated embodiment, each processing element has its own local memory 6903-6904, N. As described in more detail below, each local memory 6903-6904, N may be implemented as a stacked DRAM.

In one embodiment, the cache coherent interface 6930 provides cache coherent connectivity between the cores 6910-6911 and the accelerator 6900, in effect treating the accelerator as a peer of the cores 6910-6911. For example, the cache coherent interface 6930 may implement a cache coherency protocol to ensure that data accessed/modified by the accelerator 6900 and stored in the accelerator cache 6907 and/or local memory 6903-6904, N is coherent with data stored in the core caches 6910-6911, the shared cache 6926, and the system memory 6950. For example, the cache coherent interface 6930 may participate in the snooping mechanisms used by the cores 6910-6911 and the MMU 6920 to detect the state of cache lines in the shared cache 6926 and the local caches 6914-6915, and may act as a proxy to provide snoop updates in response to accesses and attempted modifications to cache lines by the processing elements 6901-6902, N. Additionally, if a cache line is modified by a processing element 6901-6902, N, the cache coherent interface 6930 may update the status of the cache lines if they are stored in the shared cache 6926 or the local caches 6914-6915.

In one embodiment, the data management unit 1005 includes memory management circuitry that provides the accelerator 6900 with access to the system memory 6950 and the shared cache 6926. Additionally, the data management unit 6905 provides updates to and receives updates from the cache coherent interface 6930 as needed (e.g., to determine state changes for cache lines). In the illustrated embodiment, the data management unit 6905 includes a scheduler 6906 for scheduling instructions/operations to be executed by the processing elements 6901-6902. To perform its scheduling process, the scheduler 6906 evaluates dependencies between instructions/operations to ensure that the instructions/operations are executed in a coherent order (e.g., ensure that a first instruction executes before a second instruction that depends on a result from the first instruction). Instructions/operations that do not have internal dependencies may be executed in parallel by the processing elements 6901-6902.

Figure 70 shows another diagram of the accelerator 6900 and other components described above, including a data management unit 6905 (e.g., implemented using stacked local DRAM in one embodiment), multiple processing elements 6901-N, and high-speed on-chip storage 7000. In one embodiment, the accelerator 6900 is a hardware accelerator architecture, and the processing elements 6901-N include circuitry for performing matrix-by-vector and vector-by-vector operations, including operations on sparse and dense matrices. In particular, the processing elements 6901-N may include hardware support for column- and row-wise matrix processing, and may include microarchitectural support for "scale and update" operations such as those used in machine learning (ML) algorithms.

The described implementation performs matrix/vector operations that are optimized by keeping frequently used, randomly accessed, potentially sparse (e.g., gather/scatter) vector data in fast on-chip storage 7000, by maintaining large, less frequently used matrix data in off-chip memory (e.g., system memory 6950) that is accessed in a streaming manner whenever possible, and by exposing intra/inter matrix block parallelism to scale up.

Examples of processing elements 6901-N process various combinations of sparse matrices, dense matrices, sparse vectors, and dense vectors. As used herein, a "sparse" matrix or vector is one in which most of the components are zero. On the other hand, a "dense" matrix or vector is one in which most of the components are non-zero. The "sparsity" of a matrix/vector may be defined based on the number of zero-valued components divided by the total number of components (e.g., m×n for an m×n matrix). In one example, a matrix/vector is considered to be "sparse" if its sparcity is above a certain threshold.

An exemplary set of operations performed by processing elements 6901-N is shown in the table in FIG. 71. In particular, the operation types include a first multiplication with a sparse matrix 7100, a second multiplication with a dense matrix 7101, a scale and update operation 7102, and a dot product operation 7103. The columns specify a first input operand 7110 and a second input operand 7111 (each of which may include a sparse or dense matrix/vector), an output format 7112 (e.g., dense vector or scalar), a matrix data format (e.g., compressed sparse row, compressed sparse column, row-wise, etc.) 7113, and an operation identifier 7114.

Run-time domain computational patterns available in some current workloads include variants of matrix multiplication on vectors in row-wise and column-wise fashion. These functions on well-known matrices combine compressed sparse row (CSR) and compressed sparse column (CSC) formats. Figure 72a illustrates an example of sparse-matrix multiplication on vector x to produce vector y. Figure 72b shows a CSR representation of matrix A where each value is stored as a (value, row index) pair. For example, (3, 2) for row 0 indicates that a value of 3 is stored in element position 2 of row 0. Figure 72c shows a CSC representation of matrix A using (value, column index) pairs.

Figures 73a, 73b, and 73c show pseudocode for each calculation pattern, which are described in detail below. In particular, Figure 73a shows row-wise sparse matrix-dense vector multiplication (spMdV_csr), Figure 73b shows column-wise sparse matrix-sparse vector multiplication (spMspC_csc), and Figure 73c shows the scale and update operation (scale_update).

A. Row-wise sparse matrix-dense vector multiplication (spMdV_csr)

This is a well-known computational pattern important in many application areas, such as high performance computing. Here, for each row of a matrix A, a dot product of that row against the vector x is performed, and the result is stored in the y vector component pointed to by the row index. This computation is used in machine learning (ML) algorithms that perform analysis over a set of samples (i.e., rows of a matrix). It may also be used in techniques such as "mini-batches". For example, in a probabilistic random variable of a learning algorithm, the ML algorithm may simply perform a dot product of a sparse vector against a dense vector (i.e., iterating the spMdV_csr loop).

A known factor that can affect the performance of this computation is the need to randomly access the sparse x-vector components in the dot product computation. For traditional server systems, if the x-vector is large, this would result in irregular accesses (gathering) to memory or last level caches.

To address this, one embodiment of the processing element splits matrix A into column blocks and the x vector into multiple subsets (each corresponding to a column block of matrix A). The block size can be chosen to allow a subset of the x vector to fit on a chip, so random access to it can be localized on-chip.

B. Column-wise sparse matrix/vector multiplication (spMspV_csc)

This pattern of sparse vector multiplication by a sparse matrix is less well known than spMdV_csr. However, it is important in several ML algorithms. It is used when the algorithm operates on a set of features, represented as columns of matrices in the dataset (hence the need for column-wise matrix access).

In this computation pattern, each column of matrix A is read and multiplied against the corresponding non-zero components of vector x. The result is used to update the partial dot product held in the y vector. Once all columns associated with non-zero x vector components have been processed, the y vector will contain the final dot product.

While accesses to matrix A (i.e., streams on the columns of A) are normal, accesses to the y vector that updates the partial dot products are irregular. The y component to access depends on the row index of the A vector component being processed. To address this, matrix A can be partitioned into row blocks. As a result, vector y can be partitioned into subsets corresponding to these blocks. In this manner, when processing a row block of the matrix, it is only necessary to irregularly access (gather/scatter) that y vector subset. By appropriately selecting the block size, the y vector subset can be kept on-chip.

C. Scale and update (scale_update)

This pattern is typically used by ML algorithms, which apply a scaling factor to each sample in a matrix, reducing them to a set of weights, each corresponding to a feature (i.e., a column in A), where the x vector contains the scaling factors. For each row of matrix A (in CSR format), the scaling factor for that row is read from the x vector and then applied to each component of A in that row. The result is used to update the components of the y vector. Once all rows have been processed, the y vector will contain the reduced weights.

Similar to the previous computation pattern, irregular accesses to the y vector can impact performance when y is large. Splitting the matrix A into column blocks and the y vector into subsets corresponding to these blocks can help localize irregular accesses within each y subset.

In one embodiment, the system includes a hardware accelerator that can efficiently execute the computational patterns described above. The accelerator is a hardware IP block that can be integrated with a general-purpose processor. In one embodiment, the accelerator 6900 executes the computational patterns by independently accessing memory 6950 through an interconnect shared with the processor. It supports any arbitrarily large matrix data set residing in off-chip memory.

Figure 74 shows the process flow for one embodiment of the data management unit 6905 and processing elements 6901-6902. In this embodiment, the data management unit 6905 includes a processing element scheduler 7401, a read buffer 7402, a write buffer 7403, and a reduction unit 7404. Each PE 6901-6902 includes an input buffer 7405-7406, multipliers 7407-7408, adders 7409-7410, local RAMs 7421-7422, sum registers 7411-7412, and an output buffer 7413-7414.

The accelerator supports the matrix blocking scheme described above (i.e., row and column blocking) to support any arbitrarily large matrix data. The accelerator is designed to process blocks of matrix data. Each block is further divided into sub-blocks that are processed in parallel by the PEs 6901-6902.

In operation, the Data Management Unit 6905 reads rows or columns of a matrix from the memory subsystem into its read buffer 7402, which are then dynamically distributed by the PE Scheduler 7401 across the PEs 6901-6902 for processing. It also writes the results from its write buffer 7403 to memory.

Each PE 6901-6902 is responsible for processing a subblock of the matrix. The PE contains on-chip RAM 7421-7422 to store vectors that need to be randomly accessed (i.e., a subset of the x or y vectors as described above). It also contains a floating-point multiply-accumulate (FMA) unit that contains multipliers 7407-7408 and adders 7409-7410, unpack logic in input buffers 7405-7406 that extracts matrix elements from the input data, and sum registers 7411-7412 that hold the accumulated FMA results.

One embodiment of the accelerator (1) places irregularly accessed (gathered/scattered) data in on-chip PE RAMs 7421-7422, (2) utilizes a hardware PE scheduler 7401 to ensure that the PEs are fully utilized, and (3) achieves maximum efficiency because, unlike with general-purpose processors, the accelerator consists only of hardware resources essential for sparse matrix operations. Overall, the accelerator efficiently translates the available memory bandwidth provided to it into performance.

Performance scaling can be done by using many PEs in an accelerator block to process multiple matrix sub-blocks in parallel, and/or by using more accelerator blocks (each with a set of PEs) to process multiple matrix blocks in parallel. Combinations of these options are considered below. The number of PEs and/or accelerator blocks should be adjusted to match the memory bandwidth.

In one embodiment of the accelerator 6900, it may be programmed through a software library. Such a library prepares the matrix data in memory, sets control registers in the accelerator 6900 with information about the computation (e.g., computation type, memory pointer to the matrix data), and starts the accelerator. The accelerator then independently accesses the matrix data in memory to perform the computation and writes the results back to memory for the consuming software.

The accelerator processes various computation patterns by configuring its PEs for the appropriate datapath configuration, as illustrated in Figures 75a-75b. In particular, Figure 75a highlights (with dotted lines) the paths for the spMspV_csc and scale_update operations, while Figure 75b shows the paths for the spMdV_csr operation. The operation of the accelerator to perform each computation pattern is described in detail below.

For spMspV_csc, the DMU 6905 loads the initial y-vector subset into the PE's RAM 7421. It then reads the x-vector components from memory. For each x-component, the DMU 6905 streams the corresponding matrix column components from memory and provides them to the PE 6901. Each matrix component includes a value (A.val) and an index (A.idx) that points to the y-component to read from the PE's RAM 7421. The DMU 6905 also provides the x-vector component (x.val) which is multiplied by the functional multiply-accumulate (FMA) unit against A.val. The result is used to update the y-component in the PE's RAM pointed to by A.idx. Note that the accelerator supports column-like multiplication on dense x-vectors (spMdV_csc) by processing all matrix columns instead of only a subset (since x is dense), even if not used by the present workload.

The scale_update operation is similar to spMspV_csc, except that the DMU 6905 reads the rows of matrix A represented in CSR format instead of CSC format. For spMdV_csr, a subset of the x vectors is loaded into the PE's RAM 7421. The DMU 6905 streams the row components of the matrix (i.e., {A.val, A.idx} pairs) from memory. A.idx is used to read the appropriate x vector component from RAM 7421 and multiplied against A.val by the FMA. The results are accumulated in the sum register 7412. The sum register is written to the output buffer each time the PE sees an end-of-row marker provided by the DMU 6905. In this way, each PE generates a sum for its sub-block of rows. To generate the final sum for the row, the subblock sums generated by all the PEs are added together by the reduction unit 7404 in the DMU (see FIG. 74). The final sum is written to output buffers 7413-7414, which the DMU 6905 then writes to memory.

Graph data processing

In one embodiment, the accelerator architecture described herein is configured to process graph data. Graph analysis relies on graph algorithms to extract knowledge about relationships between data represented as graphs. The proliferation of graph data (from sources such as social media) has led to strong demand and widespread use of graph analysis. As a result, it is critical to be able to perform graph analysis as efficiently as possible.

To address this need, one embodiment automatically maps user-defined graph algorithms to hardware accelerator architecture "templates" that are customized for a given input graph algorithm. The accelerator may have the architecture described above and may be implemented as an FPGA/ASIC that can perform with the highest efficiency. In summary, one embodiment includes the following components:

(1) A hardware accelerator architecture template based on the Generic Sparse Matrix Vector Multiplication (GSPMV) accelerator. It supports arbitrary graph algorithms since it has been shown that graph algorithms can be formulated as matrix operations.

(2) An automated approach to map and align widely used "vertex-centric" graph programming abstractions to architectural templates.

There are existing sparse matrix multiplication hardware accelerators, but they do not support the customizability that would allow for mapping of graph algorithms.

In one embodiment of the design framework, it works as follows:

(1) A user specifies a graph algorithm as a "vertex program" that follows a vertex-centric graph programming abstraction. This abstraction is chosen as an example here due to its popularity. Vertex programs do not expose hardware details, so even users without hardware expertise (e.g., data scientists) can create them.

(2) Along with the graph algorithm in (1), one embodiment of the framework accepts the following inputs:

Parameters for the target hardware accelerator to be generated (e.g., maximum amount of on-chip RAM). These parameters may be provided by the user, or may be taken from an existing library of known parameters if targeting an existing system (e.g., a specific FPGA board).

b. Design optimization objectives (e.g., maximum performance, minimum area)

c. Characteristics of the target graph data (e.g., type of graph) or the graph data itself, which is optional, are used to aid in auto-tuning.

(3) Given the above inputs, one embodiment of the framework performs auto-tuning to determine a set of customizations to apply to the hardware template to optimize the input graph algorithm, maps these parameters onto the architecture template, generates an accelerator instance in synthesizable RTL, and performs functionality and performance validation of the generated RTL against a software model of functionality and performance derived from the input graph algorithm specification.

In one embodiment, the accelerator architecture described above is extended to support vertex program execution by (1) making it a customizable hardware template and (2) supporting the functionality required by the vertex program. Based on this template, a design framework is described to map a user-supplied vertex program to the hardware template to generate an optimized synthesizable RTL (e.g., Verilog) implementation instance for the vertex program. The framework also performs automatic verification and tuning to ensure that the generated RTL is correct and optimized. There are multiple use cases for this framework. For example, the generated synthesizable RTL can be placed on an FPGA platform (e.g., Xeon-FPGA) to efficiently execute a given vertex program. Or, it can be further improved to generate an ASIC implementation.

Graphs can be represented as adjacency matrices, and graph processing can be formulated as sparse matrix operations. Figures 76a-b show an example of representing a graph as an adjacency matrix. Each non-zero in the matrix represents an edge between two nodes in the graph. For example, a 1 in row 0, column 2 represents an edge from nodes A to C.

One of the most popular models for expressing computations on graph data is the vertex programming model. In one embodiment, we support a variant of the vertex programming model from the GraphMat software framework that formulates vertex programs as generic sparse matrix vector multiplication (GSPMV). As shown in Figure 76c, a vertex program consists of multiple types of data (e-data/v-data) associated with edges/vertices in the graph (as shown at the top of the program code), messages (m-data) and temporary data (t-data) sent through the vertices in the graph, and stateless user-defined computation functions with predefined APIs to read and update the graph data (as shown at the bottom of the program code).

Figure 76d shows example program code for executing the vertex program. Edge data is represented as an adjacency matrix A (as shown in Figure 76b), vertex data as vector y, and messages as sparse vector x. Figure 76e shows a formulation of GSPMV, where the multiply() and add() operations in SPMV are generalized with user-defined types PROCESS_MSG() and REDUCE().

One view here is that the transformation of GSPMV required to execute a vertex program is to perform a column-wise multiplication of a sparse matrix A (i.e., the adjacency matrix) on a sparse vector x (i.e., the message) to produce an output vector y (i.e., the vertex data). This operation is referred to as col_spMspV (described above with respect to the accelerator above).

One embodiment of the framework is shown in FIG. 77, including a design framework template mapping component 7711, a validation component 7712, and an auto-tuning component 7713. The ingredients are user-specified vertex programs 7701, design optimization goals 7703 (e.g., maximum performance, minimum area), and target hardware design constraints 7702 (e.g., maximum amount of on-chip RAM, memory interface width). As optional ingredients to aid in auto-tuning, the framework also allows graph data characteristics 7704 (e.g., type=natural graph) or sampling graph data.

Given these materials, the template mapping component 7711 of the framework maps the input vector program to a hardware accelerator architecture template and generates an RTL implementation 7705 of an accelerator instance optimized to execute the vertex program 7701. The auto-tuning component 7713 performs auto-tuning 7713 to optimize the generated RTL for a given design goal while satisfying hardware design constraints. Furthermore, the verification component 7712 automatically verifies the generated RTL against functional and performance models derived from the materials. A verification testbench 7706 and a tuning report 7707 are generated along with the RTL.

Generic sparse matrix vector multiplication (GSPMV) hardware architecture template

One embodiment of an architecture template for GSPMV is shown in FIG. 77, which is based on the accelerator architecture described above (see, e.g., FIG. 74 and related text). Many of the components shown in FIG. 77 are customizable. In one embodiment, the architecture to support vertex program execution is extended as follows:

As shown in FIG. 78, customizable logic blocks are provided within each PE to support PROCESS_MSG() 1910, REDUCE() 7811, APPLY 7812, and SEND_MSG() 7813 required by the vertex program. Additionally, in one embodiment, a customizable on-chip storage structure and pack/unpack logic 7805 are provided to support user-defined graph data (i.e., v-data, e-data, m-data, t-data). The illustrated data management unit 6905 includes a PE scheduler 7401 (for scheduling PEs as described above), an auxiliary buffer 7801 (for storing active column, x-data), a read buffer 7402, a memory controller 7803 for controlling access to system memory, and a write buffer 7403. Additionally, in the embodiment shown in FIG. 78, old and new v-data and t-data are stored in the local PE memory 7421. The various control state machines may be modified to execute vertex programs and support invariants to the functionality defined by the algorithms in Figures 76d and 76e.

The processing of each accelerator tile is summarized in Figure 79. At 7901, the y vector (v data) is loaded into the PE RAM 7421. At 7902, the x vector and column pointers are loaded into the auxiliary buffer 7801. At 7903, for each x vector component, the columns are streamed (e data) and the PE executes PROC_MSG() 7810 and REDUCE() 7811. At 7904, the PE executes APPLY() 7812. At 7905, the PE executes SEND_MSG() 7813 to generate a message and the Data Management Unit 6905 writes these to memory as x vectors. At 7906, the Data Management Unit 6905 writes the updated y vector (v data) stored in the PE RAM 7421 back to memory. The above techniques are adapted to the vertex program execution algorithm shown in Figures 76d and 76e. To increase performance, the architecture allows the design to increase the number of PEs in a tile and/or the number of tiles. In this manner, the architecture exploits multiple levels of graph parallelism (i.e., across subgraphs (of the adjacency matrix, or across blocks in each subgraph)). The table in Figure 80a summarizes the customizable parameters for one embodiment of the template. It is also possible to assign asymmetric parameters across tiles for optimization (e.g., one tile with more PEs than another tile).

Automatic mapping, validation and tuning

Tuning. Based on the input, one embodiment of the framework performs auto-tuning to determine the best design parameters to use to customize the hardware architecture template to optimize it for the input vector program and (optionally) the graph data. There are a number of tuning considerations, which are summarized in the table in Figure 80b. As shown, these include data locality, graph data size, graph computation function, graph data structure, graph data access attributes, graph data type, and graph data pattern.

Template Mapping. In this phase, the framework takes the template parameters determined by the tuning phase and generates the accelerator instance by "filling" in the customizable parts of the template. User-defined compute functions (e.g., Fig. 76c) may be mapped from the input specification to the appropriate PE compute blocks using existing high-level synthesis (HLS) tools. Storage structures (e.g., RAM, buffers, cache) and memory interfaces are instantiated using their corresponding design parameters. Pack/unpack logic may be automatically generated from the data type specification (e.g., Fig. 76a). Parts of the control finite state machine (FSM) are also generated based on the provided design parameters (e.g., PE scheduling scheme).

Verification. In one embodiment, the accelerator architecture instance (synthesizable RTL) generated by template mapping is then automatically verified. To do this, one embodiment of the framework derives a functional model of the vertex program that is used as a "golden" reference. A testbench is generated to compare the execution of this golden reference against a simulation of an RTL implementation of the architecture instance. The framework also performs performance verification by comparing the RTL simulation against an analytical performance model and a cycle-accurate software simulator. It reports a runtime breakdown and identifies design bottlenecks that impact performance.

Computations on sparse data sets -- vectors or matrices where most of the values are zero -- are critical to an ever-increasing number of commercially important applications, yet typically achieve only a few percent of peak performance when run on today's CPUs. In scientific computing, sparse matrix computations have been the key kernel of linear solvers for decades. In recent years, the explosive growth of machine learning and graph analysis has moved sparse computations into the mainstream. Sparse matrix computations are at the heart of many machine learning applications and form the core of many graph algorithms.

Sparse matrix computations tend to be memory bandwidth-limited rather than computationally-limited, which makes it difficult to improve their performance with CPU changes. They perform few operations per matrix data element and often iterate over the entire matrix before reusing any data, thus not benefiting from caches. Furthermore, many sparse matrix algorithms contain a significant number of data-dependent gathers and scatters, e.g., the result[row]+=matrix[row][i].value*vector[matrix[row][i].index] operations that occur in sparse matrix-vector multiplication, which are difficult to predict and reduce the effectiveness of prefetchers.

To achieve better sparse matrix performance than conventional microprocessors, the system must provide significantly higher memory bandwidth than current CPUs and a very energy-efficient computing architecture. Increasing memory bandwidth can improve performance, but the high energy/bit cost of DRAM access limits the amount of power available to process that bandwidth. Without an energy-efficient computing architecture, a system may be placed in a situation where it is unable to process data from a high-bandwidth memory system without exceeding its power budget.

In one embodiment, we have an accelerator for sparse matrix computations that uses stacked DRAM to provide the bandwidth that sparse matrix algorithms need combined with a custom computational architecture to process that bandwidth in an energy efficient manner.

Sparse - matrix overview

Many applications create data sets where the majority of the values are zero. Finite element methods model objects as a mesh of points where the state of each point is a function of the states of points close to it in the mesh. Mathematically, this becomes a system of equations represented as a matrix where each row represents the state of one point and the row value is zero for all of the points that do not directly affect the state of the point it represents. A graph can be represented as an adjacency matrix, where each entry {i,j} in the matrix gives the weight for the edge between vertices i and j in the graph. Most of the entries in the adjacency matrix are zero because many vertices connect only a small fraction of the other vertices in the graph. In machine learning, models are typically trained with a dataset of many examples, each of which contains a set of features (opinions about the state of the system or object) and the desired output for a model of that set of features. It is common for many of the examples to include only a small subset of the possible features, for example if the features represent the various words that may be present in a document, again creating a dataset where most of the values are zero.

A data set in which most of the values are zero is described as "sparse", meaning that fewer than 1% of the elements have nonzero values; it is not uncommon for sparse data sets to be extremely sparse. These data sets are often represented as matrices, using data structures that specify only the values of the nonzero elements in the matrix. While this increases the amount of space required to represent each nonzero element, since both the element's location and its value need to be specified, the overall space (memory) savings can be significant if the matrix is sufficiently sparse. For example, one of the simplest representations of a sparse matrix is the coordinated list (COO) representation, where each nonzero is specified by a tuple of {row index, column index, value}. While this triples the amount of storage required per nonzero value, if only 1% of the elements in the matrix have nonzero values, the COO representation only takes up 3% of the space that a dense representation (one that represents the value of each element in the matrix) would take.

Figure 81 shows one of the most common sparse matrix formats, the Compressed Row Store (CRS, sometimes abbreviated CSR) format. In the CRS format, a matrix 8100 is represented by three arrays: a value array 8101 that contains the values of the non-zero elements, an index array 8102 that specifies the location of each non-zero element in that row of the matrix, and a row start array 8103 that specifies where each row of the matrix begins in the list of indices and values. Thus, the first non-zero element of the second row of the example matrix can be found at position 2 in the index and value array and is represented by the tuple {0,7}, indicating that the element is at position 0 in the row and has value 7. Other commonly used sparse matrix formats include Compressed Sparse Column (CSC), which is dual column-major to CRS, and ELLPACK, which represents each row of a matrix as a fixed-width list of non-zero values and their indices, padding with explicit zeros if the row has fewer non-zero elements than the longest row in the matrix.

Although sparse matrix computations have the same structure as their dense counterparts, the properties of sparse data tend to make them much more bandwidth intensive than their dense counterparts. For example, both sparse and dense variants of matrix-matrix multiplication know that C = A · B by computing Ci,j = Ai, · B,j for all i,j. In dense matrix-matrix computations, each element of A participates in N multiply-add operations (assuming an N × N matrix), as does each element of B, leading to significant data reuse. As long as matrix-matrix multiplication is blocked for cache locality, this reuse has a low byte/op rate and causes computationally limited computation. However, in the sparse variant, each element of A only participates in as many multiply-add operations as there are nonzero values in the corresponding row of B, while each element of B only participates in as many multiply-add operations as there are nonzero elements in the corresponding column of A. As matrix sparsity improves, as does the byte/op rate, memory bandwidth limits the performance of many sparse matrix-matrix computations, despite the fact that dense matrix-matrix multiplication is a benchmark-bound computation.

Four operations make up the bulk of sparse matrix computations found in today's applications, namely sparse matrix-sparse vector multiplication (SpMV), sparse matrix-sparse vector multiplication, sparse matrix-sparse matrix multiplication, and relaxation/smoothing operations, e.g., Gauss-Seidel smoothers used in implementing high-performance conjugate gradient criteria. These operations share two properties that make sparse matrix accelerators practical. First, they are dominated by vector dot products, allowing for the implementation of simple hardware that can implement all four key computations. For example, matrix-vector multiplication is performed by taking the dot product of a vector with each row in a matrix, while matrix-matrix multiplication takes the dot product of each column of one matrix with each row of the other matrix. Second, applications typically perform multiple computations on the same matrix, e.g., thousands of multiplications of the same matrix with different vectors, which support vector machine algorithms perform to train models. This repeated use of the same matrices makes it practical to transfer matrices to/from the accelerator during program execution and/or reformat matrices in a manner that simplifies the hardware's tasks, since the cost of data transfer/conversion can be amortized over many operations on each matrix.

Sparse matrix computations typically achieve only a few percent of the peak performance of the systems on which they are executed. To clarify why this occurs, Figure 82 shows steps 8201-8204 for implementing sparse matrix-dense vector multiplication using the CRS data format. First, in 8201, a data structure representing a row of the matrix is read from memory, typically in a set of sequential reads that are easy to predict and prefetch. Second, in 8202, the index of a non-zero element in the row of the matrix is used to gather the corresponding element of the vector, which requires a large number of data-dependent and hard-to-predict memory accesses (gather operations). Furthermore, these memory accesses often only touch one or two words in each referenced cache line, resulting in a significant amount of wasted bandwidth if the vector does not fit in the cache.

Third, in 8203, the processor computes the dot product of the non-zero components of the row of the matrix and the corresponding components of the vector. Finally, in 8204, the result of the dot product is written to a result vector, which is also successively accessed and the program proceeds to the next row of the matrix. Note that this is a conceptual/algorithmic view of the computation, and the exact sequence of operations the program performs depends on the processor's ISA and vector width.

This example illustrates a number of important properties of sparse matrix computations. Assuming 32-bit data types and that neither matrices nor vectors fit in the cache, computing the first component of the output row requires reading 36 bytes from DRAM, but only five computation instructions (three multiplications and two additions) for a byte/op rate of 7.2:1.

However, memory bandwidth is not the only challenge for high performance sparse matrix computation. As Figure 82 shows, accesses to vectors in SpMV are data dependent and hard to predict, exposing vector access latencies to applications. If vectors do not fit in the cache, SpMV performance becomes sensitive to DRAM latency as well as bandwidth unless the processor provides enough parallelism to saturate the DRAM bandwidth, even if many threads are stalled waiting for data.

Therefore, an architecture for sparse matrix computation must enable several things: it must provide high memory bandwidth to meet the bytes/op need for sparse computations; and it must support high bandwidth collection from large vectors that may not fit in the cache. Finally, while performing enough arithmetic operations/second to keep up with DRAM bandwidth is not a challenge in itself, the architecture must perform those operations and all of the memory accesses they require in an energy-efficient manner to stay within the system's power budget.

In one embodiment, we have an accelerator designed to provide the three functions required for high sparse-matrix performance: high memory bandwidth, high bandwidth collection from large vectors, and energy-efficient computation. As shown in FIG. 83, one embodiment of the accelerator includes an accelerator logic die 8305 and one or more stacks of DRAM dies 8301-8304. Stacked DRAM, described in more detail below, provides high memory bandwidth with low energy/bit. For example, stacked DRAM is expected to achieve 256-512 GB/sec at 2.5 pJ/bit, while LPDDR4 DIMMs are expected to achieve only 68 GB/sec, with an energy cost of 12 pJ/bit.

The accelerator logic chip 8305 at the bottom of the accelerator stack is customized for the needs of sparse matrix computations and is able to consume the bandwidth provided by the DRAM stacks 8301-8304 while consuming only 2-4 watts of power, with energy consumption proportional to the stack bandwidth. For the remainder of this application, a stack bandwidth of 273 GB/sec (the expected bandwidth of the WIO3 stack) is assumed. Designs based on higher bandwidth stacks would incorporate more parallelism to consume memory bandwidth.

Figure 84a shows one embodiment of an accelerator logic chip 8305 oriented from a top-down perspective through the stack of DRAM dies 8301-8304. The stacked DRAM channel block 8405 faces toward the center of the diagram, which represents the silicon vias connecting the logic chip 8305 to the DRAMs 8301-8304, while the memory controller block 7410 contains the logic to generate the control signals for the DRAM channels. While eight DRAM channels 8405 are shown in the diagram, the actual number of channels implemented on the accelerator chip will vary depending on the stacked DRAM used. Most stacked DRAM technologies under development offer either four or eight channels.

The Dot Product Engine (DPE) 8420 is the computational element of the architecture. In the particular embodiment shown in Figures 84a-b, each set of eight DPEs is associated with a vector cache 8415. Figure 85 provides a high-level overview of a DPE, which includes two buffers 8505-8506, two 64-bit multiply-add ALUs 8510, and control logic 8500. During computation, the chip control unit 8500 streams chunks of data to be processed into the buffer memories 8505-8506. Once each buffer is full, the DPE's control logic sequences through the buffers, computes the dot products of the vectors they contain, and writes the results into the DPE's result latches 8512, which are daisy-chained with the result latches of other DPEs, and writes the results of the computation back into the stacked DRAMs 8301-8304.

In one embodiment, the accelerator logic chip operates at approximately 1 GHz and 0.65V to minimize power consumption (although the specific operating frequency and voltage may be modified for different applications). Analysis based on 14nm design studies indicates that 32-64KB buffers meet this frequency spec at that voltage, but strong ECC may be required to prevent weak errors. The multiply-accumulate unit may be operated at half the base clock rate to meet timing with a 0.65V supply voltage and shallow pipeline. Two ALUs are used to provide a throughput of one double precision multiply-accumulate operation/cycle per DPE.

At 273 GB/sec and a clock rate of 1.066 MHz, the DRAM stacks 8301-8304 provide 256 bytes of data per clock cycle to the logic chip. Assuming that the array indexes and values are at least 32-bit quantities, this translates to 32 sparse matrix elements per cycle (4 bytes for index + 4 bytes for value = 8 bytes/element), requiring the chip to perform 32 multiply-accumulate operations per cycle to keep up. (This is for matrix-vector multiplications and assumes a high hit rate in the vector cache such that 100% of the stack DRAM bandwidth is used to fetch matrices). The 64 DPEs shown in Figures 84a and 84b provide the required computational throughput of 2-4x, allowing the chip to process data at the peak stack DRAM bandwidth even if the ALU 8510 is not used 100% of the time.

In one embodiment, the vector cache 8415 caches the components of vectors in matrix-vector multiplication. This significantly improves the efficiency of the matrix-blocking scheme described below. In one embodiment, each vector cache block contains 32-64 KB of cache, for a total capacity of 256-512 KB in an eight channel architecture.

The chip control unit 8401 manages the flow of computations and handles communication with other stacks in the accelerator and other sockets in the system. To reduce complexity and power consumption, the dot product engine never requests data from memory. Instead, the chip control unit 8401 manages the memory system and initiates transfers that push the appropriate blocks of data to each of the DPEs.

In one embodiment, the stacks in a multi-stack accelerator communicate with each other via a network of KTI links 8430 implemented with adjacent connections 8431 as shown in the figure. The chip also provides three additional KTI links that are used to communicate with other sockets in a multi-socket system. In a multi-stack accelerator, only one of the off-package KTI links 8430 of a stack is activated. KTI transactions targeting memory on other stacks are forwarded to the appropriate stack via the on-package KTI network.

Techniques and hardware for implementing sparse matrix-dense vector and sparse matrix-sparse vector multiplication for one embodiment of the accelerator are described herein. This can also be extended to support matrix-matrix multiplication, relaxation operations, and other functions to create accelerators that support sparse matrix operations.

While sparse-sparse and sparse-dense matrix-vector multiplication perform the same basic algorithm (taking the dot product of each row in the matrix and the vector), there are significant differences in how this algorithm is implemented when the vector is sparse compared to when it is dense, which are summarized in the table below.

In a sparse matrix-dense vector multiplication, the size of the vector is fixed and equal to the number of columns in the matrix. Since many matrices resulting from scientific computing have an average of around 10 nonzero elements per row, it is not uncommon for the vector in a sparse matrix-dense vector multiplication to occupy 5-10% of the space of the matrix itself. On the other hand, sparse vectors are often much shorter and contain a similar number of nonzero values to the matrix rows, making them much easier to cache in on-chip memory.

In sparse matrix-dense vector multiplication, the position of each component in the vector is determined by its index, which collects the vector components that correspond to non-zero values in the domain of the matrix, making it feasible to pre-compute the set of vector components that need to be collected for any dense vector to be multiplied by the matrix. However, the position of each component in the sparse vector is unpredictable and depends on the distribution of the non-zero components in the vector. This requires examining the non-zero components of the sparse vector and matrix to determine which non-zeros in the matrix correspond to the non-zero values in the vector.

Because the number of instructions/operations required to compute a sparse matrix-sparse vector dot product is unpredictable and depends on the structure of the matrix and vector, it helps to compare the non-zero components in the matrix and vector with their indices. For example, consider taking the dot product of a row of a matrix with a single non-zero component and a vector with many non-zero components. If the non-zero values in the row have a lower index than any of the non-zero values in the vector, the dot product requires only one index comparison. If the non-zero values in the row have a higher index than any of the non-zero values in the vector, computing the dot product requires comparing the index of the non-zero values in the row with each index in the vector. This assumes a linear search through the vector, which is common practice. Other searches, such as a binary search, would be faster in the worst case. However, they would add significant overhead in the common case where non-zero values in the row and vector overlap. On the other hand, since the number of operations required to perform a sparse matrix-dense vector multiplication is fixed and determined by the number of non-zero values in the matrix, it makes the time required for the calculation easier to predict.

Due to these differences, one embodiment of the accelerator uses the same high-level algorithms to perform sparse matrix-dense vector and sparse matrix-sparse vector multiplications, with differences in how the vectors are distributed across the dot product engines and how the dot products are computed. Because the accelerator is targeted at large sparse matrix computations, it is assumed that either the matrix or the vectors cannot fit into on-chip memory. Instead, one embodiment uses a blocking scheme outlined in Figure 86.

In particular, in this embodiment, the accelerator is sized to fit into on-chip memory, splits the matrix into fixed-sized blocks of data 8601-8602, and multiplies rows in a block by a vector to generate a chunk of an output vector before proceeding to the next block. This approach poses two challenges. First, the number of nonzeros in each row of a sparse matrix varies widely between datasets, from as low as 1 to as high as 46,000 in the datasets studied. This makes it impractical to assign one or a fixed number of rows to each dot-product engine. Thus, in one embodiment, a fixed-sized chunk of matrix data is assigned to each dot-product engine for processing when a chunk contains multiple matrix rows and when a single row is split across multiple chunks.

The second challenge is that fetching the entire vector from stack DRAM for each block of the matrix can waste a large amount of bandwidth (i.e., fetching vector components that have no corresponding nonzeros in the block). This is particularly problematic for sparse matrix-dense vector multiplication, where the vectors can occupy a significant portion of the sparse matrix. To address this, one embodiment constructs a fetch list 8611-8612 for each block 8601-8602 in the matrix, enumerates the set of components of the vector 8610 that correspond to the nonzero values in the block, and only fetches those components when processing the block. While the fetch list must also be fetched from stack DRAM, it has been determined that the fetch list for most blocks occupies a small portion of the block. Techniques such as run-length encoding may be used to reduce the size of the fetch lists.

So matrix-vector multiplication on an accelerator involves the following sequence of operations:

1. Fetch a block of matrix data from the DRAM stack and distribute it across the dot product engines.

2. Generate a fetch list based on the non-zero elements in the matrix data.

3. Fetch each vector component in the fetch list from the stack DRAM and distribute it to the dot product engines.

4. Calculate the dot product of the rows in the block with the vector and write the results to stack DRAM.

5. In parallel with the computation, fetch the next block of matrix data and repeat until the entire matrix has been processed.

If the accelerator contains multiple stacks, "partitions" of the matrix may be statically assigned to different stacks, and then the block algorithm may be executed in parallel for each partition. This block and broadcast scheme has the advantage that all of the memory references originate from a central controller, greatly simplifying the design of the on-chip network, as the network does not need to transport unpredictable requests and responses between the dot product engines and the memory controller. It also saves energy by issuing only one memory request per vector component that a given block requires, as opposed to having individual dot product engines issue memory requests for the vector components they need to perform these parts of the calculation. Finally, fetching vector components from a structured list of indexes makes it easier to schedule memory request fetch requests in a manner that maximizes page hits in the stacked DRAM, and therefore bandwidth utilization.

One challenge in performing sparse matrix-dense vector multiplication in the accelerator implementation described herein is matching vector components streamed from memory to the matrix component indices in each dot-product engine's buffer. In one embodiment, 256 bytes (32-64 components) of the vector arrive at the dot-product engine every cycle, and a fixed-size block of matrix data is fetched into each dot-product engine's matrix buffer, so that each vector component can correspond to any of the nonzeros in the dot-product engine's matrix buffer.

Performing that many comparisons every cycle would be very expensive in area and power. Instead, in one embodiment, using the format shown in FIG. 87, we take advantage of the fact that many sparse matrix applications repeatedly multiply the same matrix by either the same or different vectors, and pre-compute the fetch list elements that each dot product engine needs to process its chunk of the matrix. In the baseline CRS format, a matrix is described by an array of indexes 8702 that define the location of each non-zero value in its row, and the array contains each non-zero value 8703 and an array 8701 that indicates where each row begins in the index and value array. To do so, in one embodiment, we add an array of block descriptors 8705 that identify which bursts of vector data each dot product engine needs to capture to perform its part of the overall computation.

As shown in Figure 87, each block descriptor consists of eight 16-bit values and a list of burst descriptors. The first 16-bit value tells the hardware how many burst descriptors are in the block descriptor, while the remaining seven identify the starting point in the list of burst descriptors for all of the stacked DRAM data channels except the first. The number of these values changes depending on the number of data channels the stacked DRAM provides. Each burst descriptor contains a 24-bit burst count that tells the hardware which burst of data it needs to pay attention to, and a "required word" bit vector that identifies the word in the burst that contains the value the dot processing engine needs.

Another data structure included in one embodiment is an array of matrix buffer indexes (MBIs) 8704, one MBI for each non-zero in the matrix. Each MBI gives the location where the dense vector component corresponding to the non-zero is stored in the vector value buffer of the associated dot product engine (see, e.g., FIG. 89). When performing a sparse matrix-dense vector multiplication, the matrix buffer index, rather than the original matrix index, is loaded into the matrix index buffer 8704 of the dot product engine and takes the place of the address used to look up the corresponding vector value when computing the dot product.

Figure 88 shows how this works for a two row matrix that fits into a single dot product engine's buffer in a system with only one stacked DRAM data channel and four word data bursts. The original CRS representation is shown on the left of the figure, including row start value 8801, matrix index 8802, and matrix value 8803. Row 2 has non-zero components in columns {2,5,6} and {2,4,5}, so vector components 2, 4, 5, and 6 are needed to compute the dot product. The block descriptor reflects this, showing that word 2 of the first four word burst (vector component 2) and words 0, 1, and 2 of the second four word burst (vector components 4-6) are needed. Vector component 2 goes into position 0 in the vector value buffer, as that is the first word of the vector the dot product engine needs. Vector component 4 goes into position 1, etc.

The matrix buffer index array data 8804 holds the locations in the vector value buffer where the hardware found values corresponding to non-zeros in the matrix. Because the first entry in the matrix index array has a value of "2", the first entry in the matrix buffer index array gets a value of "0", which corresponds to the location where component 2 of the vector is stored in the vector value buffer. Similarly, whenever a "4" appears in the matrix index array, a "1" appears in the matrix buffer index, each "5" in the matrix index array has a corresponding "2" in the matrix buffer index, and each "6" in the matrix index array corresponds to a "3" in the matrix buffer index.

One embodiment of the present invention performs the precomputations necessary to support high speed collection from dense vectors when the matrices are loaded onto the accelerator, taking advantage of the fact that the aggregate bandwidth of a multi-stack accelerator is much larger than the bandwidth of the KTI link used to transfer data from the CPU to the accelerator. This precomputed information improves the amount of memory required to hold the matrix by up to 75%, depending on how many copies of the same matrix index occur within the chunk of the matrix that is mapped onto the dot product engine. However, because a 16-bit matrix buffer index array is fetched instead of the matrix index array when matrix-vector multiplication is performed, the amount of data fetched from the stack DRAM is often less than the original CRS representation, especially for matrices that use 64-bit indexes.

Figure 89 shows one embodiment of hardware within a dot product engine using this format. To perform matrix-vector multiplication, the chunks of matrices that make up a block are copied to a matrix index buffer 8903 and a matrix value buffer 8905 (copying the matrix buffer index instead of the original matrix index) and the associated block descriptor is copied to a block descriptor buffer 8902. The fetch list is then used to load the required elements from the dense vector and broadcast them to the dot product engines. Each dot product engine counts the number of bursts of vector data that pass through each data channel. If the count for a given data channel matches the value specified in the burst descriptor, the match logic 8920 captures the specified words and stores them in its vector value buffer 8904.

Figure 90 shows the contents of the Match Logic 8920 unit which performs this capture. A latch 9005 captures the value on the wire of the data channel if the counter matches the value in the burst descriptor. A shifter 9006 extracts the required words 9002 from the burst 9001 and transfers them to the appropriate locations in the line buffer 9007 whose size matches the rows in the vector value buffer. When the burst count 9001 is equal to the internal counter 9004, a load signal is generated. When the line buffer becomes full, it is loaded (through mux 9008) into the vector value buffer 8904. This word assembly from multiple bursts into lines reduces the number of writes/cycles that the vector value buffer needs to support and reduces its size.

Once all of the necessary components of the vector are captured in the vector value buffer, the dot product engine uses the ALU 8910 to calculate the necessary dot product. The control logic 8901 goes through the matrix index buffer 8903 and the matrix value buffer 8904 in order, one component per cycle. The output of the matrix index buffer 8903 is used as the read address for the vector value buffer 8904 in the next cycle, while the output of the matrix value buffer 8904 is latched to arrive at the ALU 8910 at the same time as the corresponding value from the vector value buffer 8904. For example, using the matrix from FIG. 88, in the first cycle of the dot product calculation, the hardware would read the matrix buffer index "0" from the matrix index buffer 8903 along with the value "13" from the matrix value buffer 8905. In the second cycle, the value "0" from the matrix index buffer 8903 serves as the address for the vector value buffer 8904 to fetch the value of the vector component "2", which is then multiplied by "13" in cycle 3.

The value in the row start bit vector 8901 indicates to the hardware when to end a row of the matrix and start a new row. When the hardware reaches the end of a row, it places the accumulated dot products for the row in its output latches 8911 and begins accumulating dot products for the next row. The dot product latches of each dot product engine are connected in a daisy chain that assembles the output vector for write-back.

In sparse matrix-sparse vector multiplication, the vector tends to occupy much less memory than in sparse matrix-dense vector multiplication, but because it is sparse, it is not possible to directly fetch the vector component corresponding to a given index. Instead, the vector must be searched, and it is impractical to transfer to the dot product engine only the components that each dot product engine needs, making the time required to compute the dot product of the matrix data assigned to each dot product engine unpredictable. Due to this, the fetch list for sparse matrix-sparse vector multiplication only specifies the indices of the lowest and maximum nonzero components in the matrix block, and all of the nonzero components of the vector between those points must be broadcast to the dot product engines.

Figure 91 shows details of the dot product engine design supporting sparse matrix-sparse vector multiplication. To process a block of matrix data, the index (not the matrix buffer index used for sparse-dense multiplication) and value of the matrix dot product engine chunk are written to the matrix index and value buffers, as are the index and value of the region of the vector needed to process the block. The dot product engine control logic 9140 then sequences through index buffers 9102-9103 and outputs a block of four indices to the 4x4 comparator 9120. The 4x4 comparator 9120 compares each of the indices from the vector 9102 with each of the indices from the matrix 9103 and outputs any matching buffer addresses to the matched index queue 9130. The output of the matched index queue 9130 drives the read address inputs of the matrix value buffer 9105 and the vector value buffer 9104, which outputs the value corresponding to the match to the multiply-add ALU 9110. This hardware allows the dot product engine to consume eight indexes per cycle, at least four, as long as the matched index queue 9130 has free space, reducing the time required to process a block of data when index matching is rare.

As with the sparse matrix-dense vector dot product engine, the row start 9101 bit vector identifies an entry in the matrix buffers 9102-9103 that starts a new row of the matrix. When such an entry is encountered, the control logic 9140 resets to the beginning of the vector index buffer 9102 and compares them to the output of the matrix index buffer 9103, starting from the lowest of these values to examine the vector index. Similarly, when the end of the vector is reached, the control logic 9140 advances to the beginning of the next row in the matrix index buffer 9103 and resets to the beginning of the vector index buffer 9102. The "done" output informs the chip control unit when the dot product engine has finished processing a block of data or region of the vector, and is ready to proceed to the next one. To simplify one embodiment of the accelerator, the control logic 9140 does not advance to the next block/region until all of the dot product engines have finished processing.

In many cases, the vector buffer is large enough to hold all of the sparse vectors needed to process a block. In one embodiment, buffer space for 1024 or 2048 vector components is provided, depending on whether 32 or 64 bit values are used.

If the required elements of the vector do not fit in the vector buffer, a multi-pass approach may be used. The control logic 9140 broadcasts the complete buffer of the vector to each dot product engine and begins iterating through the rows in its matrix buffer. If the dot product engine reaches the end of the vector buffer before reaching the end of the row, it sets a bit in the bit vector 9111 of the current row position to indicate where it should resume processing the row when the next region of the vector is reached, saves the accumulated partial dot product in the matrix value buffer 9105 location corresponding to the start of the row as long as the start of the row has a higher index value than any of the vector indexes processed so far, and proceeds to the next row. After all of the rows in the matrix buffer have been processed, the dot product engine asserts its done signal to request the next region of the vector, and repeats the process until the entire vector has been read out.

Figure 92 shows an example using specific values. At the start of the calculation, a chunk of four components of the matrix is written to the matrix buffers 9103, 9105, and a region of four components of the vector is written to the vector buffers 9102, 9104. The row start 9101 and the current row position bit-vector 9106 both have a "value of 1010", indicating that the dot product engine chunk of the matrix contains two rows, one starting at the first component in the matrix buffer, and one starting at the third component.

When the first region is processed, the first row in the chunk looks up the index match at index 3, calculates the product of the corresponding elements of the matrix and vector buffers (4 x 1 = 4), and writes that value to the matrix value buffer 9105 location that corresponds to the start of the row. The second row looks up the index match at index 1, calculates the product of the corresponding components of the vector and matrix, and writes the result (6) to the matrix value buffer 9105 at the location that corresponds to its start. The state of the bit vector at the current row position changes to "0101", indicating that the first component of each row has been processed and the calculation should resume with the second component. The dot product engine then asserts its done line to signal that it is ready for another region of the vector.

When the dot product engine processes the second region of the vector, it sees that row 1 has an index match at index 4, calculates the product of the corresponding values of the matrix and the vector (5 x 2 = 10), adds that value to the partial dot product saved after the first vector region was processed, and outputs the result (14). As shown, the second row finds a match at index 7 and outputs the result 38. Saving the partial dot product and state of the calculation in this way avoids redundant work processing elements of the matrix that may not be able to match the index in later regions of the vector (because the vectors are sorted by index in ascending order) without requiring a large amount of additional storage for the partial product.

Figure 93 shows how the sparse-dense and sparse-sparse dot product engines described above can be combined to produce a dot product engine that can handle both types of calculations. Given the similarities between the two designs, the only modification required is to instantiate both the match logic 9311 of the sparse-dense dot product engine and the comparator 9320 of the sparse-sparse dot product engine, as well as the matched index queue 9330, along with a set of multiplexers 9350 that determine which module drives the read address and write buffers 9104-9105, and a multiplexer 9351 that selects whether the output of the matrix value buffer or the latched output of the matrix value buffer is sent to the multiply-accumulate ALU 9110. In one embodiment, these multiplexers are controlled by configuration bits in the control unit 9140 that are set at the start of the matrix-vector multiplication and remain in the same configuration throughout the operation.

A single accelerator stack provides performance comparable to a server CPU on sparse matrix operations, making accelerators attractive for smartphones, tablets, and other mobile devices. For example, there are many proposals for machine learning applications that train models on one or more servers and then deploy those models on mobile devices to process the arriving data streams. Since models tend to be much smaller than the datasets used to train them, the limited capacity of a single accelerator stack is less limiting in these applications, while the performance and power efficiency of accelerators allow mobile devices to process much more complex models than are feasible on their primary CPUs. Accelerators for non-mobile systems combine multiple stacks to provide extremely high bandwidth and performance.

Two examples of multi-stack implementations are shown in Fig. 94a and Fig. 94b. Both of these examples integrate several accelerator stacks on a package that is pin-compatible with modern server CPUs. Fig. 94a shows a socket-swap implementation with 12 accelerator stacks 9401-9412, and Fig. 94b shows a multi-chip package (MCP) implementation with a set of processors/cores 9430 (e.g., low-core count Xeon) and eight stacks 9421-9424. The 12 accelerator stacks in Fig. 94a are placed in an array that fits under the 39mm x 39mm heat spreader used in current packages, while the example in Fig. 94b incorporates eight stacks and a set of processors/cores in the same footprint. In one example, the physical dimensions used for the stacks are those for an 8GB WIO3 stack. Other DRAM technologies may have different dimensions and may change the number of stacks that fit into the package.

Both of these implementations provide low latency memory based communication over a KTI link between the CPU and accelerator. The socket replacement design for the Xeon implementation replaces one or more of the CPUs in a multi-socket system, providing 96GB capacity and 3.2TB/s stacked DRAM bandwidth. The expected power consumption is 90W, within the power budget of the Xeon socket. The MCP approach provides 64GB capacity and 2.2TB/s bandwidth, while consuming 60W of power in the accelerator. This leaves 90W for the CPU, assuming a power budget of 150W per socket, sufficient to support a medium-sized Xeon CPU. If the detailed package design allows for more logic space in the package, additional stacks or more powerful CPUs could be used, but this would require mechanisms such as the core parking technique being investigated for the hybrid part of the Xeon+FPGA to keep the total power consumption within the socket power budget.

Both of these designs can be implemented without the need for silicon interposers or other sophisticated integration technologies. Organic substrates used in current packages allow roughly 300 signals per cm of die perimeter, sufficient to support mid-stack KTI networks and off-package KTI links. Stacked DRAM designs can typically support logic chips that consume ~10W of power before cooling becomes an issue, well above the 2W logic die power estimate for a stack providing 256GB/sec of bandwidth. Finally, multi-chip packages require 1-2mm of space between chips for routing to be consistent with current designs.

In an embodiment, it may be implemented on a PCIe card and/or with a DDR4-T based accelerator. A 300W power limit for the PCIe card is assumed to allow the card to support 40 accelerator stacks for a total capacity of 320GB and 11TB/sec bandwidth. However, the long latency and limited bandwidth of the PCIe channel limits PCIe based accelerators to a major problem requiring only infrequent interaction with the CPU.

Alternatively, the accelerator stacks can be used to implement DDR-T DIMM-based accelerators 9501-9516, as shown in FIG. 95. DDR-T is a memory interface designed to be compatible with DDR4 sockets and motherboards. Using the same pinout and connector format as DDR4, DDR-T provides a transaction-based interface 9500 that allows the use of memory devices with different timing characteristics. In this example, the accelerator stacks 9501-9516 act as simple memory when not being used to perform computations.

Given a memory capacity of 126-256 GB and an aggregate bandwidth of 4-8 TB/s, if both sides of the card are used, DDR-T DIMMs provide enough space for 16 accelerator stacks or 32 accelerator stacks. However, such a system would consume 120-240 watts of power, much more than the 10W consumed by a DDR4-DIMM. This would require active cooling that would be difficult to fit into the limited space allotted per DIMM on the motherboard. Furthermore, DDR-T based accelerators may be attractive for applications where the user is not willing to give up any CPU performance for acceleration and is willing to pay the cost of a custom motherboard design that includes sufficient space between accelerator DIMMs for a fan or other cooling system.

In one embodiment, stacks in a multi-stack accelerator are split into separate KTI nodes and managed by system software as separate devices. The system firmware statically determines the routing table in the multi-stack accelerator at boot time based on the number of accelerator stacks present, and should uniquely determine the topology.

In one embodiment, the low level interface to the accelerator is implemented using Accelerator Abstraction Layer (AAL) software due to its suitability for socket-based accelerators. The accelerator may implement a caching agent as described by the Core Cache Interface Specification (CCI) and treat the stacked DRAM as private (non-coherent) memory for the accelerator that is not accessible by the host system (i.e., caching agent + private cache memory configuration, e.g., CA + PCM). The CCI specification mandates a separate Config/Status Register (CSR) address space per accelerator that is used by the driver that controls the accelerator. According to the specification, each accelerator communicates its status to the host via a Device Status Memory (DSM), and pinned memory regions are mapped into host memory that are used to indicate the accelerator's status. Thus, in a 12-stack system, there are 12 separate DSM regions managed by a single unified driver agent. These mechanisms may be used to create command buffers per stack. The command buffer is a pinned memory area mapped into system memory and is implemented as a circular queue managed by the AAL driver. The driver writes commands to each stack's command buffer, and each stack consumes items from its dedicated command buffer. Thus, command production and consumption are decoupled in this embodiment.

As an example, consider a system consisting of a single accelerator stack connected to a host CPU. A user writes code to perform the following calculation: wn+1=wn-αAwn, where A is a matrix and wx is a vector. The software framework and AAL driver decompose this code into the following sequence of commands:

TRANSMIT - Load the set of segments (wn+1, wn, α, A) into the private cache memory.

MULTIPLY - Multiply a series of segments (tmp = wn x α x A).

SUBTRACT - Submit a set of segments (wn+1=wn-tmp).

RECEIVE - Store a series of segments in host memory including the result (wn+1).

These commands operate on "partitions", coarse-grained units of data (approximately 16MB to 512MB) that are located in either the host or private cache memory. Partitions are intended to be easily mapped onto blocks of data that the MapReduce or Spark distributed computing systems use to facilitate the acceleration of distributed computations using accelerators. The AAL driver is responsible for creating a static one-to-one mapping of partitions to host memory regions or accelerator stacks. Accelerator stacks individually map their assigned partitions to their private cache memory (PCM) address space. A partition is represented by a partition index, which is a unique identifier, and the corresponding memory region (for the partition located in host memory) and data format. Partitions located in PCM are managed by a central controller, which determines the PCM address region for the partition.

In one embodiment, to initialize the accelerator's PCM, the host instructs the accelerator to load data from host memory. The TRANSMIT operation causes the accelerator to read host memory and store the read data in the accelerator's PCM. The data to be sent is described by a series of {partition index, host memory region, data format} tuples. To avoid data overhead being bundled by the host driver, the accelerator may implement System Protocol 2 (SPL2) Shared Virtual Memory (SVM).

The data format in each tuple represents the layout of the partition in memory. Examples of formats supported by the accelerator are compressed sparse rows (CSR) and multi-dimensional dense arrays. For the above example, A may be in CSR format, while wn may be in array format. The command specification contains the host memory addresses and information necessary to instruct the accelerator to load all partitions referenced by the TRANSMIT operation into the PCM.

Each operation may reference a small number of operands in the form of a set of partitions. For example, a multiplication operation causes the accelerator to read a stacked DRAM and perform a matrix-vector multiplication. Thus, in this example, we have four operands: destination vector tmp, multiplier A, multiplicand wn, and scalar α. The destination vector tmp is accumulated into a set of partitions specified by the driver as part of the command containing the operation. The command instructs the accelerator to initialize the set of partitions, if necessary.

The RECEIVE operation causes the accelerator to read PCM and write host memory. This operation may be implemented as an optional field on all other operations, potentially fusing commands to perform operations such as MULTIPLY with an instruction to store the result in host memory. The destination operand of the RECEIVE operation must be accumulated on-chip and then streamed to a partition in host memory and pinned by the driver prior to command dispatch (unless the accelerator implements SPL2 SVM).

Command dispatch flow

In one embodiment, after inserting a command into the command buffer for a stack, the driver generates a CSR write to inform the stack of the new command to be consumed. The CSR write by the driver is consumed by the accelerator stack's central controller, causing the control unit to generate a series of reads to the command buffer to read the command dispatched by the driver to the stack. When the accelerator stack completes a command, it writes status bits to its DSM. The AAL driver either polls or monitors these status bits to determine command completion. The output for a TRANSMIT or MULTPLY operation to the DSM is a status bit indicating completion. For a RECEIVE operation, the output to the DSM is a status bit and a series of segments to be written to host memory. The driver is responsible for identifying the region of memory to be written to by the accelerator. The control unit on the stack is responsible for generating a series of read operations to the stacked DRAM and corresponding writes to the destination segments in host memory.

Software enable

In one embodiment, users interact with the accelerator by calling a library of routines to move data onto the accelerator, perform sparse matrix calculations, etc. The API for this library can be as similar as possible to existing sparse matrix libraries to reduce the effort required to modify existing applications to take advantage of the accelerator. Another advantage of a library-based interface is that it hides the details of the accelerator and its data formats, allowing programs to take advantage of different implementations by dynamically linking in correct versions of the library at run-time. The library may also be implemented to call the accelerator from a distributed computing environment such as Spark.

The area and power consumption of the accelerator stack may be estimated by partitioning the design into modules (memory, ALU, etc.) and collecting data from a 14 nm design of similar structure. To scale to the 10 nm process, a 50% area reduction may be assumed along with a 25% Cdyn reduction and a 20% leakage power reduction. The area is estimated to include all on-chip memories and ALUs. We assume that wires run on the logic/memory. The power estimation includes active energy for the ALUs and memories, leakage power for the memories, and wire power for our main network. A base clock rate of 1 GHz is assumed, with a supply voltage of 0.65V for both the 14 nm and 10 nm processes. As already mentioned above, the ALUs may run at half the base clock rate, which shall be taken into account in the power estimation. The KTI links and mid-stack network are not included in the power estimation, as they are expected to be idle or near idle when the accelerator is performing computations. In one embodiment, we track activity on these networks and include it in the power estimation.

The estimate predicts that an accelerator as described herein would occupy 17 ^mm2 of chip area in a 14 nm process and ^8.5 mm2 in a 10 nm process, with most of the chip area being occupied by memory. Figure 96 shows a potential layout of an accelerator intended to be located under a WIO3 DRAM stack that includes 64 dot product engines 8420, 8 vector caches 8415, and an integrated memory controller 8410. The size and placement of the DRAM stack I/O bumps 9601, 9602 shown are defined by the WIO3 standard, and the accelerator logic would fit in the space between them. However, for ease of assembly, the logic die below the DRAM stack should be at least approximately as large as the DRAM die. Thus, the actual accelerator chip would be approximately 8 mm to 10 mm, but most of the area would be unused. In one embodiment, this unused area could be used for accelerators for different types of bandwidth-limited applications.

Stacked DRAM, as the name suggests, is a memory technology that stacks multiple DRAM dies vertically to provide higher bandwidth, closer physical integration with the compute die, and lower energy/bit than traditional DRAM modules such as DDR4 DIMMs. The table in Figure 97 compares seven DRAM technologies: non-stacked DDR4 and LPDDR4, pico-modules, JEDEC standard high bandwidth ( _HBM2 ) and wide I/O ( _WIO3 ) stacks, stacked DRAM, and dis-integrated RAM (DiRAM).

Stacked DRAMs come in two forms: on-die and lateral die. On-die stacks 8301-8304 connect directly to the logic die or SoC 8305 using through silicon vias, as shown in Figure 98a. On the other hand, lateral die stacks 8301-8304 are placed next to the logic/SoC die 8305 on a silicon interposer or bridge 9802, as shown in Figure 98b, with the connections between the DRAM and logic die running through the interposer 9802 and the interface layer 9801. On-die DRAM stacks have the advantage that they allow for a smaller package than lateral die stacks, but the disadvantage that it is difficult to attach more than one stack to a logic die, limiting the amount of memory they can provide per die. On the other hand, the use of silicon interposers 9802 allows the logic die to communicate with multiple lateral die stacks, at some cost depending on the area.

Two important properties for DRAM are the bandwidth per stack and the energy per bit, as they define the bandwidth that can be matched to the package and the power required to consume that bandwidth. These properties make _WIO3 , ITRI and DiRAM the most promising technologies for accelerators such as those described herein, as pico-modules do not provide sufficient bandwidth and the energy per bit of _HBM2 significantly increases power consumption.

Of the three technologies, DiRAM has the highest bandwidth and capacity as well as the lowest latency, making it very attractive. WIO ₃ is yet another promising option that is envisioned to become a JEDEC standard and offers good bandwidth and capacity. ITRI memory has the lowest energy/bit of the three, allowing more bandwidth to fit into a given power budget. It also has low latency and its SRAM-like interface will reduce the complexity for the accelerator's memory controller. However, ITRI RAM has the smallest capacity of the three, and its design trades off capacity for performance.

The accelerators described herein are designed to tackle data analytics and machine learning algorithms that are built on a core sparse matrix vector multiplication (SpMV) primitive. While SpMV often dominates the runtime of these algorithms, other operations are required to implement them as well.

As an example, consider the breadth-first search (BFS) listing shown in Figure 99. In this example, the bulk of the work is performed by SpMV on line 4. However, there is also a vector-vector subtraction (line 8), a dot product operation (line 9), and a data-parallel map operation (line 6). Vector subtraction and dot product are relatively simple operations commonly supported in vector ISAs and require little explanation.

On the other hand, data-parallel map operations are much more interesting because they introduce programmability to conceptually element-wise operations. A BFS example reveals the programmability provided by the mapping function of one embodiment. In particular, the lambda function in the BFS (see line 6 in Figure 99) is used to keep track of when a vertex was first being visited. This is done in one embodiment by passing two arrays and a scalar to the lambda function. The first array passed to the lambda function is the output of the SpMV operation and reflects which vertices are currently reachable. The second array has an entry for each vertex whose value is the iteration number the vertex was first seen, or 0 if the vertex has not yet been reached. The scalar passed to the lambda function is a simple loop iteration counter. In one embodiment, the lambda function is compiled into a series of scalar operations that are performed on each component of the input vector to generate the output vector.

The intermediate representation (IR) of the sequence of operations for BFS is shown in Figure 99. The BFS lambda IR reveals several interesting properties. The generated lambda code is guaranteed to have only a single basic block. In one embodiment, we perform if-conversions to prevent repetitive constructions in the lambda function and avoid control flow. This constraint significantly reduces the complexity of the computational structures used to execute lambdas, since it is not necessary to support general control flow.

All memory operations are performed at the start of the basic block (lines 2 to 4 in Figure 99). When translated to assembly, the memory operations are hoisted into the codelet preamble (lines 2 to 5).

Statistical evaluation was performed on benchmarks implemented with accelerators that use lambda functions. The number of instructions was recorded, as was the total number of registers and the total number of loads that quantifies the "complexity" of the various lambda functions of interest. Furthermore, the critical path length reflects the longest chain of dependent instructions in each lambda function. If the number of instructions is significantly longer than the critical path, instruction-level parallelism techniques are an applicable solution to improve performance. Some loads are invariant (all executions of a lambda function load the same value) for a given invocation or reduction call of the mapping. This situation is called "lambda invariant loads" and an analysis is performed to detect it.

Based on the analysis, a relatively small instruction store is required for the register file to support the execution of lambda functions. Techniques to increase parallelism (interleaving the execution of multiple lambda functions) will improve the size and complexity of the register file. However, the baseline design may be as small as 16 entries. Furthermore, if a single-bit predicate register file is also provided for use in compare and condition move operations, a 2R1W register file should be sufficient for all operations.

As described below, lambda immutable loads are performed in the gather engine, so that they are only executed once per invocation of a lambda function. The values returned by these loads are passed to the Processing Elements so that they can be read into the lambda datapath's local register file as needed.

In one embodiment, the execution of lambda functions is split between the gather engine and the processor elements (PEs) (e.g., dot product engines as described above) to exploit the different capabilities of each unit. Lambda functions have three types of arguments: constants, scalars, and vectors. Constants are arguments whose values can be determined at compile time. Scalar variables correspond to the lambda immutable loads described above, and are arguments whose values change between invocations of the lambda function, but remain constant across all of the elements that a given lambda function operates on. Vector arguments are arrays of data that the lambda function operates on, applying the instructions in the function to each element in the vector argument.

In one embodiment, a lambda function is specified by a descriptor data structure that contains the code that implements the function, any constants the function references, and pointers to its input and output variables. To execute a lambda function, a top-level controller sends a command to one or more gather engines that specifies a descriptor of the lambda function and the start and end indices of a portion of the function's vector arguments that the gather engines and their associated PEs are intended to process.

When the gather engine receives a command to execute a lambda function, it fetches the function's descriptor from memory and passes the descriptor, including the addresses of the function's scalar variables, to its associated PE until the last section of the descriptor is reached. It then fetches each of the function's scalar variables from memory, replaces the address of each argument in the descriptor with its value, and passes the modified descriptor to the PE.

When a PE receives the start of a function descriptor from its gather engine, it copies the addresses of the function's vector inputs into a control register, and the PE's fetch hardware begins loading pages of the vector inputs into the PE's local buffer. It then decodes each of the instructions that implement the lambda function and stores the results in a small decoded instruction buffer. The PE then waits for the values of the function's scalar variables to arrive from its gather engine, and for the first page of each of the function's vector arguments to arrive from memory. Once the function's arguments arrive, the PE begins applying the lambda function to each component within its range of input vectors, relying on the PE's fetch and writeback hardware to fetch pages of input data and, if necessary, write back pages of output values. When the PE reaches the end of its allocated range of data, it signals the top-level controller that it is done, and prepares to begin another process.

FIG. 100 illustrates the format of a descriptor used to specify a lambda function according to one embodiment. In particular, FIG. 100 illustrates the lambda descriptor format in memory 10001 and the lambda format descriptor passed to PE 10002. All fields in the descriptor except the instruction are 64-bit values. The instruction is a 32-bit value, packed two into a 64-bit word. The descriptor is organized with scalar variables appearing last, allowing the gather engine to pass everything but the scalar variables to the PE before fetching the scalar variables from memory. This allows the PE to decode the function's instructions and begin fetching its vector arguments while waiting for the gather engine to fetch the scalar variables. The descriptor and scalar variables of the lambda function are fetched through a vector cache to eliminate redundant DRAM accesses when a lambda function is distributed across multiple gather engine/PE pairs. As shown, the lambda descriptor format in memory 10001 may include a pointer to a scalar variable 10003, while the gather engine fetches the value of the scalar variable 10004 in the lambda descriptor format when it is passed to the PE 10002.

In one embodiment, the first word of each descriptor is a header that specifies the meaning of each word in the descriptor. As shown in FIG. 101, the lowest 6 bytes of the header word specify the number of vector arguments to the lambda function 10101, the number of constant arguments 10102, the number of vector and scalar outputs 10103-10104, the number of instructions 10105 in the function, and the number of scalar variables 10106 in the function (ordered to match where each type of data appears in the descriptor). The seventh byte of the header word specifies the location of the loop start instruction 10107 (e.g., the instruction where the hardware should start each iteration after the first byte) in the function's code. The higher order byte in the word is unused 10108. The remaining words contain function instructions, constants, and input/output addresses in the order shown in the figure.

No changes to the gather engine datapath are required to support lambda functions since all necessary operations can be supported by modifying the control logic. When the gather engine fetches a lambda descriptor from memory, it copies the line of the descriptor into both the vector component line buffer and the column descriptor buffer. Descriptor lines that do not contain addresses of scalar variables are passed to the PEs unmodified, while they keep running in the line buffers until the values of the scalar variables are fetched from memory and inserted into the line buffers at the locations of these addresses. The existing gather and unresponse buffer hardware can support this operation without modification.

Changes to processing elements to support lambda functions

In one embodiment, to support lambda functions, separate data paths are added to the PE as shown in FIG. 102, showing the matrix value buffer 9105, matrix index buffer 9103, and vector value buffer 9104 described above. While the PE's buffers remain the same, their names have been changed to Input Buffer 1, Input Buffer 2, and Input Buffer 3 to reflect their more common use in current implementations. The SpMV data path 9110 also remains unchanged from the base architecture. While it would be possible to implement SpMV as a lambda function, building dedicated hardware 10201 reduces power and improves performance of SpMV. Results from the SpMV data path 9110 and lambda data path 10201 are sent to the output buffer 10202 and ultimately to system memory.

Figure 103 shows details of one embodiment of a lambda datapath, including predicate register file 10301, register file 10302, decode logic 10303, decoded instruction buffer 10305, and revolves around an in-order execution pipeline 10304 implementing a load-store ISA. If a single issue execution pipeline cannot provide sufficient performance, one may exploit the data parallelism inherent in lambda operations to vectorize the execution pipeline to process multiple vector components in parallel, which should be a more energy efficient way to improve parallelism than leveraging ILP on individual lambda functions. The execution pipeline reads its input from a 16-32 entry register file 10302 with 64 bits per register and writes the results back to the 16-32 entry register file 10302. The hardware does not distinguish between integer and floating point registers, and any register may hold any type of data. The predicate register file 10301 holds the output of the compare operation, which is used for predicated instruction execution. In one embodiment, the lambda datapath 10304 does not support branch instructions, so any conditional execution must be done through predicate instructions.

At the start of each lambda function, the gather engine places the function's instructions into Input Buffer 3 9104 (a vector-valued buffer). The decode logic 10303 then sequentially decodes each instruction and places the results into the 32-entry decoded instruction buffer 10305. This saves the energy cost of repeatedly decoding each instruction for every iteration of Loop 1.

The lambda datapath contains four special control registers 10306. The index counter register holds the index of the vector component the lambda datapath is currently processing and is automatically incremented at the end of each iteration of the lambda. The last index register holds the index of the last vector component the PE is to process. The loop begin register holds the position in the decoded instruction buffer of the first instruction in the repeated portion of the lambda function, while the loop end register holds the position of the last instruction in the lambda function.

Execution of a lambda function begins with the first instruction in the decoded instruction buffer and proceeds until the pipeline reaches the instruction pointed to by the loop end register. At that point, the pipeline compares the value of the index counter register with the value of the last index register, and if the index counter is less than the last index, it makes an implicit branch back to the instruction pointed to by the loop begin register. Since the index counter register is only incremented at the end of each iteration, this check can be done in advance to avoid bubbles in the pipeline.

This scheme makes it easy to include "preamble" instructions that need only be executed on the first iteration of a lambda function. For example, a lambda function with two scalar and one constant input might start with three load instructions to fetch the values of those inputs into the register file, and set the loop begin register to point to the fourth instruction in the decoded instruction buffer so that the inputs are only read once, rather than on every iteration of the function.

In one embodiment, the Lambda Datapath implements a load-store ISA similar to many RISC processors. Lambda Datapath load and store instructions reference locations in the PE's SRAM buffers. All transfers of data between the SRAM buffers and DRAM are managed by the PE's fetch and writeback hardware. The Lambda Datapath supports two types of load instructions: scalar and component. A scalar load fetches the contents of a specified location in one of the SRAM buffers and places it in a register. Most of the scalar load instructions in a Lambda function occur in the function's preamble, but register pressure may occasionally require a scalar load to be placed in the loop body.

A component load fetches a component of an input vector for a lambda function. The PE maintains a computation pointer for each buffer that points to the current component of the first input vector that is mapped to that buffer. A component load specifies the target buffer and offset from the computation pointer. When a component instruction is executed, the hardware adds the specified offset to the value of the computation pointer modulo the size of the appropriate buffer and loads the data from that location in the register. A component store instruction is similar, but writes the data to the appropriate address in the PE output buffer 10202.

This approach allows multiple input and output vectors to be supported with the PE's existing fetch hardware. The input vectors alternate between input buffers 1 9105 and 2 9103 in the order specified by the lambda function's descriptor, and the fetch hardware reads an entire page of each vector in the buffer at the same time.

As an example, consider a function with three input vectors, A, B, and C. Input vector A is mapped onto the PE's Input Buffer 1 9105 at an offset of 0. Input B is mapped onto Input Buffer 2 9103, again at an offset of 0. Input C is mapped onto Input Buffer 1 9105 at an offset of 256 (assuming Tezzaron-style 256-byte pages). The PE's fetch hardware interleaves the pages of inputs A and C into Input Buffer 1 9105, while Input Buffer 2 9103 is filled with pages of input B. Each iteration of the lambda function fetches the appropriate component of input A by performing a component load from Buffer 1 9105 with an offset of 0, the appropriate component of input B with a component load from Buffer 2 9103 with an offset of 0, and that component of input C with a component load from Buffer 1 9105 with an offset of 256. At the end of each iteration, the hardware increments the computation pointer to advance to the next component of each input vector. When the computation pointer reaches the end of a page, the hardware increments it by (page size x (# of vector inputs mapped onto -1)) bytes to advance it to the first component of the next page of the first input vector in the buffer. A similar scheme is used to process lambda functions that generate multiple output vectors.

As shown in FIG. 104, in one embodiment, 8 bits are dedicated to opcode 10401. The remaining 24 bits are split between a single destination 10402 resulting in a 6-bit register specifier and three input operands 10403-10405. Control flow instructions are not used in one embodiment, constants are sourced from an auxiliary register file, and no bit allocation acrobatics are required to match large immediate values within the instruction word. In one embodiment, all instructions match the instruction encoding present in FIG. 104. The encoding for one particular set of instructions is shown in FIG. 105.

In one embodiment, the compare instruction uses a compare predicate. Exemplary compare predicate encodings are listed in the table of FIG. 106.

As described in detail above, in some instances it is advantageous to use an accelerator for a given task. However, there may be instances where it is not feasible and/or advantageous. For example, the accelerator may not be available, sending data to the accelerator may be too penalized, the accelerator may be slower than the processor core, etc. As a result, in some embodiments, additional instructions may provide performance and/or energy efficiency for some tasks.

An example of matrix multiplication is shown in Figure 109. Matrix multiplication is C[rowsA, colsB] += A[rowsA, comm] x B[comm, colsB]. As used herein for MADD (multiply and add instruction), a matrix x vector multiplication instruction is specified by setting colsB = 1. This instruction takes a matrix input A, a vector input B and a vector output C. In the context of 512 bit vectors, rowsA = 8 for double precision and 16 for single precision.

Many CPUs perform dense matrix multiplication via SIMD instructions that operate on one-dimensional vectors. Detailed herein are instructions (and underlying hardware) that extend the SIMD approach to include two-dimensional matrices (tiles) of size 8x4, 8x8, and larger. Through the use of this instruction, small matrices can be multiplied by vectors with the result appended to a destination vector. All operations are performed with a single instruction, amortizing the energy cost of fetching instructions and data over multiple multiply-and-accumulate operations. Additionally, some embodiments utilize a binary tree to perform sums (reductions) and/or include a register file that is integrated into the multiplier array to hold the input matrices as a collection of registers.

For matrix multiplication, an embodiment of the MADD instruction executes as follows:
//rowsA (e.g., 8) for (i=0; i<N; i++) packed data element size (e.g., vector length) for N=8
For (k=0; k<M; k++), //comm=M
C[i]+=A[i,k]×B[k]
Calculate.

Typically, the "A" operands are stored in eight packed data registers. The "B" operand may be stored in one packed data register or may be read from memory. The "C" operand is stored in one packed data register.

Throughout the remaining discussion of this instruction, an "octoMADD" version is described, which multiplies eight packed data element sources (e.g., eight packed data registers) by a packed data element source (e.g., a single register). Expanding the inner loop provides the following execution (for the octoMADD instruction) for a sequential implementation:
For (i=0; i<8; i++), C[i]+=A[i,0]×B[0]+
A[i,1]×B[1]+
A[i,2]×B[2]+
A[i,3]×B[3]+
A[i,4]×B[4]+
A[i,5]×B[5]+
A[i,6]×B[6]+
A[i,7]×B[7].

As shown, each multiplication of packed data elements from the corresponding packed data element positions of the "A" and "B" operands is followed by an addition. The sequential additions are decomposed into multiple simpler operations with minimal temporary storage.

In some embodiments, a binary tree approach is used. A binary tree minimizes latency by summing two subtrees in parallel and then adding the results together. This is applied recursively through the binary tree. The final result is added to the "C" destination operand.

Expanding the inner loop gives the following execution for a binary implementation (in terms of the octoMADD instruction):
For (i=0; i<8; i++),
C[i]+=((A[i,0]×B[0]+A[i,1]×B[1])+
(A[i,2]×B[2]+A[i,3]×B[3]))+
((A[i,4]×B[4]+A[i,5]×B[5])+
(A[i,6]×B[6]+A[i,7]×B[7])).

Figure 110 shows octoMADD instruction processing using a binary tree reduction network. The diagram shows one vector lane of the operation. With 512-bit vectors, double precision octoMADD has 8 lanes while single precision octoMADD has 16 lanes.

As shown, multiple multiplication circuits 11001-11015 perform multiplications for A[i,0] x B[0], A[i,1] x B[1], A[i,2] x B[2], A[i,3] x B[3], A[i,4] x B[4], A[i,5] x B[5], A[i,6] x B[6], and A[i,7] x B[7], respectively. In this example, i is the A register. Typically, the multiplications are performed in parallel.

Addition circuits 11017-11023 coupled to the multiplication circuits 11001-11015 add the results of the multiplication circuits 11001-11015. For example, the addition circuits perform A[i,0]xB[0]+A[i,1]xB[1], A[i,2]xB[2]+A[i,3]xB[3], A[i,4]xB[4]+A[i,5]xB[5], and A[i,6]xB[6]+A[i,7]xB[7]. Typically, the summations are performed in parallel.

The results of the first summation are summed and added together using adder circuit 11025. The result of this addition is added by adder circuit 11027 to the original (old) value 11031 from the destination to produce a new value 11033 that is stored in the destination.

In most implementations, the instruction cannot specify eight independent source registers plus other sources and registers for the destination or memory operands. Thus, in some examples, the octoMADD instruction specifies a limited range of eight registers for the matrix operand. For example, the octoMADD matrix operand may be registers 0-7. In some embodiments, a first register is specified and the registers consecutive to the first are additional (e.g., seven) registers.

Figure 111 illustrates an embodiment of a method performed by a processor to process a multiply-accumulate instruction.

At 11101, an instruction is fetched. For example, a multiply-and-accumulate instruction is fetched. The multiply-and-accumulate instruction includes an opcode, a field for a first packed data operand (either memory or register), one or more fields for the second through Nth packed data source operands, and a packed data destination operand. In some embodiments, the multiply-and-accumulate instruction includes a write mask operand. In some embodiments, the instruction is fetched from an instruction cache.

At 11103, the fetched instruction is decoded. For example, the fetched multiply-accumulate instruction is decoded by a decode circuit as described in detail herein.

At 11105, data values associated with the source operands of the decoded instruction are obtained. To avoid the need to repeatedly read these values from the main register file when executing a series of multiply-accumulate instructions, copies of these registers are built in the multiplier-adder array itself (as described in more detail below). These copies are maintained as a cache of the main register file.

At 11107, the decoded instruction is executed by an execution circuit (hardware) as described in detail herein to, for each packed data element position of the second through Nth packed data source operands, 1) multiply the data element of that packed data element position of the source operand by the data element of the corresponding packed data element position of the first source operand to generate a temporary result, 2) sum the temporary results, 3) add the sum of the temporary results to the data element of the corresponding packed data element position of the packed data destination operand, and 4) store the sum of the temporary results for the data element of the corresponding packed data element position of the destination in the corresponding packed data element position of the packed data destination operand. N is typically indicated by the opcode or prefix. For example, for octoMADD, N is 9 (so that there are 8 registers for A). The multiplications may be performed in parallel.

In some embodiments, at 11109, the instruction is committed or retired.

Figure 112 illustrates an embodiment of a method performed by a processor to process a multiply-accumulate instruction.

At 11201, an instruction is fetched. For example, a multiply-and-accumulate instruction is fetched. The fused multiply-and-accumulate instruction includes an opcode, a field for a first packed data operand (either memory or register), one or more fields for the second through Nth packed data source operands, and a packed data destination operand. In some embodiments, the fused multiply-and-accumulate instruction includes a write mask operand. In some embodiments, the instruction is fetched from an instruction cache.

At 11203, the fetched instruction is decoded. For example, the fetched multiply-accumulate instruction is decoded by a decode circuit as described in detail herein.

At 11205, data values associated with the source operands of the decoded instruction are obtained. To avoid the need to repeatedly read these values from the main register file when executing a series of multiply-accumulate instructions, copies of these registers are built in the multiplier-adder array itself (as described in more detail below). These copies are maintained as a cache of the main register file.

At 11207, the decoded instruction is executed by an execution circuit (hardware) as described in detail herein to: 1) multiply, for each packed data element position of the second through Nth packed data source operands, the data element at that packed data element position of the source operand by the data element at the corresponding packed data element position of the first source operand to generate a temporary result; 2) sum the temporary results pairwise; 3) add the sum of the temporary results to the data element at the corresponding packed data element position of the packed data destination operand; and 4) store the sum of the temporary results for the data element at the corresponding packed data element position of the destination in the corresponding packed data element position of the packed data destination operand. N is typically indicated by the opcode or prefix. For example, for octoMADD, N is 9 (so that there are 8 registers for A). The multiplications may be performed in parallel.

In some embodiments, at 11209, the instruction is committed or retired.

In some embodiments, when a MADD instruction is first encountered, the renamer injects micro-ops to copy the main registers to the cache, synchronizing the cached copies with the main register file. Subsequent MADD instructions continue to use the cached copies as long as they remain unchanged. In some embodiments, the octoMADD instruction anticipates the use of a limited range of registers, broadcasting writes to both the main register file and the cached copies when register values are generated.

Figures 113(A)-113(C) show exemplary hardware for executing the MADD instruction. Figure 113(A) shows the components that execute the MADD instruction. Figure 113(B) shows a subset of these components. In particular, multiple multiplier circuits 11323 are used to multiply packed data elements of source registers with each multiplier circuit 11323 coupled to an adder circuit 11327. Each adder circuit feeds the adder circuit 11327 in a chain fashion. Selector 11321 is used to select the external input or feedback of the adder circuit. The register file is embedded in the multiple adder array as part of the register file and reads multiplexer 11325. A specific register is hardwired to each column of multipliers.

Figure 113(B) shows the register file and read mux 11325. Register file 11327 is a number of registers (e.g., 4 or 8 registers) that store A as a cache. The corrected register is selected using read mux 11329.

The expected uses of the octoMADD instruction are as follows:
//Calculate C+=A*B
//A is loaded as an 8x8 tile in REG0-7
//B is loaded from memory as a 1x8 tile
//C is loaded and stored as 24x8 tiles in REG8-31
for (outer loop) {
load [24,8] tile of C matrix into REG 8-31 // 24 loads
for (middle loop) {
load [8,8] tile of A matrix into REG 0-7 // 8 loads
for (inner loop) {
// 24 iterations
REG [8-31 from inner loop] += REG 0-7 * memory[inner loop];
// 1 load
}
}
store [24,8] tile of C matrix from REG8-31 // 24 stores
}

The inner loop contains 24 octoMADD instructions, each of which reads one "B" operand from memory and sums it into one of 24 "C" accumulators. The middle loop loads the eight "A" registers with the new tile. The outer loop loads and stores the 24 "C" accumulators. The inner loop is unrolled to add prefetching in order to achieve high utilization (>90%) of the octoMADD hardware.

The following figures detail exemplary architectures and systems for implementing the above-described embodiments. In particular, aspects (e.g., registers, pipelines, etc.) of the core types (e.g., out-of-order, scalar, SIMD) discussed above are described. Additionally, systems including co-processors (e.g., accelerators, cores) and system implementations on chips are shown. In some embodiments, one or more of the hardware components and/or instructions described above are emulated and implemented as software modules, as described in more detail below.
Exemplary Register Architecture

Figure 125 is a block diagram of a register architecture 12500 according to one embodiment of the invention. In the embodiment shown, there are 32 vector registers 12510 that are 512 bits wide, and these registers are referenced as zmm0 through zmm31. The lower 256 bits of the lower 16 zmm registers are overlaid onto registers ymm0-16. The lower 128 bits of the lower 16 zmm registers (the lower 128 bits of the ymm registers) are overlaid onto registers xmm0-15. The instruction format QAC00 appropriate for a particular vector operates on these overlaid register files as shown in the table below.

That is, vector length field QAB59B selects between a maximum length and one or more other shorter lengths, each such shorter length being half the length of the preceding length, and without vector length field QAB59B the instruction template operates with the maximum vector length. Furthermore, in one embodiment, class B instruction templates of instruction format QAC00 suitable for a particular vector operate on packed or scalar single/double precision floating point data, and packed or scalar integer data. Scalar operations are operations performed on the lowest data element positions in zmm/ymm/xmm registers. Higher data element positions either remain the same as they were before the instruction or are zeroed out depending on the embodiment.

Write Mask Registers 12515 - In the illustrated embodiment, there are eight write mask registers (k0 through k7), each 64 bits in size. In an alternative embodiment, the write mask registers 12515 are 16 bits in size. As previously mentioned, in one embodiment of the invention, the vector mask register k0 cannot be used as a write mask. If an encoding that would normally indicate k0 is used for the write mask, it selects a hardwired write mask of 0xFFFF, effectively disabling write masking for that instruction.

General Purpose Registers 12525 - In the illustrated embodiment, there are sixteen 64-bit general purpose registers that are used in conjunction with existing x86 addressing modes to address memory operands. These registers are referenced by the names RAX, RBX, RCX, RDX, RBP, RSI, RDI, RSP, and R8 through R15.

Scalar floating point stack register file (x87 stack) 12545, into which the MMX packed integer flat register file 12550 is aliased - in the illustrated embodiment, the x87 stack is an 8-element stack used to perform scalar floating point operations on 32/64/80 bit floating point data using the x87 instruction set extensions, while the MMX registers are used to perform operations on 64 bit packed integer data, as well as to hold operands for some operations performed between the MMX and XMM registers.

Alternative embodiments of the invention may use wider or narrower registers. Additionally, alternative embodiments of the invention may use more, fewer or different register files and registers.
Exemplary Core Architectures, Processors and Computer Architectures

Processor cores may be implemented in different ways and in different processors for different purposes. By way of example, implementations of such cores may include: 1) a general purpose in-order core targeted for general purpose computing; 2) a high performance general purpose out-of-order core targeted for general purpose computing; 3) a special purpose core targeted primarily for graphics and/or scientific (throughput) computing. Implementations of different processors may include: 1) a CPU including one or more general purpose in-order cores targeted for general purpose computing and/or one or more general purpose out-of-order cores targeted for general purpose computing; and 2) a co-processor including one or more special purpose cores targeted primarily for graphics and/or scientific (throughput). Such various processors result in different computer system architectures, which may include: 1) a coprocessor on a separate chip from the CPU, 2) a coprocessor on a separate die in the same package as the CPU, 3) a coprocessor on the same die as the CPU (in which case the coprocessor may be referred to as special purpose logic or a special purpose core, such as integrated graphics and/or scientific (throughput) logic, and 4) a system on a chip that may include the described CPU (which may be referred to as an application core or application processor), the above-described coprocessor, and additional functionality on the same die. Exemplary core architectures are described next, followed by descriptions of exemplary processors and computer architectures.
Exemplary Core Architecture In-Order and Out-of-Order Core Block Diagram

Figure 126A is a block diagram illustrating both an exemplary in-order pipeline and an exemplary register renaming, out-of-order issue/execution pipeline according to an embodiment of the present invention. Figure 126B is a block diagram illustrating both an exemplary embodiment of an in-order architecture core and an exemplary register renaming, out-of-order issue/execution architecture core included in a processor according to an embodiment of the present invention. The solid boxes in Figures 126A-B show the in-order pipeline and the in-order core, while the optional addition of dashed boxes shows the register renaming, out-of-order issue/execution pipeline and core. The out-of-order aspect is described with the understanding that the in-order aspect is a subset of the out-of-order aspect.

In FIG. 126A, the processor pipeline 12600 includes a fetch stage 12602, a length decode stage 12604, a decode stage 12606, an allocation stage 12608, a renaming stage 12610, a scheduling (also known as dispatch or issue) stage 12612, a register read/memory read stage 12614, an execution stage 12616, a writeback/memory write stage 12618, an exception handling stage 12622, and a commit stage 12624.

126B illustrates a processor core 12690 including a front end unit 12630 coupled to an execution engine unit 12650, both coupled to a memory unit 12670. The core 12690 may be a reduced instruction set computing (RISC) core, a complex instruction set computing (CISC) core, a very long instruction word (VLIW) core, or a hybrid or alternative core type. As yet another option, the core 12690 may be a special purpose core such as, for example, a network or communication core, a compression engine, a co-processor core, a general purpose computing graphics processing unit (GPGPU) core, or a graphics core.

The front-end unit 12630 includes a branch prediction unit 12632 coupled to an instruction cache unit 12634, which is coupled to an instruction translation lookaside buffer (TLB) 12636, which is coupled to an instruction fetch unit 12638, which is coupled to a decode unit 12640. The decode unit 12640 (or decoder) may decode instructions and generate as output one or more micro-operations, microcode entry points, microinstructions, other instructions, or other control signals that are decoded from or otherwise reflect or are derived from the original instruction. The decode unit 12640 may be implemented using a variety of different mechanisms. Examples of suitable mechanisms include, but are not limited to, a lookup table, a hardware implementation, a programmable logic array (PLA), a microcode read only memory (ROM), etc. In one embodiment, the core 12690 includes a microcode ROM or other medium that stores microcode for particular macro-instructions (e.g., in the decode unit 12640 or otherwise in the front-end unit 12630). The decode unit 12640 is coupled to a rename/allocator unit 12652 in the execution engine unit 12650.

The execution engine unit 12650 includes a rename/allocator unit 12652 coupled to a retirement unit 12654 and a set of one or more scheduler units 12656. The scheduler units 12656 represent any number of various schedulers, including reservation stations, central instruction windows, and the like. The scheduler units 12656 are coupled to physical register file units 12658. Each of the physical register file units 12658 represents one or more physical register files, different ones of which store one or more different data types, e.g., scalar integer, scalar floating point, packed integer, packed floating point, vector integer, vector floating point, status (e.g., an instruction pointer, which is the address of the next instruction to be executed), and the like. In one embodiment, the physical register file units 12658 include a vector register unit, a write mask register unit, and a scalar register unit. These register units may provide the vector registers, vector mask registers, and general purpose registers of the architecture. The physical register files unit 12658 is overlaid with the retirement unit 12654 to illustrate various ways in which register renaming and out-of-order execution may be implemented (e.g., with a reorder buffer and retirement register file, with a future file, a history buffer and retirement register file, with register mapping and pooling of registers, etc.). The retirement unit 12654 and the physical register files unit 12658 are coupled to the execution clusters 12660. The execution clusters 12660 include a set of one or more execution units 12662 and a set of one or more memory access units 12664. The execution units 12662 may perform various operations (e.g., shift, add, subtract, multiply) on various types of data (e.g., scalar floating point, packed integer, packed floating point, vector integer, vector floating point). Some embodiments may include multiple execution units dedicated to a particular function or set of functions, while other embodiments may include only one execution unit, or multiple execution units that all perform all functions. Scheduler unit 12656, physical register file unit 12658, and execution cluster 12660 are sometimes shown as multiple because certain embodiments create separate pipelines for particular types of data/operations (e.g., scalar integer pipeline, scalar floating point/packed integer/packed floating point/vector integer/vector floating point pipeline, and/or memory access pipeline, each with their own scheduler unit, physical register file unit, and/or execution cluster - in the case of a separate memory access pipeline, the execution cluster of this pipeline is implemented in certain embodiments with only memory access unit 12664). It should also be understood that when separate pipelines are used, one or more of these pipelines may be out-of-order issue/execution and the rest may be in-order.

The set of memory access units 12664 is coupled to a memory unit 12670, which includes a data TLB unit 12672 coupled to a data cache unit 12674 coupled to a level 2 (L2) cache unit 12676. In one exemplary embodiment, the memory access units 12664 may include a load unit, a store address unit, and a store data unit, each coupled to the data TLB unit 12672 in the memory unit 12670. The instruction cache unit 12634 is further coupled to a level 2 (L2) cache unit 12676 in the memory unit 12670. The L2 cache unit 12676 is coupled to one or more other levels of cache and ultimately to main memory.

By way of example, an exemplary register renaming, out-of-order issue/execution core architecture may implement a pipeline 12600 as follows: 1) instruction fetch 12638 performs fetch and length decoding stages 12602 and 12604; 2) decode unit 12640 performs decode stage 12606; 3) rename/allocator unit 12652 performs allocation stage 12608 and renaming stage 12610; 4) scheduler unit 12656 performs scheduling stage 12612; 5) physical register file unit 12658 and memory unit 12670 perform 5) the execution cluster 12660 executes the execution stage 12616; 6) the memory unit 12670 and the physical register file unit 12658 execute the writeback/memory write stage 12618; 7) various units may relate to the exception handling stage 12622; and 8) the retirement unit 12654 and the physical register file unit 12658 execute the commit stage 12624.

Core 12690 may support one or more instruction sets including the instructions described herein (e.g., the x86 instruction set (with some extensions added in newer versions), the MIPS instruction set from MIPS Technology of Sunnyvale, Calif., the ARM instruction set (with optional additional extensions such as NEON from ARM Holdings, Inc. of Sunnyvale, Calif.). In one embodiment, core 12690 includes logic to support packed data instruction set extensions (e.g., AVX1, AVX2), thereby enabling operations used by many multimedia applications to be executed with packed data.

It should be appreciated that a core may support multithreading (executing two or more parallel sets of operations or threads) in a variety of ways, including time-sliced multithreading, simultaneous multithreading (where a single physical core provides a logical core for each thread that the physical core simultaneously multithreads), or a combination thereof (e.g., simultaneous multithreading and time-sliced fetching and decoding, such as Intel's later Hyper-Threading Technology).

While register renaming is described in the context of out-of-order execution, it should be understood that register renaming may be used in an in-order architecture. The illustrated embodiment of the processor also includes separate instruction and data cache units 12634/12674 and a shared L2 cache unit 12676, while alternative embodiments may have a single internal cache for both instructions and data, such as a level 1 (L1) internal cache or multiple levels of internal cache. In some embodiments, the system may include a combination of an internal cache and an external cache that is external to the core and/or processor. Alternatively, all of the cache may be external to the core and/or processor.
Specific Exemplary In-Order Core Architecture

Figures 127A-127B show block diagrams of a more specific exemplary in-order core architecture, where the core may be one of several logic blocks (including other cores of the same and/or different types) within a chip. The logic block communicates with some fixed function logic, memory I/O interfaces, and other necessary I/O logic, depending on the application, through a high bandwidth interconnect network (e.g., a ring network).

127A is a block diagram of a single processor core along with its connection to an on-die interconnect network 12702 and its connection to a local subset of a level 2 (L2) cache 12704, according to an embodiment of the present invention. In one embodiment, the instruction decoder 12700 supports the x86 instruction set with the packed data instruction set extension. The L1 cache 12706 allows low latency access to cache memory in the scalar and vector units. In one embodiment (to simplify the design), the scalar unit 12708 and the vector unit 12710 use separate register sets (scalar registers 12712 and vector registers 12714, respectively) and data transferred between them is written to memory and then read back from the level 1 (L1) cache 12706, while alternative embodiments of the present invention may use different approaches (e.g., using a single register set or including a communication path that allows data to be transferred between the two register files without being written and read back).

The local subset of the L2 cache 12704 is part of a global L2 cache that is divided into separate local subsets, one for each processor core. Each processor core has a direct access path to its own local subset of the L2 cache 12704. Data read by a processor core is stored in its L2 cache subset 12704 and can be accessed quickly in parallel with other processor cores accessing their own local L2 cache subsets. Data written by a processor core is stored in its own L2 cache subset 12704 and is flushed from other subsets when necessary. The ring network ensures coherency for shared data. The ring network is bidirectional to allow agents such as processor cores, L2 caches, and other logic blocks to communicate with each other within the chip. Each ring data path is 1012 bits wide per direction.

127B is an expanded view of a portion of the processor core in FIG. 127A, according to an embodiment of the present invention. FIG. 127B includes the L1 data cache 12706A portion of the L1 cache 12704, as well as further details regarding the vector unit 12710 and vector registers 12714. In particular, the vector unit 12710 is a 16-wide vector processing unit (VPU) (see 16-wide ALU 12728), which executes one or more of integer, single precision float, and double precision float instructions. The VPU supports swizzling of register inputs using swizzle unit 12720, numeric conversion using numeric conversion units 12722A-B, and duplication on memory inputs using duplication unit 12724. A write mask register 12726 allows for predicating of the resulting vector writes.

128 is a block diagram of a processor 12800 that may have more than one core, an integrated memory controller, and integrated graphics, according to an embodiment of the present invention. The solid box in FIG. 128 illustrates a processor 12800 having a single core 12802A, a system agent 12810, and a set of one or more bus controller units 12816, while the optional additional portion of the dashed box illustrates an alternative processor 12800 having multiple cores 12802A-N, a set of one or more integrated memory controller units 12814 in the system agent unit 12810, and special purpose logic 12808.

Thus, different implementations of the processor 12800 may include: 1) a CPU with special purpose logic 12808 (which may include one or more cores) that is integrated graphics and/or science and technology (throughput) logic, and one or more general purpose cores 12802A-N (e.g., general purpose in-order cores, general purpose out-of-order cores, combinations of the two); 2) a coprocessor with multiple special purpose cores 12802A-N that are primarily targeted at graphics and/or science and technology (throughput); and 3) a coprocessor with multiple general purpose in-order cores 12802A-N. Thus, the processor 12800 may be a general purpose processor, coprocessor, or special purpose processor, such as a network or communication processor, compression engine, graphics processor, GPGPU (general purpose graphics processing unit), high throughput multi-integrated core (MIC) coprocessor (containing 30 or more cores), or embedded processor. The processor may be implemented on one or more chips. The processor 12800 may be part of one or more substrates and/or may be implemented on one or more substrates using any of a number of process technologies, such as, for example, BiCMOS, CMOS, or NMOS.

The memory hierarchy includes one or more levels of cache within the cores, a set of one or more shared cache units 12806, and an external memory (not shown) coupled to a set of integrated memory controller units 12814. The set of shared cache units 12806 may include one or more mid-level caches, such as level 2 (L2), level 3 (L3), level 4 (L4) or other level caches, a last level cache (LLC), and/or combinations thereof. In one embodiment, a ring-based interconnect unit 12812 interconnects the integrated graphics logic 12808 (the integrated graphics logic 12808 is an example of special purpose logic, also referred to herein as special purpose logic), the set of shared cache units 12806, and the system agent unit 12810/integrated memory controller unit 12814, while alternative embodiments may use any number of well-known techniques for interconnecting such units. In one embodiment, coherency is maintained between the one or more cache units 12806 and the cores 12802A-N.

In some embodiments, one or more of the cores 12802A-N are capable of multithreading. The system agent 12810 includes those components that coordinate and operate the cores 12802A-N. The system agent unit 12810 may include, for example, a power control unit (PCU) and a display unit. The PCU may be or may include the logic and components required to coordinate the power state of the cores 12802A-N and the integrated graphics logic 12808. The display unit is for driving one or more externally connected displays.

Cores 12802A-N may be homogeneous or heterogeneous in terms of architectural instruction set, that is, two or more of cores 12802A-N may be capable of executing the same instruction set, while others may be capable of executing only a subset of that instruction set or a different instruction set.
Exemplary Computer Architecture

129-132 are block diagrams of exemplary computer architectures. Other system designs and configurations known in the art for laptops, desktops, handheld PCs, personal digital assistants, engineering workstations, servers, network devices, network hubs, switches, embedded processors, digital signal processors (DSPs), graphics devices, video game devices, set-top boxes, microcontrollers, mobile phones, portable media players, handheld devices, and a variety of other electronic devices are also suitable. In general, a wide variety of systems or electronic devices that may incorporate a processor and/or other execution logic as disclosed herein are generally suitable.

129, shown is a block diagram of a system 12900 in accordance with one embodiment of the present invention. The system 12900 may include one or more processors 12910, 12915 coupled to a controller hub 12920. In one embodiment, the controller hub 12920 includes a graphics memory controller hub (GMCH) 12990 and an input/output hub (IOH) 12950 (which may be on separate chips), where the GMCH 12990 includes a memory and graphics controller coupled to the memory 12940 and the coprocessor 12945, and the IOH 12950 couples the input/output (I/O) devices 12960 to the GMCH 12990. Alternatively, one or both of the memory and the graphics controller may be integrated within the processor (as described herein), with the memory 12940 and the coprocessor 12945 being directly coupled to the processor 12910 and the controller hub 12920 in a single chip with the IOH 12950.

Optional features of additional processor 12915 are shown in FIG. 129 with dashed lines. Each processor 12910, 12915 may include one or more of the processing cores described herein and may be some version of processor 12800.

Memory 12940 may be, for example, dynamic random access memory (DRAM), phase change memory (PCM), or a combination of the two. For at least one embodiment, controller hub 12920 communicates with processors 12910, 12915 via, for example, a multi-drop bus such as a front side bus (FSB), a point-to-point interface such as a QuickPath Interconnect (QPI), or a similar connection 12995.

In one embodiment, the coprocessor 12945 is a special purpose processor, such as a high throughput MIC processor, a network or communication processor, a compression engine, a graphics processor, a GPGPU, or an embedded processor. In one embodiment, the controller hub 12920 may include an integrated graphics accelerator.

There may be various differences between the physical resources 12910, 12915 in terms of a wide range of value criteria, including architectural characteristics, microarchitectural characteristics, thermal characteristics, and power consumption characteristics.

In one embodiment, the processor 12910 executes instructions that control a general type of data processing operation. Embedded within the instructions may be coprocessor instructions. The processor 12910 recognizes these coprocessor instructions as being of a type to be executed by an attached coprocessor 12945. Optionally, the processor 12910 issues these coprocessor instructions (or control signals representing the coprocessor instructions) to the coprocessor 12945 via a coprocessor bus or other interconnect. The coprocessor 12945 accepts and executes the received coprocessor instructions.

Referring now to FIG. 130, shown is a block diagram of a first more specific exemplary system 13000 according to an embodiment of the present invention. As shown in FIG. 130, the multiprocessor system 13000 is a point-to-point interconnect system and includes a first processor 13070 and a second processor 13080 coupled via a point-to-point interconnect 13050. Each of the processors 13070 and 13080 may be some version of the processor 12800. In one embodiment of the present invention, the processors 13070 and 13080 are processors 12910 and 12915, respectively, while the coprocessor 13038 is coprocessor 12945. In another embodiment, the processors 13070 and 13080 are processors 12910 and coprocessor 12945, respectively.

Processors 13070 and 13080 are shown including integrated memory controller (IMC) units 13072 and 13082, respectively. Processor 13070 also includes point-to-point (PP) interfaces 13076 and 13078 as part of its bus controller unit. Similarly, second processor 13080 includes PP interfaces 13086 and 13088. Processors 13070, 13080 may exchange information via point-to-point (PP) interface 13050 using PP interface circuits 13078, 13088. As shown in FIG. 130, IMCs 13072 and 13082 couple the processors to memory, respectively, i.e., memory 13032 and memory 13034, and may be part of a main memory locally attached to the respective processor.

The processors 13070, 13080 may exchange information with each chipset 13090 via separate P-P interfaces 13052, 13054 using point-to-point interface circuits 13076, 13094, 13086, 13098. The chipset 13090 may selectively exchange information with the coprocessor 13038 via a high performance interface 13092. In one embodiment, the coprocessor 13038 is a special purpose processor such as, for example, a high throughput MIC processor, a network or communication processor, a compression engine, a graphics processor, a GPGPU, or an embedded processor.

A shared cache (not shown) may be included within either processor or external to both processors and may be further connected to the processors via the P-P interconnect such that local cache information of one or both processors may be stored in the shared cache when the processors are placed in a low power mode.

Chipset 13090 may be coupled to a first bus 13016 via an interface 13096. In one embodiment, first bus 13016 may be a bus such as a Peripheral Component Interconnect (PCI) bus, or a PCI Express bus or another third generation I/O interconnect bus, without limiting the scope of the present invention.

As shown in FIG. 130, various I/O devices 13014 may be coupled to the first bus 13016 along with a bus bridge 13018 coupling the first bus 13016 to a second bus 13020. In one embodiment, one or more additional processors 13015, such as a co-processor, a high throughput MIC processor, an accelerator of a GPGPU (e.g., a graphics accelerator or a digital signal processing (DSP) unit), a field programmable gate array, or any other processor, are coupled to the first bus 13016. In one embodiment, the second bus 13020 may be a low pin count (LPC) bus. In one embodiment, various devices may be coupled to the second bus 13020 including, for example, a keyboard and/or mouse 13022, a communication device 13027, and a storage unit 13028, such as a disk drive or other mass storage device, which may contain instructions/code and data 13030. Additionally, audio I/O 13024 may be coupled to the second bus 13020. Note that other architectures are possible. For example, instead of the point-to-point architecture of FIG. 130, the system may implement a multi-drop bus or other such architecture.

Referring now to FIG. 131, shown is a block diagram of a second, more specific, exemplary system 13100 in accordance with an embodiment of the present invention. Like elements in FIG. 130 and FIG. 131 are labeled with like reference numbers, and certain aspects of FIG. 130 have been omitted from FIG. 131 to avoid obscuring other aspects of FIG. 131.

Figure 131 shows that the processors 13070, 13080 may include integrated memory and I/O control logic ("CL") 13072 and 13082, respectively. Thus, the CLs 13072, 13082 include integrated memory controller units and include I/O control logic. Figure 131 shows that not only are the memories 13032, 13034 coupled to the CLs 13072, 13082, but also the I/O devices 13114 are coupled to the control logic 13072, 13082. Legacy I/O devices 13115 are coupled to the chipset 13090.

Referring now to FIG. 132, shown is a block diagram of a SoC 13200 according to an embodiment of the present invention. Like elements in FIG. 128 are labeled with like reference numbers. Also, dashed boxes are optional features on more advanced SoCs. In FIG. 132, an interconnect unit 13202 is coupled to an application processor 13210 including a set of one or more cores 12802A-N including cache units 12804A-N and a shared cache unit 12806, a system agent unit 12810, a bus controller unit 12816, an integrated memory controller unit 12814, a set of one or more coprocessors 13220 that may include integrated graphics logic, image processors, audio processors, and video processors, a static random access memory (SRAM) unit 13230, a direct memory access (DMA) unit 13232, and a display unit 13240 for coupling to one or more external displays. In one embodiment, the coprocessor 13220 includes a special purpose processor, such as, for example, a network or communication processor, a compression engine, a GPGPU, a high-throughput MIC processor, or an embedded processor.

Embodiments of the mechanisms disclosed herein may be implemented in hardware, software, firmware, or a combination of such implementation approaches. Embodiments of the invention may be implemented as a computer program or program code running on a programmable system having at least one processor, a storage system (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device.

Program code, such as code 13030 shown in FIG. 130, may be applied to the input instructions to perform functions described herein and to generate output information. The output information may be applied to one or more output devices, in known fashion. For purposes of this application, a processing system includes any system having a processor, such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

The program code may be implemented in a high level procedural or object oriented programming language to communicate with a processing system. The program code may also be implemented in assembly or machine language, if desired. Indeed, the mechanisms described herein are not limited in scope to any particular programming language. In any case, the language may be a compiled or interpreted language.

In one or more aspects of at least one embodiment, the techniques may be implemented by representative instructions stored on a machine-readable medium that represent various logic within a processor that, when read by a machine, causes the machine to construct the logic to perform the techniques described herein. Such representations, known as "IP cores," may be stored on tangible machine-readable media and supplied to various customers or manufacturing facilities for loading into manufacturing machines that actually create the logic or processors.

Such machine-readable storage media may include, without limitation, non-transitory tangible articles of construction manufactured or formed by a machine or device, including storage media such as hard disks, other types of disks including floppy disks, optical disks, compact disk read-only memories (CD-ROMs), compact disk re-writeables (CD-RWs), and magneto-optical disks, read-only memories (ROMs), random access memories (RAMs) such as dynamic random access memories (DRAMs), static random access memories (SRAMs), erasable programmable read-only memories (EPROMs), flash memories, electrically erasable programmable read-only memories (EEPROMs), phase change memories (PCMs), semiconductor devices such as magnetic or optical cards, or other types of media suitable for storing electronic instructions.

Depending on the circumstances, embodiments of the invention may include a non-transitory, tangible, machine-readable medium that includes instructions, such as a hardware description language (HDL), or that includes design data, that defines the structures, circuits, devices, processors and/or system functions described herein. In such embodiments, they may also be referred to as program products.
Emulation (including binary conversion, chording, morphing, etc.)

In some cases, an instruction converter may be used to convert instructions from a source instruction set to a target instruction set. For example, the instruction converter may translate (e.g., using static binary translation, dynamic binary translation including dynamic compilation), morph, emulate, or otherwise convert instructions into one or more other instructions for processing by the core. The instruction converter may be implemented in software, hardware, firmware, or a combination thereof. The instruction converter may be on-processor, off-processor, or some on-processor and some off-processor.

Figure 133 is a block diagram contrasting the use of a software instruction converter to convert binary instructions in a source instruction set to binary instructions in a target instruction set, according to an embodiment of the present invention. In the illustrated embodiment, the instruction converter is a software instruction converter, but alternatively, the instruction converter may be implemented in software, firmware, hardware, or various combinations thereof. Figure 133 illustrates that a program in a high-level language 13302 may be compiled using an x86 compiler 13304 to generate x86 binary code 13306 that may be substantially executed by a processor having at least one x86 instruction set core 13316. A processor having at least one x86 instruction set core 13316 represents any processor that can perform substantially the same functions as an Intel processor having at least one x86 instruction set core by compatibly executing or otherwise processing (1) a substantial portion of the instruction set of an Intel x86 instruction set core, or (2) an object code version of an application or other software intended to run on an Intel processor having at least one x86 instruction set core, to achieve substantially the same results as an Intel processor having at least one x86 instruction set core. x86 compiler 13304 represents a compiler operable to generate x86 binary code 13306 (e.g., object code) that can be executed on a processor having at least one x86 instruction set core 13316, with or without additional linkage processing. Similarly, diagram 133 shows that a program in high level language 13302 may be compiled using an alternative instruction set compiler 13308 to generate alternative instruction set binary code 13310 that may be natively executed by a processor without at least one x86 instruction set core 13314 (e.g., a processor having a core that implements the MIPS instruction set MIPS Technology of Sunnyvale, Calif. and/or implements the ARM instruction set of ARM Holdings, Inc. of Sunnyvale, Calif.). An instruction converter 13312 is used to convert the x86 binary code 13306 into code that may be natively executed by a processor without the x86 instruction set core 13314. This converted code is unlikely to be the same as the alternative instruction set binary code 13310, since an instruction converter that can do this would be difficult to create. However, the converted code implements common operations and is composed of instructions from the alternative instruction set. Thus, the instruction converter 13312 represents software, firmware, hardware, or a combination thereof that enables a processor or other electronic device that does not have an x86 instruction set processor or core to execute the x86 binary code 13306 through emulation, simulation, or other processes.

Exemplary implementations, embodiments, and specific combinations of features and aspects are described in detail below. These examples are provided for illustrative purposes only and are not limiting.

Example 1. A system including a plurality of heterogeneous processing elements and a hardware heterogeneous scheduler that dispatches instructions for execution to one or more of the plurality of heterogeneous processing elements, the instructions corresponding to code fragments to be processed by one or more of the plurality of heterogeneous processing elements, the instructions being native instructions for at least one of the one or more of the plurality of heterogeneous processing elements.

Example 2: The system described in Example 1, in which the multiple heterogeneous processing elements include an in-order processor core, an out-of-order processor core, and a packed data processor core.

Example 3: The system of Example 2, wherein the plurality of heterogeneous processing elements further comprises an accelerator.

Example 4: The system of any of Examples 1-3, wherein the hardware heterogeneous scheduler further includes a program phase detector that detects a program phase of the code fragment, the plurality of heterogeneous processing elements including a first processing element having a first microarchitecture and a second processing element having a second microarchitecture different from the first microarchitecture, the program phase being one of a plurality of program phases including a first phase and a second phase, the dispatching of instructions being based in part on the detected program phase, and the processing of the code fragment by the first processing element improves performance per watt characteristics compared to the processing of the code fragment by the second processing element.

Example 5: The system of any of Examples 1-4, wherein the hardware heterogeneous scheduler further comprises a selector that selects a processing element type for the plurality of processing elements to execute the received code fragment and schedules the code fragment to a processing element of the selected type among the plurality of processing elements using dispatch.

Example 6: The system of example 1, wherein the code fragment is one or more instructions associated with a software thread.

Example 7: The system of any of Examples 5-6, wherein for a data parallel program phase, the selected type of processing element is a processing core that executes single instruction multiple data (SIMD) instructions.

Example 8: The system of any of Examples 5-7, wherein for a data parallel program phase, the selected type of processing element is a circuit that supports dense arithmetic primitives.

Example 9: A system as described in any of Examples 5-7, where in the case of a data parallel program phase, the selected type of processing element is an accelerator.

Example 10: The system of any of Examples 5-9, wherein the data parallel program phases have data elements that are processed simultaneously using the same control flow.

Example 11: A system as described in any of Examples 5-10, in which for a thread-parallel program phase, the selected type of processing element is a scalar processing core.

Example 12: A system as described in any of Examples 5-11, in which a thread-parallel program phase has a data-dependent branch with a unique control flow.

Example 13: A system as described in any of Examples 2-12, wherein in the case of a serial program phase, the selected type of processing element is an out-of-order core.

Example 14: The system of any of Examples 2-13, wherein for a data parallel program phase, the selected type of processing element is a processing core that executes single instruction multiple data (SIMD) instructions.

Example 15: A system as described in any of Examples 1-14, in which the hardware heterogeneous scheduler supports multiple code types including compiled, intrinsic, assembly, library, intermediate, offload and device.

Example 16: A system according to any of Examples 5-15, in which the hardware heterogeneous scheduler emulates functionality if a processing element of a selected type cannot process the code fragment natively.

Example 17: A system according to any of Examples 1-15, in which the hardware heterogeneous scheduler emulates functionality when the number of available hardware threads is oversubscribed.

Example 18: A system according to any of Examples 5-15, in which the hardware heterogeneous scheduler emulates functionality if a processing element of a selected type cannot process the code fragment natively.

Example 19: A system as described in any of Examples 5-18, in which the selection of the processing element type for the plurality of heterogeneous processing elements is transparent to a user.

Example 20: A system as described in any of Examples 5-19, in which the selection of the processing element type for the plurality of heterogeneous processing elements is transparent to the operating system.

Example 21: A system as described in any of Examples 1-20, in which the hardware heterogeneous scheduler presents a homogeneous multiprocessor programming model so that each thread appears to the programmer as if it is running on a scalar core.

Example 22: The system of Example 21, in which the presented homogeneous multiprocessor programming model presents emergent support for the complete instruction set.

Example 23: A system as described in any of Examples 1-22, in which multiple heterogeneous processing elements share a memory address space.

Example 24: A system as described in any of Examples 1-23, wherein the hardware heterogeneous scheduler includes a binary translator executing on one of the heterogeneous processing elements.

Example 25: A system as described in any of Examples 5-24, in which the default selection of processing element type for multiple heterogeneous processing elements is a latency optimized core.

Example 26: A system according to any of Examples 1-25, in which the heterogeneous hardware scheduler selects a protocol to use in the multi-protocol interface for a dispatched instruction.

Example 27: A system as described in any of Examples 26, wherein the first protocol supported by the multi-protocol bus interface comprises a memory interface protocol used to access a system memory address space.

Example 28: A system as described in any of Examples 26-27, wherein the second protocol supported by the multi-protocol bus interface includes a cache coherency protocol that maintains coherency between data stored in the accelerator's local memory and the host processor's memory subsystem, including the host cache hierarchy and the system memory.

Example 29: The system of any of Examples 26-28, wherein the third protocol supported by the multiprotocol bus interface includes a serial link protocol supporting device discovery, register access, configuration, initialization, interrupts, direct memory access, and address translation services.

Example 30: The system of Example 29, wherein the third protocol comprises a Peripheral Component Interface Express (PCIe) protocol.

Example 31: A system including: a plurality of heterogeneous processing elements in a heterogeneous processor including an accelerator; and a memory storing program code executable by at least one of the plurality of heterogeneous processing elements in the heterogeneous processor, the program code being a heterogeneous scheduler dispatching instructions for execution on one or more of the plurality of heterogeneous processing elements, the instructions corresponding to code fragments to be processed by one or more of the plurality of heterogeneous processing elements, the instructions being native instructions for at least one of the one or more of the plurality of heterogeneous processing elements.

Example 32: The system of Example 31, wherein the plurality of heterogeneous processing elements includes an in-order processor core, an out-of-order processor core, and a packed data processor core.

Example 33: The system of Example 32, wherein the plurality of heterogeneous processing elements further comprises an accelerator.

Example 34: The system of any of Examples 31-33, wherein the heterogeneous scheduler further includes a program phase detector that detects a program phase of the code fragment, the plurality of heterogeneous processing elements including a first processing element having a first microarchitecture and a second processing element having a second microarchitecture different from the first microarchitecture, the program phase being one of a plurality of program phases including a first phase and a second phase, the dispatching of instructions being based in part on the detected program phase, and the processing of the code fragment by the first processing element improves performance per watt characteristics compared to the processing of the code fragment by the second processing element.

Example 35: The system of any of Examples 31-34, wherein the heterogeneous scheduler further comprises a selector for selecting a processing element type for the plurality of processing elements to execute the received code fragment, and using dispatch to schedule the code fragment to a processing element of the selected type among the plurality of processing elements.

Example 36: The system of any of Examples 31-35, wherein the code fragment is one or more instructions associated with a software thread.

Example 37: The system of any of Examples 34-36, wherein for a data parallel program phase, the selected type of processing element is a processing core that executes single instruction multiple data (SIMD) instructions.

Example 38: The system of any of Examples 34-37, wherein for a data parallel program phase, the selected type of processing element is a circuit that supports dense arithmetic primitives.

Example 39: The system of any of Examples 34-38, wherein for a data parallel program phase, the selected type of processing element is an accelerator.

Example 40: The system of any of Examples 34-39, wherein the data parallel program phases have data elements that are processed simultaneously using the same control flow.

Example 41: A system as described in any of Examples 30-35, wherein for a thread-parallel program phase, the selected type of processing element is a scalar processing core.

Example 42: A system as described in any of Examples 30-36, in which a thread-parallel program phase has a data-dependent branch with a unique control flow.

Example 43: A system as described in any of Examples 30-37, wherein in the serial program phase, the selected type of processing element is an out-of-order core.

Example 44: The system of any of Examples 30-38, wherein for a data parallel program phase, the selected type of processing element is a processing core that executes single instruction multiple data (SIMD) instructions.

Example 45: A system as described in any of Examples 31-44, in which the heterogeneous scheduler supports multiple code types including compiled, intrinsic, assembly, library, intermediate, offload and device.

Example 46: A system according to any of Examples 31-45, in which the heterogeneous scheduler emulates functionality if a processing element of a selected type cannot process the code fragment natively.

Example 47: A system according to any of Examples 31-46, in which the heterogeneous scheduler emulates functionality when the number of available hardware threads is oversubscribed.

Example 48: A system according to any of Examples 31-47, in which the heterogeneous scheduler emulates functionality if a processing element of a selected type cannot process the code fragment natively.

Example 50: A system as described in any of Examples 31-49, in which the selection of the processing element type for a plurality of heterogeneous processing elements is transparent to a user.

Example 51: A system as described in any of Examples 31-50, in which the selection of the processing element type for the plurality of heterogeneous processing elements is transparent to the operating system.

Example 52: A system as described in any of Examples 31-51, in which the heterogeneous scheduler presents a homogeneous programming model so that each thread appears to the programmer as if it is running on a scalar core.

Example 53: A system as described in any of Examples 52, wherein the presented homogeneous multiprocessor programming model presents emergent support for the complete instruction set.

Example 54a: A system as described in any of Examples 31-53, in which multiple heterogeneous processing elements share a memory address space.

Example 54b: A system as described in any of Examples 31-53, wherein the heterogeneous scheduler includes a binary translator executing on one of the plurality of heterogeneous processing elements.

Example 55: A system as described in any of Examples 31-54, in which the default selection of processing element type for multiple heterogeneous processing elements is a latency optimized core.

Example 56: A system according to any of Examples 31-55, in which the heterogeneous software scheduler selects a protocol to use in the multi-protocol interface for a dispatched instruction.

Example 57: A system as described in any of Examples 56, wherein the first protocol supported by the multi-protocol bus interface comprises a memory interface protocol used to access a system memory address space.

Example 58: A system as described in any of Examples 56-57, wherein the second protocol supported by the multi-protocol bus interface comprises a cache coherency protocol that maintains coherency between data stored in the accelerator's local memory and the host processor's memory subsystem, including the host cache hierarchy and the system memory.

Example 59: A system as described in any of Examples 56-58, wherein the third protocol supported by the multiprotocol bus interface includes a serial link protocol supporting device discovery, register access, configuration, initialization, interrupts, direct memory access, and address translation services.

Example 60: The system of Example 59, wherein the third protocol is a Peripheral Component Interface Express (PCIe) protocol.

Example 61: A method comprising: receiving a plurality of instructions; and dispatching the received plurality of instructions for execution on one or more of a plurality of heterogeneous processing elements, where the received plurality of instructions correspond to a code fragment to be processed by one or more of the plurality of heterogeneous processing elements, such that the plurality of instructions are native instructions for at least one of the one or more of the plurality of heterogeneous processing elements.

Example 62: The method of example 61, wherein the plurality of heterogeneous processing elements includes an in-order processor core, an out-of-order processor core, and a packed data processor core.

Example 63: The method of example 62, wherein the plurality of heterogeneous processing elements further comprises an accelerator.

Example 64: The method of any of Examples 61-63, further comprising detecting a program phase of the code fragment, the plurality of heterogeneous processing elements including a first processing element having a first microarchitecture and a second processing element having a second microarchitecture different from the first microarchitecture, the program phase being one of a plurality of program phases including a first phase and a second phase, and processing of the code fragment by the first processing element improves performance per watt characteristics compared to processing of the code fragment by the second processing element.

Example 65: The method of any of Examples 61-64, further comprising selecting a processing element type for the plurality of processing elements for executing the received code fragment, and scheduling the code fragment to a processing element of the selected type among the plurality of processing elements.

Example 66: The method of any of Examples 61-63, wherein the code fragment is one or more instructions associated with a software thread.

Example 67: The method of any of Examples 64-66, wherein for a data parallel program phase, the selected type of processing element is a processing core that executes single instruction multiple data (SIMD) instructions.

Example 68: The method of any of Examples 64-66, in which, for a data parallel program phase, the selected type of processing element is a circuit that supports dense arithmetic primitives.

Example 69: The method of any of Examples 64-68, in which, for a data parallel program phase, the selected type of processing element is an accelerator.

Example 70: The method of any of Examples 64-69, wherein a data-parallel program phase is characterized by data elements that are processed simultaneously using the same control flow.

Example 71: The method of any of Examples 64-70, in which, for a thread-parallel program phase, the selected type of processing element is a scalar processing core.

Example 72: The method of any of Examples 64-71, in which a thread-parallel program phase is characterized by data-dependent branching with unique control flow.

Example 73: In the case of a serial program phase, the selected type of processing element is an out-of-order core, as described in any of Examples 64-72.

Example 74: The method of any of Examples 64-73, wherein for a data parallel program phase, the selected type of processing element is a processing core that executes single instruction multiple data (SIMD) instructions.

Example 75: The method of any of Examples 61-74, further comprising emulating the functionality if the selected type of processing element cannot process the code fragment natively.

Example 76: The method of any of Examples 61-74, further comprising emulating functionality if the number of available hardware threads is oversubscribed.

Example 77: A method according to any of Examples 61-76, in which the selection of the processing element type for a plurality of heterogeneous processing elements is transparent to a user.

Example 78: A method according to any of examples 61-77, in which the selection of the processing element type for a plurality of heterogeneous processing elements is transparent to the operating system.

Example 79: The method of any of Examples 61-74, further comprising presenting a homogeneous multiprocessor programming model such that each thread appears to be executing on a scalar core.

Example 80: The method of example 79, wherein the presented homogeneous multiprocessor programming model presents the emergence of support for a complete instruction set.

Example 81: A method according to any of examples 61-79, in which multiple heterogeneous processing elements share a memory address space.

Example 82: The method of any of Examples 61-81, further comprising a step of binary converting a code fragment to be executed on one of the plurality of heterogeneous processing elements.

Example 83: The method of any of Examples 61-82, in which the default selection of processing element type for multiple heterogeneous processing elements is a latency optimized core.

Example 84: A non-transitory machine-readable medium storing instructions that, when executed by hardware, cause a processor to perform a method according to one of examples 51-83.

Example 85: A method including receiving a code fragment in a heterogeneous scheduler; determining whether the code fragment is in a parallel phase; if the code fragment is not in a parallel phase, selecting a latency-sensitive operation element to execute the code fragment; if the code fragment is in a parallel phase, determining a type of parallelism and, for a thread-parallel code fragment, selecting a scalar processing element to execute the code fragment; for a data-parallel code fragment, determining a data layout for the data-parallel code fragment; for a packed data layout, selecting one of a single instruction multiple data (SIMD) processing element and an arithmetic primitive processing element, and for a random data layout, selecting one of a gather instruction, a spatial computation array, or a SIMD processing element using one scalar core from an array of multiple scalar cores; and sending the code fragment to the processing element for execution.

Example 86: The method of example 85, further comprising, prior to the step of determining whether the code fragment is in a parallel phase, determining when the code fragment is eligible for offloading to an accelerator, and sending the code fragment to the accelerator when the code fragment is eligible for offloading.

Example 87: The method of any of Examples 85-86, wherein determining whether the code fragment is in a parallel phase is based on one or more of detected data dependencies, instruction types, and control flow instructions.

Example 88: The method of example 87, in which the instruction type for single instruction, multiple data instructions indicates a parallel phase.

Example 89: A method as described in any of Examples 85-88, in which each operating system thread processed by the heterogeneous scheduler is assigned a logical thread identifier.

Example 90: The method of example 89, in which the heterogeneous scheduler utilizes striped mapping of logical thread identifiers such that each logical thread identifier is mapped to a tuple consisting of a processing element type, a processing element identifier, and a thread identifier.

Example 91: The method of example 90, in which the mapping from logical thread identifiers to processing element identifiers and thread identifiers is calculated using division and modulo.

Example 92: The method of example 91, in which the mapping from logical thread identifiers to processing element identifiers and thread identifiers is fixed to preserve thread commonality.

Example 93: The method of example 90, wherein the mapping from logical thread identifiers to processing element types is performed by a heterogeneous scheduler.

Example 94: The method of example 93, in which the mapping from logical thread identifiers to processing element types is flexible to accommodate future processing element types.

Example 95: The method of example 91, in which the heterogeneous scheduler utilizes multiple core groups such that at least one of the multiple core groups has at least one out-of-order tuple and scalars and SIMD tuples whose logical thread identifiers map to the same out-of-order tuple.

Example 96: The method of example 95, in which a non-parallel phase is determined by a thread having a page directory base register value that is unique among threads belonging to one of a plurality of core groups.

Example 97: The method of example 96, in which threads belonging to a process share the same address space, page table, and page directory base register value.

Example 98: The method of any of Examples 85-97, further comprising detecting an event, the event being one of a thread wake-up command, a write to a page directory base register, a sleep command, a phase change of a thread, and one or more instructions indicating a desired reallocation to a different core.

Example 99: The method of Example 98, further comprising the steps of: if the event is a thread wakeup command, determining that the code fragment is in a parallel phase, counting the number of processing elements that share the same page table base pointer as the woken up thread, and determining whether the number of counted processing elements is greater than one, where if the count of the number of processing elements that share the same page table base pointer as the woken up thread is one, the thread is in a serial phase, and if the count of the number of processing elements that share the same page table base pointer as the woken up thread is not one, the thread is in a parallel phase.

Example 100: The method of example 98, further comprising, if the event is a thread sleep command, clearing an execution flag associated with the thread, counting the number of threads of the processing element that share the same page table base pointer as the affected thread, and determining whether the out-of-order processing element is idle, where if the page table base pointer is shared by exactly one thread in the core group, the sharing thread is moved out of the out-of-order processing element, and if the page table base pointer is shared by more than one thread, the first executing thread of the core group is migrated to the out-of-order processing element.

Example 101: The method of example 100, wherein the thread sleep command is one of a stop, wait entry, and timeout or pause command.

Example 102: The method of example 98, further comprising, if the event is a phase change, migrating the thread to a SIMD processing element if the thread is executing on a scalar processing element and the logical thread identifier of the thread indicates that there are SIMD instructions, and migrating the thread to a scalar processing element if the thread is executing on a SIMD processing element and the logical thread identifier of the thread indicates that there are no SIMD instructions.

Example 103: The method of any of Examples 85-102, further comprising transforming the code fragment to better suit the selected processing element before transmitting the code fragment.

Example 104: The method of example 103, wherein the heterogeneous scheduler includes a binary translator that performs the translation.

Example 105: The method of example 103, wherein the heterogeneous scheduler includes a JIT compiler that performs the transformation.

Example 106: The method of any of Examples 85-105, further comprising the method steps of any of the method examples for Examples 61-83.

Example 107: A system including a plurality of heterogeneous processing elements and a heterogeneous scheduler that determines a phase of a code fragment and sends the code fragment to one of the plurality of heterogeneous processing elements for execution based at least in part on the determined phase.

Example 108: The system of Example 107, in which the heterogeneous scheduler determines whether the code fragment is in a parallel phase, selects a latency-sensitive operation element to execute the code fragment if the code fragment is not in a parallel phase, determines a type of parallelism if the code fragment is in a parallel phase, and for a thread-parallel code fragment, selects a scalar processing element to execute the code fragment, for a data-parallel code fragment, determines a data layout for the data-parallel code fragment, selects one of a single instruction multiple data (SIMD) processing element and an arithmetic primitive processing element for a packed data layout, and for a random data layout, selects one of a SIMD processing element using a gather instruction, a spatial computation array, or one scalar core from an array of multiple scalar cores.

Example 109: The system of Example 108, wherein the heterogeneous scheduler further determines when the code fragment is eligible for offloading to an accelerator before determining whether the code fragment is in a parallel phase, and sends the code fragment to the accelerator when the code fragment is eligible for offloading.

Example 110: The system of any of Examples 108-109, wherein the heterogeneous scheduler further determines whether the code fragment is in a parallel phase based on one or more of the detected data dependencies, instruction types, and control flow instructions.

Example 111: The system of example 110, in which the instruction type for single instruction, multiple data instructions indicates parallel phase.

Example 112: A system as described in any of Examples 108-111, in which each operating system thread processed by the heterogeneous scheduler is assigned a logical thread identifier.

Example 113: The system of Example 112, in which the heterogeneous scheduler utilizes striped mapping of logical thread identifiers such that each logical thread identifier is mapped to a tuple consisting of a processing element type, a processing element identifier, and a thread identifier.

Example 114: The system of example 112, wherein the mapping from logical thread identifiers to processing element identifiers and thread identifiers is calculated using division and modulo.

Example 115: The system of Example 114, in which the mapping from logical thread identifiers to processing element identifiers and thread identifiers is fixed to preserve thread commonality.

Example 116: The system of Example 115, wherein the mapping from logical thread identifiers to processing element types is performed by a heterogeneous scheduler.

Example 117: The system of Example 116, in which the mapping from logical thread identifiers to processing element types is flexible to accommodate future processing element types.

Example 118: A system according to any of Examples 108-117, in which the heterogeneous scheduler utilizes core groups such that the core groups have at least one out-of-order tuple and scalars and SIMD tuples whose logical thread identifiers map to the same out-of-order tuple.

Example 119: The system of Example 118, in which a non-parallel phase is determined by a thread having a page directory base register value that is unique among threads belonging to one of a plurality of core groups.

Example 120: The system described in Example 119, in which threads belonging to a process share the same address space, page table, and page directory base register value.

Example 121: The system of any of Examples 108-120, wherein the heterogeneous scheduler detects an event, the event being one of a thread wakeup command, a write to a page directory base register, a sleep command, a thread phase change, and one or more instructions indicating a desired reallocation.

Example 122: The system of Example 121, in which the heterogeneous scheduler determines that the code fragment is in a parallel phase if the event is a thread wakeup command, counts the number of processing elements that share the same page table base pointer as the woken up thread, and determines whether the counted number of processing elements is greater than one, and if the count of the number of processing elements that share the same page table base pointer as the woken up thread is one, the thread is in a serial phase, and if the count of the number of processing elements that share the same page table base pointer as the woken up thread is not one, the thread is in a parallel phase.

Example 123: The system of Example 121, in which the heterogeneous scheduler clears the execution flag associated with the thread if the event is a thread sleep command, counts the number of threads of the processing element that share the same page table base pointer as the affected thread, determines whether the out-of-order processing element is idle, and if the page table base pointer is shared by exactly one thread in the core group, the sharing thread is moved out of the out-of-order processing element, and if the page table base pointer is shared by more than one thread, the first executing thread of the group is migrated to the out-of-order processing element.

Example 124: The system of Example 123, wherein the thread sleep command is one of a stop, wait entry, and timeout or pause command.

Example 125: The system of Example 121, in which the heterogeneous scheduler migrates a thread to a SIMD processing element if the event is a phase change, if the thread is running on a scalar processing element and the logical thread identifier of the thread indicates that there is a SIMD instruction, and migrates the thread to a scalar processing element if the thread is running on a SIMD processing element and the logical thread identifier of the thread indicates that there is no SIMD instruction.

Example 126: A system as described in any of Examples 108-125, in which the heterogeneous scheduler transforms the code fragment to better suit the selected processing element before transmitting the code fragment.

Example 127: The system of Example 126, wherein the heterogeneous scheduler includes a binary translator stored in a non-transitory machine-readable medium to perform the translation when executed.

Example 128: The system of Example 126, wherein the heterogeneous scheduler includes a JIT compiler stored on a non-transitory machine-readable medium to perform the transformation when executed.

Example 129: The system of any of Examples 108-128, further comprising a memory storing program code executable by at least one of the plurality of heterogeneous processing elements in a heterogeneous processor that provides a heterogeneous scheduler.

Example 130: A system as described in any of Examples 108-128, in which the heterogeneous scheduler has circuitry.

Example 131: A processor including a processor core, the processor core including a decoder that decodes at least one instruction native to the processor core, and one or more execution units that execute the at least one decoded instruction, the at least one decoded instruction corresponding to an acceleration start instruction, the acceleration start instruction indicating the start of a region of code to be offloaded to an accelerator.

Example 132: The processor of Example 131, wherein a region of code is offloaded based on whether a target accelerator is coupled to a processor core and available to process the region of code, and if a target accelerator is not coupled to a processor core that processes the region of code, the region of code is processed by the processor core.

Example 133: The processor of Example 131, in which the processor core transitions from a first mode of execution to a second mode of execution in response to execution of at least one decoded instruction corresponding to an acceleration start instruction.

Example 134: The processor of example 133, wherein in a first mode of execution, the processor core checks for self-modifying code, and in a second mode of execution, the processor core disables checking for self-modifying code.

Example 135: The processor of Example 134, wherein the self-modifying code detection circuit is disabled to disable the self-modifying code check.

Example 136: A processor according to any one of Examples 133-135, in which in a first mode of execution, memory consistency model restrictions are relaxed by relaxing memory ordering requirements.

Example 137: A processor according to any one of Examples 133-136, wherein in a first mode of execution, floating point semantics are changed by setting a floating point control word register.

Example 138: A method including: decoding instructions native to a processor core; and executing the decoded instructions corresponding to an acceleration start instruction, the acceleration start instruction indicating the start of a region of code to be offloaded to an accelerator.

Example 139: The method of example 138, wherein the region of code is offloaded based on whether a target accelerator is coupled to a processor core and available to process the region of code, and if the target accelerator is not coupled to the processor core that processes the region of code, the region of code is processed by the processor core.

Example 140: The method of example 138, in which the processor core transitions from a first mode of execution to a second mode of execution in response to execution of a decoded instruction corresponding to the acceleration start instruction.

Example 141: The method of example 140, in which in a first mode of execution, the processor core checks for self-modifying code and in a second mode of execution, the processor core disables checking for self-modifying code.

Example 142: The method of example 141, in which the self-modifying code detection circuit is disabled to disable the self-modifying code check.

Example 143: The method of any one of Examples 140-142, in which in a first mode of execution, memory consistency model restrictions are relaxed by relaxing memory ordering requirements.

Example 144: The method of any one of Examples 140-143, wherein in a first mode of execution, floating point semantics are changed by setting a floating point control word register.

Example 145: A non-transitory machine-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method, the method including: decoding instructions native to a processor core; and executing the decoded instructions corresponding to an acceleration start instruction, the acceleration start instruction indicating the start of a region of code to be offloaded to an accelerator.

Example 146: The method of example 145, in which the region of code is offloaded based on whether a target accelerator is coupled to a processor core and available to process the region of code, and if the target accelerator is not coupled to a processor core that processes the region of code, the region of code is processed by the processor core.

Example 147: The method of example 145, in which the processor core transitions from a first mode of execution to a second mode of execution in response to execution of a decoded instruction corresponding to the acceleration start instruction.

Example 148: The method of example 147, in which in a first mode of execution, the processor core checks for self-modifying code and in a second mode of execution, the processor core disables checking for self-modifying code.

Example 149: The method of example 148, in which the self-modifying code detection circuit is disabled to disable the self-modifying code check.

Example 150: The method of any one of Examples 148-149, in which in a first mode of execution, memory consistency model restrictions are relaxed by relaxing memory ordering requirements.

Example 151: The method of any one of Examples 148-150, wherein in a first mode of execution, floating point semantics are changed by setting a floating point control word register.

Example 152: A system including a processor core, the processor core including a decoder that decodes at least one instruction native to the processor core, and one or more execution units that execute the at least one decoded instruction, the at least one decoded instruction corresponding to an acceleration start instruction, the acceleration start instruction indicating the start of a region of code to be offloaded to an accelerator.

Example 153: The system of Example 152, in which the region of code is offloaded based on whether a target accelerator is coupled to a processor core and available to process the region of code, and if the target accelerator is not coupled to the processor core that processes the region of code, the region of code is processed by the processor core.

Example 154: The system of Example 152, in which the processor core transitions from a first mode of execution to a second mode of execution in response to execution of at least one decoded instruction corresponding to the acceleration start instruction.

Example 155: The system of example 154, wherein in a first mode of execution, the processor core checks for self-modifying code, and in a second mode of execution, the processor core disables checking for self-modifying code.

Example 156: The system of example 155, in which the self-modifying code detection circuit is disabled to disable the self-modifying code check.

Example 157: The processor of any one of Examples 152-156, wherein in a first mode of execution, memory consistency model restrictions are relaxed by relaxing memory ordering requirements.

Example 158: A processor according to any one of Examples 152-157, wherein in a first mode of execution, floating point semantics are changed by setting a floating point control word register.

Example 159: A processor including a processor core, the processor core including a decoder that decodes instructions native to the processor core, and one or more execution units that execute decoded instructions corresponding to an acceleration end instruction, the acceleration end instruction indicating the end of a region of code to be offloaded to an accelerator.

Example 160: The processor of example 159, wherein a region of code is offloaded based on whether a target accelerator is coupled to a processor core and available to process the region of code, and if a target accelerator is not coupled to a processor core that receives and processes the region of code, the region of code is processed by the processor core.

Example 161: The processor of example 159, wherein the region of code is described by execution of decoded instructions corresponding to an acceleration start instruction that transitions the processor core from a first mode of execution to a second mode of execution.

Example 162: The processor of example 161, wherein in a first mode of execution, the processor checks for self-modifying code, and in a second mode of execution, the processor disables checking for self-modifying code.

Example 163: The processor of Example 162, in which the self-modifying code detection circuit is disabled to disable the self-modifying code check.

Example 164: A processor according to any one of Examples 161-163, in which in a first mode of execution, memory consistency model restrictions are relaxed.

Example 165: A processor according to any one of Examples 161-164, wherein in a first mode of execution, floating point semantics are changed by setting a floating point control word register.

Example 166: A processor according to any one of Examples 159-165, in which execution of an accelerator start instruction gates execution of a region of code on a processor core until an accelerator end instruction is executed.

Example 167: A method including: decoding instructions native to a processor core; and executing the decoded instructions corresponding to an acceleration end instruction, the acceleration end instruction indicating an end of a region of code to be offloaded to an accelerator.

Example 168: The method of example 167, in which the region of code is offloaded based on whether a target accelerator is coupled to a processor core and available to process the region of code, and if a target accelerator is not coupled to a processor core that receives and processes the region of code, the region of code is processed by the processor core.

Example 169: The method of example 167, in which the region of code is described by execution of decoded instructions corresponding to an acceleration start instruction that transitions the processor core from a first mode of execution to a second mode of execution.

Example 170: The method of example 169, in which in a first mode of execution, the processor checks for self-modifying code and in a second mode of execution, the processor disables checking for self-modifying code.

Example 171: The method of example 170, in which the self-modifying code detection circuit is disabled to disable the self-modifying code check.

Example 172: The method of any one of Examples 169-171, in which in a first mode of execution, memory consistency model restrictions are relaxed.

Example 173: The method of any one of Examples 169-172, wherein in a first mode of execution, floating point semantics are changed by setting a floating point control word register.

Example 174: The method of any one of Examples 167-173, wherein execution of an accelerator start instruction gates execution of a region of code on a processor core until an accelerator end instruction is executed.

Example 175: A non-transitory machine-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method, the method including: decoding instructions native to a processor core; and executing the decoded instructions corresponding to an acceleration end instruction, the acceleration end instruction indicating an end of a region of code to be offloaded to an accelerator.

Example 176: The non-transitory machine-readable medium of Example 175, wherein the region of code is offloaded based on whether a target accelerator is coupled to a processor core and available to process the region of code, and if a target accelerator is not coupled to a processor core that receives and processes the region of code, the region of code is processed by the processor core.

Example 177: The non-transitory machine-readable medium of Example 175, in which a region of code is described by execution of decoded instructions corresponding to an acceleration start instruction that transitions a processor core from a first mode of execution to a second mode of execution.

Example 178: The non-transitory machine-readable medium of example 177, wherein in a first mode of execution, the processor checks for self-modifying code and in a second mode of execution, the processor disables checking for self-modifying code.

Example 179: The non-transitory machine-readable medium of Example 178, wherein the self-modifying code detection circuit is disabled to disable the self-modifying code check.

Example 180: The non-transitory machine-readable medium of any one of Examples 177-179, in which in a first mode of execution, memory consistency model restrictions are relaxed.

Example 181: The non-transitory machine-readable medium of any one of Examples 177-180, wherein in a first mode of execution, floating point semantics are changed by setting a floating point control word register.

Example 182: The non-transitory machine-readable medium of any one of Examples 175-181, wherein execution of an accelerator start instruction gates execution of a region of code on a processor core until an accelerator end instruction is executed.

Example 183: A system including a processor core, the processor core including a decoder that decodes instructions native to the processor core, one or more execution units that execute decoded instructions corresponding to an accelerated end instruction, the accelerated end instruction indicating an end of a region of code that is offloaded to an accelerator, and an accelerator that executes the offloaded instructions.

Example 184: The system of Example 183, in which the region of code is offloaded based on whether a target accelerator is coupled to a processor core and available to process the region of code, and if a target accelerator is not coupled to a processor core that receives and processes the region of code, the region of code is processed by the processor core.

Example 185: The system of Example 184, wherein the region of code is described by execution of decoded instructions corresponding to an acceleration start instruction that transitions the processor core from a first mode of execution to a second mode of execution.

Example 186: The system of example 185, in which in a first mode of execution, the processor checks for self-modifying code and in a second mode of execution, the processor disables checking for self-modifying code.

Example 187: The system of Example 186, in which the self-modifying code detection circuit is disabled to disable the self-modifying code check.

Example 188: A system as described in any one of Examples 185-187, in which in a first mode of execution, memory consistency model restrictions are relaxed.

Example 189: The system of any one of Examples 185-188, wherein in a first mode of execution, floating point semantics are changed by setting a floating point control word register.

Example 190: The system of any one of Examples 183-190, wherein execution of an accelerator start instruction gates execution of a region of code on a processor core until an accelerator end instruction is executed.

Example 191: A system including an accelerator that executes threads.

The system includes a processor core and a memory having software stored therein that implements a heterogeneous scheduler, which, when executed by the processor core, detects code sequences in threads suitable for possible execution on an accelerator, selects an accelerator for executing the detected code sequence, and transmits the detected code sequence to the selected accelerator.

Example 192: The system of Example 191, further comprising a plurality of heterogeneous processing elements that execute program phases of threads that are not suitable for execution by the accelerator.

Example 193: The system of any of Examples 191-192, wherein the heterogeneous scheduler further includes a pattern matcher that recognizes code sequences by comparing the code sequences to a predetermined set of patterns.

Example 194: The system of Example 193, wherein the predetermined set of patterns is stored in memory.

Example 195: The system of any of Examples 191-194, wherein the heterogeneous scheduler recognizes code having a pattern match and uses performance monitoring to adjust an operating mode associated with a thread by configuring a processor core to do one or more of the following: ignore self-modifying code is ignored, relax memory consistency model restrictions, change floating point semantics, change performance monitoring, and change utilization of architectural flags.

Example 196: The system of any of Examples 191-195, wherein the heterogeneous scheduler further includes a conversion module that converts the recognized code into accelerator code for execution on the accelerator.

Example 197: A system according to any of Examples 191-196, wherein the processor core has pattern matching circuitry that detects code sequences within a thread using stored patterns.

Example 198: A system according to any of Examples 191-197, in which the processor core maintains the execution status of each thread executing in the system.

Example 199: A system as described in any of Examples 191-197, in which the heterogeneous scheduler maintains the status of each thread executing in the system.

Example 200: A system as described in any of Examples 191-199, in which the heterogeneous scheduler selects an accelerator based on one or more of processor element information, tracked threads, and detected code sequences.

Example 201: A system including a plurality of heterogeneous processing elements and a heterogeneous scheduler circuit coupled to the plurality of processing elements, the heterogeneous scheduler circuit including a thread and processing element tracker table that maintains an execution status of each executing thread and each processing element, and a selector that selects a processing element type for the plurality of heterogeneous processing elements that processes a code fragment and schedules the code fragment on one of the plurality of heterogeneous processing elements for execution based on the status and processing element information from the thread and processing element trackers.

Example 202: The system of example 201 further includes a memory storing software executable by the processor core, the software detecting code sequences in a thread for possible execution on an accelerator, the accelerator being one of a plurality of heterogeneous processing elements coupled to the heterogeneous scheduler circuit.

Example 203: The system of example 202, in which a software pattern matcher recognizes code sequences from stored patterns.

Example 204: A system as described in any of Examples 201-203, in which the heterogeneous scheduler converts the recognized code into accelerator code.

Example 205: A system according to any of Examples 201-204, in which the selector is a finite state machine executed by a heterogeneous scheduler circuit.

Example 206: A method including executing a thread, detecting a pattern in the executing thread, converting the recognized pattern into accelerator code, and forwarding the converted pattern to an available accelerator for execution.

Example 207: The method of example 206, wherein the pattern is recognized using a software pattern matcher.

Example 208: The method of example 206, in which the pattern is recognized using a hardware pattern matching circuit.

Example 209: A method including executing a thread, detecting a pattern in the executing thread, and adjusting an operating mode associated with the thread to employ mitigation requests based on the pattern.

Example 210: The method of example 209, wherein the pattern is recognized using a software pattern matcher.

Example 211: The method of example 209, in which the pattern is recognized using a hardware pattern matching circuit.

Example 212: The method of example 209, in which in the adjusted mode of operation, one or more of the following are applied: self-modifying code is ignored; memory consistency model restrictions are relaxed; floating point semantics are altered; performance monitoring is altered; and architectural flag usage is altered.

Example 213: A system including a decoder that decodes instructions native to a processor core, and one or more execution units that execute the decoded instructions, where one or more of the decoded instructions correspond to an accelerated start instruction, and the accelerated start instruction causes entry into a different mode of execution for instructions that follow the accelerated start instruction within the same thread.

Example 214: The system of Example 213, wherein the acceleration start command includes a field that specifies a pointer to a memory data block, and the format of the memory data block includes a sequence number field that indicates progress before the interrupt.

Example 215: A system as described in any of Examples 213-214, wherein the acceleration start instruction includes a block class identifier field that specifies a predefined transformation of code stored in memory.

Example 216: A system as described in any of Examples 213-215, wherein the acceleration start command includes an implementation identifier field indicating the type of hardware used for execution.

Example 217: A system as described in any of Examples 213-216, in which the acceleration start instruction includes a saved state area size field indicating the size and format of a state save area that stores registers that are modified after the acceleration start instruction executes.

Example 218: The system of any of Examples 213-217, wherein the acceleration start instruction includes a field for a local storage area size, and the local storage area provides storage beyond registers.

Example 219: The system of example 218, wherein the local storage area size is specified by an immediate operand of the acceleration start instruction.

Example 220: The system of example 218, wherein the local storage area is not accessed except for instructions following the start acceleration instruction.

Example 221: A system as described in any of Examples 213-220, in which memory dependency types are definable for instructions in different modes of execution.

Example 222: The system of Example 221, in which the definable memory dependency types include one of an independent type, in which store-load and store-store dependencies are guaranteed not to exist, a potentially dependent access type to a local storage area, in which loads and stores to a local storage area may depend on each other but are independent of other loads and stores, a potentially dependent type, in which hardware dynamically checks and enforces dependencies between instructions, and an atomic type, in which loads and stores are dependent between themselves and memory is updated atomically.

Example 223: The system of any of Examples 213-222, further comprising a memory for storing saved state including registers to be used, flags to be updated, implementation specific information, and local storage for use during execution beyond register.

Example 224: A system as described in Example 223, in which each instance of parallel execution gets its own local storage.

Example 225: A method including entering a different relaxation mode of execution for a thread, writing registers used during execution of the thread to a saved state area during execution of the different relaxation modes, reserving local storage for use by each parallel execution within the thread during execution of the different relaxation modes, executing a block of threads to track instructions in the different relaxation modes of execution, determining whether an end of the different mode of execution has been reached based on execution of an accelerator exit instruction, restoring registers and flags from the saved state area to their original state if the end of the different mode of execution has been reached, and updating the local storage with intermediate results if the end of the different mode of execution has not been reached.

Example 226: The method of example 225, in which during different relaxed mode execution, one or more of self-modifying code is ignored, memory consistency model restrictions are relaxed, floating point semantics are altered, performance monitoring is altered, and architectural flag usage is altered.

Example 227: The method of example 225 or 226, entering a different mode of execution based on execution of an accelerator start instruction.

Example 228: The method of example 225, entering different modes of execution based on the determined pattern.

Example 229: The method of any of Examples 225-228, in which the size and format of a state save area that stores registers that are modified after an accelerator start instruction executes is specified in a memory block pointed to by the accelerator start instruction.

Example 230: The method of any of Examples 225-229, further comprising transforming the thread or a portion thereof prior to execution.

Example 231: The method of example 230, in which the thread or a portion thereof is converted into accelerator code.

Example 232: The method of example 230 or 231, in which the transformed thread or a portion of the transformed thread is executed by an accelerator.

Example 233: The method of any of Examples 213-232, in which instructions of a block are tracked by updating a sequence number in a memory block associated with the block of a thread.

Example 234: A method according to any of Examples 223-233, in which the sequence number of a block of threads is updated when an instruction is successfully executed and retired.

Example 235: A method as in any of Examples 223-234, in which an accelerator exit instruction executes and retires, but does not reach the end of a different mode of execution.

Example 236: The method of any of Examples 223-235, attempting to execute a portion of a block using intermediate results if the end of a different mode of execution has not been reached as determined by accelerator end instruction execution.

Example 237: The method of example 236, wherein the non-accelerator processing element is used to execute with the intermediate results after an exception or interrupt.

Example 238: A method according to any of Examples 223-237, where if the end of a different mode of execution has not been reached, the execution is rolled back to the point where the use of the accelerator began.

Example 239: A system including a decoder that decodes an instruction having an opcode, a field for a first packed data source operand, one or more fields for the second through Nth packed data source operands, and a field for a packed data destination operand, and an execution circuit that executes the decoded instruction to, for each packed data element position of the second through Nth packed data source operands, 1) multiply a data element at that packed data element position of the packed data source operand by a data element at a corresponding packed data element position of the first packed data source operand to generate a temporary result, 2) sum the temporary results, 3) add the sum of the temporary result to a data element at the corresponding packed data element position of the packed data destination operand, and 4) store the sum of the temporary result for the data element at the corresponding packed data element position of the packed data destination operand in the corresponding packed data element position of the packed data destination operand.

Example 240: The system described in Example 239, where N is indicated by an opcode.

Example 241: A system as described in any of Examples 239-240, in which the value of the source operand is copied to a register of the multiplier-adder array.

Example 242: A system as described in any of Examples 239-241, wherein the execution circuitry includes a binary tree reduction network.

Example 243: A system as described in any of Examples 242, wherein the execution circuitry is part of an accelerator.

Example 244: The system of Example 242, wherein the binary tree reduction network has a plurality of multiplier circuits coupled in pairs to a first set of adder circuits, the first set of adder circuits coupled to a second set of adder circuits coupled to a third set of adder circuits that are also coupled to data elements in corresponding packed data element positions of the packed data destination operand.

Example 245: The system described in Example 244, in which each multiplication is processed in parallel.

Example 246: A system according to any of Examples 239-245, in which the packed data elements correspond to elements of one or more matrices.

Example 247: A method comprising: decoding an instruction having an opcode, a field for a first packed data source operand, one or more fields for the second through Nth packed data source operands, and a field for a packed data destination operand; and executing the decoded instruction to, for each packed data element position of the second through Nth packed data source operands, 1) multiply a data element for that packed data element position of the packed data source operand by a data element for a corresponding packed data element position of the first packed data source operand to generate a temporary result and sum the temporary results; 3) add the sum of the temporary results to a data element for the corresponding packed data element position of the packed data destination operand; and 4) store the sum of the temporary results for the data element for the corresponding packed data element position of the packed data destination operand in the corresponding packed data element position of the packed data destination operand.

Example 248: The method of example 247, where N is indicated by an opcode.

Example 249: The method of any of Examples 247-248, in which the value of the source operand is copied to a register of the multiplier-adder array.

Example 250: The method of any of Examples 247-249, wherein the implementation circuitry includes a binary tree reduction network.

Example 251: The method of example 247, wherein the binary tree reduction network has a plurality of multiplier circuits coupled in pairs to a first set of adder circuits, the first set of adder circuits coupled to a second set of adder circuits coupled to a third set of adder circuits that are also coupled to data elements in corresponding packed data element positions of the packed data destination operand.

Example 252: The method of example 251, wherein each packed data operand has eight packed data elements.

Example 253: The method of example 251, in which each multiplication is processed in parallel.

Example 254: A method according to any of Examples 247-253, in which the packed data elements correspond to elements of one or more matrices.

Example 255: A non-transitory machine-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method, the method including: decoding an instruction having an opcode, a field for a first packed data source operand, one or more fields for the second through Nth packed data source operands, and a field for a packed data destination operand; and executing the decoded instructions to, for each packed data element position of the second through Nth packed data source operands, 1) multiply a data element for that packed data element position of the packed data source operand by a data element for a corresponding packed data element position of the first packed data source operand to generate a temporary result; 2) sum the temporary results; 3) add the sum of the temporary result to a data element for the corresponding packed data element position of the packed data destination operand; and 4) store the sum of the temporary result for the data element for the corresponding packed data element position of the packed data destination operand in the corresponding packed data element position of the packed data destination operand.

Example 256: A non-transitory machine-readable medium as described in Example 255, wherein N is indicated by an opcode.

Example 257: The non-transitory machine-readable medium of any of Examples 255-256, wherein the value of the source operand is copied to a register of a multiplier-adder array.

Example 258: A non-transitory machine-readable medium according to any of Examples 255-257, wherein the execution circuitry includes a binary tree reduction network.

Example 259: The non-transitory machine-readable medium of Example 258, wherein the binary tree reduction network has a plurality of multiplier circuits coupled in pairs to a first set of adder circuits, the first set of adder circuits coupled to a second set of adder circuits coupled to a third set of adder circuits that are also coupled to data elements at corresponding packed data element positions of a packed data destination operand.

Example 260: The non-transitory machine-readable medium of example 259, wherein each packed data operand has eight packed data elements.

Example 261: A non-transitory machine-readable medium as described in example 259, wherein each multiplication is processed in parallel.

Example 262: A non-transitory machine-readable medium according to any of Examples 255-261, in which packed data elements correspond to elements of one or more matrices.

Example 263: A method comprising: decoding an instruction having an opcode, a field for a first packed data source operand, one or more fields for second through Nth packed data source register operands, and a field for a packed data destination operand; and executing the decoded instruction to, for each packed data element position of the second through Nth packed data source operands, 1) multiply a data element for that packed data element position of the packed data source operand by a data element for a corresponding packed data element position of the first packed data source operand to generate a temporary result; 2) sum the temporary results; 3) add the sum of the temporary results to a data element for the corresponding packed data element position of the packed data destination operand; and 4) store the sum of the temporary result for the data element for the corresponding packed data element position of the packed data destination operand.

Example 264: The method of example 263, where N is indicated by an opcode.

Example 265: The method of any of Examples 263-264, in which the value of the source operand is copied to a register of the multiplier-adder array.

Example 266: The method of example 265, wherein the execution circuit is a binary tree reduction network.

Example 267: The method of example 266, wherein the binary tree reduction network has a plurality of multiplier circuits coupled in pairs to a first set of adder circuits, the first set of adder circuits coupled to a second set of adder circuits coupled to a third set of adder circuits that are also coupled to data elements in corresponding packed data element positions of the packed data destination operand.

Example 268: A method as described in any of Examples 263-267, wherein each packed data operand has eight packed data elements.

Example 269: A method according to any of Examples 268-268, in which each multiplication is processed in parallel.

Example 270: A non-transitory machine-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method, the method including: decoding an instruction having an opcode, a field for a first packed data source operand, one or more fields for second through Nth packed data source register operands, and a field for a packed data destination operand; and executing the decoded instructions to, for each packed data element position of the second through Nth packed data source operands, 1) multiply a data element for that packed data element position of the packed data source operand by a data element for a corresponding packed data element position of the first packed data source operand to generate a temporary result; 2) sum the temporary results; 3) add the sum of the temporary result to a data element for the corresponding packed data element position of the packed data destination operand; and 4) store the sum of the temporary result for the data element for the corresponding packed data element position of the packed data destination operand.

Example 271: A non-transitory machine-readable medium as described in Example 270, where N is indicated by an opcode.

Example 272: The non-transitory machine-readable medium of any of Examples 270-271, wherein the value of the source operand is copied to a register of a multiplier-adder array.

Example 273: The non-transitory machine-readable medium of Example 272, wherein the execution circuit is a binary tree reduction network.

Example 274: The non-transitory machine-readable medium of Example 272, wherein the binary tree reduction network has a plurality of multiplier circuits coupled in pairs to a first set of adder circuits, the first set of adder circuits coupled to a second set of adder circuits coupled to a third set of adder circuits that are also coupled to data elements in corresponding packed data element positions of a packed data destination operand.

Example 275: A non-transitory machine-readable medium according to any of Examples 270-274, wherein each packed data operand has eight packed data elements.

Example 276: A non-transitory machine-readable medium according to any of Examples 270-275, wherein each multiplication is processed in parallel.

Example 277: A system including a decoder for decoding an instruction having an opcode, a field for a first packed data source operand, one or more fields for the second through Nth packed data source register operands, and a field for a packed data destination operand, and an execution circuit for executing the decoded instruction to, for each packed data element position of the second through Nth packed data source operands, 1) multiply a data element at that packed data element position of the packed data source operand by a data element at a corresponding packed data element position of the first packed data source operand to generate a temporary result, 2) sum the temporary results in pairs, 3) add the sum of the temporary result to a data element at the corresponding packed data element position of the packed data destination operand, and 4) store the sum of the temporary result for the data element at the corresponding packed data element position of the packed data destination operand in the corresponding packed data element position of the packed data destination operand.

Example 278: The system described in Example 277, where N is indicated by an opcode.

Example 279: A system as described in any of Examples 277-278, in which the value of the source operand is copied to a register of the multiplier-adder array.

Example 280: The system described in Example 279, wherein the execution circuit is a binary tree reduction network.

Example 281: The system of Example 279, wherein the binary tree reduction network has a plurality of multiplier circuits coupled in pairs to a first set of adder circuits, the first set of adder circuits coupled to a second set of adder circuits coupled to a third set of adder circuits that are also coupled to data elements in corresponding packed data element positions of the packed data destination operand.

Example 282: A system as described in any of Examples 277-281, wherein each packed data operand has eight packed data elements.

Example 283: A system as described in any of Examples 277-282, in which each multiplication is processed in parallel.

Example 284: A system including an accelerator including a multi-protocol bus interface coupling the accelerator to a host processor, the accelerator including one or more processing elements for processing commands; a shared work queue including a plurality of entries for storing work descriptors submitted by a plurality of clients, the work descriptor including an identification code identifying a client that submitted the work descriptor, at least one command to be executed by the one or more processing elements, and addressing information; and an arbiter that dispatches the work descriptors from the shared work queue to one or more processing elements according to a particular arbitration policy, each of the one or more processing elements receiving the work descriptors dispatched from the arbiter, performing source and destination address translations, reading source data identified by the source address translation, executing at least one command to generate destination data, and writing the destination data to a memory using the destination address translation.

Example 285: The system of Example 284, wherein the multiple clients include one or more of user mode applications that directly submit user mode input/output (IO) requests to the accelerator, kernel mode drivers that execute in virtual machines (VMs) that share the accelerator, and/or software agents that execute in multiple containers.

Example 286: The system of example 285, wherein at least one of the multiple clients has a user-mode application or container running within a VM.

Example 287: The system of any of Examples 284-286, wherein the client has one or more of a peer input/output (IO) agent and/or a software chain offload request.

Example 288: The system of Example 287, wherein at least one of the peer IO agents has a network interface controller (NIC).

Example 289: The system of any of Examples 284-288, further including an address translation cache that stores virtual-to-physical address translations usable by one or more processing elements.

Example 290: A system as described in any of Examples 284-289, in which the particular arbitration policy is a first-in, first-out policy.

Example 291: The system of any of Examples 284-290, wherein the particular arbitration policy comprises a quality of service (QoS) policy in which a work descriptor of a first client is given priority over a work descriptor of a second client.

Example 292: The system of example 291, in which a work descriptor of a first client is dispatched to one or more processing elements before a work descriptor of a second client, even if the work descriptor of the second client is received in the shared work queue before the work descriptor of the first client.

Example 293: A system as described in any of Examples 284-292, wherein the identification code includes a process address space identifier (PASID) that identifies an address space in system memory that is allocated to the client.

Example 294: The system of any of Examples 284-293, further including one or more dedicated work queues, each dedicated work queue including a plurality of entries storing work descriptors submitted by a single client associated with the dedicated work queue.

Example 295: The system of Example 294, further including a group configuration register programmed to combine two or more of the dedicated and/or shared work queues in a group, the group being associated with one or more of the plurality of processing elements.

Example 296: The system of example 295, wherein one or more processing elements process work descriptors from dedicated work queues and/or shared work queues within the group.

Example 297: A system as described in any of Examples 284-296, wherein a first protocol supported by the multi-protocol bus interface includes a memory interface protocol used to access a system memory address space.

Example 298: A system as described in any of Examples 284-297, wherein the second protocol supported by the multi-protocol bus interface comprises a cache coherency protocol that maintains coherency between data stored in the accelerator's local memory and the host processor's memory subsystem, including the host cache hierarchy and the system memory.

Example 299: The system of any of Examples 284-298, wherein the third protocol supported by the multiprotocol bus interface includes a serial link protocol supporting device discovery, register access, configuration, initialization, interrupts, direct memory access, and address translation services.

Example 300: The system described in Example 299, wherein the third protocol has a Peripheral Component Interface Express (PCIe) protocol.

Example 301: The system of any of Examples 284-300, further including an accelerator memory that stores source data to be processed by the processing elements and stores destination data resulting from processing by one or more processing elements.

Example 302: The system of example 301, wherein the accelerator memory comprises high bandwidth memory (HBM).

Example 303: The system of example 301, wherein the accelerator memory is allocated in a first portion of the system memory address space used by the host processor.

Example 304: The system of example 303, further including host memory allocated to a second portion of the system memory address space.

Example 305: The system of example 304, further comprising bias circuitry and/or logic for indicating, for each block of data stored in the system memory address space, whether the data contained within the block is biased towards the accelerator.

Example 306: The system of example 305, wherein each block of data has a memory page.

Example 307: The system of example 305, wherein the host refrains from processing data that is biased toward the accelerator without first sending a request to the accelerator.

Example 308: The system of example 307, wherein the bias circuitry and/or logic includes a bias table that includes one bit that is set for each fixed-size block of data to indicate a bias toward the accelerator.

Example 309: A system as described in any of Examples 301-308, in which the accelerator has a memory controller in communication with a coherency controller of a host processor that executes one or more data coherent transactions associated with data stored in the accelerator memory.

Example 310: The system of example 309, in which the memory controller operates in a device bias mode to access blocks of data stored in the accelerator memory that is set to a bias directed toward the accelerator, and when in the device bias mode, the memory controller accesses the accelerator memory without consulting a cache coherence controller of the host processor.

Example 311: The system of example 309, wherein the memory controller operates in a host bias mode to access blocks of data that are biased toward the host processor, and when in the host bias mode, the memory controller sends all requests to the accelerator memory through a cache coherence controller in the host processor.

Example 312: A system as described in any of examples 284-311, wherein the shared work queue stores at least one batch descriptor that identifies a batch of work descriptors.

Example 313: The system of Example 312, further comprising a batch processing circuit for processing the batch descriptors by reading the batch of work descriptors from the memory.

Example 314: The system of example 292, wherein a work descriptor is added to a dedicated work queue corresponding to a host processor executing a first type of instruction, and a work descriptor is added to a shared work queue corresponding to a host processor executing a second type of instruction.

Example 315: A method including arranging a first set of memory pages with a device bias; allocating the first set of memory pages from a local memory of an accelerator device coupled to a host processor; transferring operand data from a core or an input/output agent of the host processor to the allocated pages; processing the operands by the accelerator device using the local memory to generate results; and converting the first set of memory pages from the device bias to the host bias.

Example 316: The method of example 315, wherein placing the first set of memory pages in the device bias includes updating the first set of memory pages in a bias table to indicate that the pages are in the accelerator device bias.

Example 317: The method of any of Examples 315-316, wherein updating the entry includes setting a bit associated with each page in the first set of memory pages.

Example 318: A method as described in any of examples 315-317, in which when set to device bias, the first set of memory pages is guaranteed not to be cached in the host cache memory.

Example 319: The method of any of Examples 315-318, wherein allocating the first set of memory pages includes initiating a driver or application programming interface (API) call.

Example 320: The method of any of Examples 315-319, in which, to process the operands, the accelerator device executes commands to process data directly from its local memory.

Example 321: The method of any of Examples 315-320, wherein transferring operand data to the allocated pages includes submitting one or more work descriptors to the accelerator device, the work descriptors identifying or including the operands.

Example 322: The method of example 321, in which one or more work descriptors may cause allocated pages to be flushed from the host processor cache upon command.

Example 323: The method of any of Examples 315-323, wherein the host processor is permitted to access the results, cache the results, and share the results if the first set of memory pages is set to host bias.

Claims (14)

Accelerator and
a local memory having a plurality of stacked DRAM dies;
a silicon bridge coupling the accelerator to the plurality of stacked DRAM dies;
Equipped with
a connection between the accelerator and the plurality of stacked DRAM dies through the silicon bridge;
The accelerator comprises:
a plurality of processing elements for performing tasks assigned to them by an external processor;
a cache coherent interface coupling the accelerator to the external processor, the cache coherent interface ensuring that data stored in the local memory and/or accelerator cache is coherent with data stored in the external processor's cache and system memory;
logic for mapping a virtual memory space onto a heterogeneous physical system memory including said local memory and said system memory;
having
the accelerator and the external processor each access corresponding portions of the local memory and the system memory using the virtual memory space;
the accelerator maintains a table for tracking memory portions that are biased to the external processor;
The table has an entry for each fixed-size memory block, the entry being set to be biased to the external processor or not . The apparatus of claim 1, wherein the cache coherent interface provides cache coherent connectivity between the accelerator and a core of the external processor. The device of claim 1 or 2, wherein the cache coherent interface performs snooping to detect the state of cache lines in the external processor's cache. The apparatus of any one of claims 1 to 3, wherein the cache coherent interface provides snoop updates in response to accesses and attempts to modify cache lines by the multiple processing elements. the accelerator accesses the table from the local memory or an accelerator cache;
5. Apparatus according to claim 1, wherein the table indicates which memory portions are associated with the external processor. The apparatus of any one of claims 1 to 5, wherein the table includes a table containing a plurality of entries, each entry indicating whether a corresponding memory portion is associated with the external processor. The apparatus of claim 6, wherein each entry has at least one bit programmed to a first value to identify the accelerator and to a second value to identify the external processor. The apparatus of claim 7, wherein when the at least one bit is programmed to the first value, the accelerator biases a memory page associated with a corresponding entry to the accelerator, and when the at least one bit is programmed to the second value, the accelerator biases the memory page to the external processor. The apparatus of claim 8, wherein to bias the memory page to the accelerator, the accelerator is provided with access to the memory page from the local memory without first communicating with the external processor. The apparatus of any one of claims 1 to 9, wherein the accelerator has a scheduler for scheduling operations on the plurality of processing elements. The apparatus of any one of claims 1 to 10, wherein each of the plurality of processing elements includes hardware support for column- and row-wise matrix processing. The apparatus of any one of claims 1 to 11, wherein the multiple processing elements process different combinations of sparse matrices, dense matrices, sparse vectors, and dense vectors. The apparatus of any one of claims 1 to 12, wherein the plurality of processing elements perform a matrix multiplication operation using matrix data elements. The device of any one of claims 1 to 13, wherein the accelerator executes a single instruction to perform a two-dimensional matrix multiplication operation.

Images