JP7492511B2 - Streaming Platform Flow and Architecture - Google Patents

Description

Reservation of Rights for Copyrighted Material A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the U.S. Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

The present disclosure relates to integrated circuits (ICs), and more particularly to using data streams for communication between a host system and a hardware accelerated circuit and for communication between kernel circuits of the hardware accelerated circuit.

Hardware acceleration refers to implementing the functionality of a portion of program code in hardware or circuitry. Hardware accelerated program code is functionally equivalent to the original program code. Instead of running a compiled version of the program code, such as an executable binary using a processor, the program code is implemented as a circuit configured to provide the same functionality as the executable binary. The hardware accelerated version of the program code usually provides improved performance compared to running the program code using any processor. In some cases, the program code is compiled into a circuit design that is implemented within a programmable IC.

These and other implementations may each optionally include one or more of the following features, alone or in combination, or may include all of the following features.

In one or more embodiments, a system includes a host system and an IC coupled to the host system through a communications interface. The IC is configured for hardware acceleration. The IC includes a direct memory access circuit coupled to the communications interface, a kernel circuit, and a stream traffic manager circuit coupled to the direct memory access circuit and the kernel circuit. The stream traffic manager circuit is configured to control data streams exchanged between the host system and the kernel circuit.

In one aspect, the host system and the IC communicate by exchanging packetized data.

In another aspect, the IC includes interconnect circuitry connecting the stream traffic manager circuitry and the kernel circuitry.

In another aspect, the kernel circuit is one of a plurality of kernel circuits, and the stream traffic manager circuit is configured to interleave the data streams provided to the plurality of kernel circuits.

In another aspect, the IC includes an input buffer coupled to the interconnect circuitry and the kernel circuitry, the input buffer configured to temporarily hold packetized data from the stream traffic manager circuitry and convert the packetized data into a data stream provided to the kernel circuitry. The IC further includes an output buffer coupled to the interconnect circuitry and the kernel circuitry, the output buffer configured to temporarily hold a data stream output from the kernel circuitry and convert the data stream into packetized data.

In another aspect, a host system includes a processor coupled to a memory, the processor configured to implement in the memory a write queue corresponding to the input buffer and a read queue corresponding to the output buffer. The write queue stores a descriptor specifying data to be streamed to the input buffer. The read queue stores a descriptor specifying data to be streamed from the output buffer to a memory of the host system.

In another aspect, the host system is configured to send packetized data with in-band instructions to the kernel circuitry.

In one or more embodiments, the method includes using computer hardware to select a container file that includes a configuration bitstream that specifies a kernel circuit and metadata about the kernel circuit; using computer hardware to extract the configuration bitstream from the container file and load the configuration bitstream into the IC to implement the kernel circuit in the IC; and using computer hardware to determine pipe properties from the metadata, the pipe properties specifying settings for streaming data from the host system to the kernel circuit. The method may also include implementing a direct data transfer from the host system to the kernel circuit as packetized data that is converted into a data stream and provided to the kernel circuit using the settings specified by the pipe properties.

In one aspect, the method includes implementing a further data transfer from the kernel circuitry directly to the host system as a further data stream specifying the results.

In another aspect, implementing the further data transfer includes determining whether a write queue corresponding to the kernel circuit, located in the host system, has space to receive a complete packet of data, and in response to determining that the write queue has space, initiating a data transfer from the kernel circuit to the host system.

In another aspect, the method includes sending settings to a stream traffic manager circuit in the IC, the stream traffic manager circuit implementing the settings to stream data between the host system and the kernel circuit.

In another aspect, the method includes including instructions for the kernel circuitry in-band within the data stream.

In another aspect, the method includes determining that the data transfer should be implemented as a data stream based on a data type used by a user application requesting the data transfer.

In another aspect, implementing the data transfer includes determining whether an input buffer coupled to a kernel circuit in the IC has space to receive the data, and in response to determining that the input buffer has space, initiating a data transfer to the kernel circuit.

In one or more embodiments, an IC includes a communications interface coupled to a host system, a direct memory access circuit coupled to the communications interface, a kernel circuit implemented using programmable circuits, and a stream traffic manager circuit coupled to the direct memory access circuit and the kernel circuit. The stream traffic manager circuit is configured to control data streams exchanged between the host system and the kernel circuit.

In one aspect, the IC includes a first interconnect configured to receive packetized data from the stream traffic manager circuit and deliver the packetized data to the kernel circuit, and a second interconnect configured to receive data from the kernel circuit and provide the data to the stream traffic manager circuit.

In another aspect, the IC includes an input buffer coupled to an output port of the first interconnect and to an input port of the kernel circuit, the input buffer configured to temporarily store packetized data, convert the packetized data into a data stream, and provide the data stream to the kernel circuit. In response to the input buffer determining that it has available space, the stream traffic manager circuit initiates a data transfer to the kernel circuit.

In another aspect, the IC includes an output buffer coupled to an output port of the kernel circuit and an input port of the stream traffic manager circuit, the output buffer configured to temporarily store a data stream output from the kernel circuit, convert the data stream into packetized data, and provide the packetized data to a second interconnect. The stream traffic manager circuit initiates a data transfer from the kernel circuit to the host system in response to determining that a buffer in the host system corresponding to the output buffer has available space and that the output buffer contains at least one complete packet.

In another aspect, the kernel circuit is one of a plurality of kernel circuits implemented in a programmable circuit. The stream traffic manager circuit is coupled to each of the plurality of kernel circuits and configured to interleave data streams exchanged with the plurality of kernel circuits.

In another aspect, each kernel circuit of the plurality of kernel circuits is coupled to a stream traffic manager circuit through a buffer and an interconnect. The stream traffic manager circuit implements a round-robin arbitration scheme for streaming data to each of the plurality of kernel circuits based on space availability in a buffer corresponding to each respective kernel circuit.

In one or more embodiments, an IC includes a first kernel circuit implemented in a programmable circuit, a second kernel circuit implemented in the programmable circuit, and a stream traffic manager circuit coupled to the first kernel circuit and the second kernel circuit. The stream traffic manager circuit is configured to control data streams exchanged between the first kernel circuit and the second kernel circuit.

In one aspect, the selected data stream sent from the first kernel circuit to the second kernel circuit includes in-band instructions for the second kernel circuit.

In another aspect, the first kernel circuit is coupled to the first interconnect through a first input buffer and a first output buffer, the second kernel circuit is coupled to the second interconnect through a second input buffer and a second output buffer, and the first interconnect and the second interconnect are coupled to a stream traffic manager.

In another aspect, the stream traffic manager circuit is configured to provide a selected data stream from a host system coupled to the integrated circuit to the first kernel circuit or directly to the second kernel circuit, and provide a resulting data stream from the first kernel circuit or the second kernel circuit to the host system.

In another aspect, the selected data stream includes in-band instructions for the first kernel circuit or the second kernel circuit.

In another aspect, the first kernel circuit is located on a first die of the integrated circuit and the second kernel circuit is located on a second die of the integrated circuit.

In another aspect, the stream traffic manager circuitry is located on the first die.

In another aspect, the IC includes an input buffer coupled to an input port of a second kernel circuit in the second die and configured to temporarily store data streamed to the second kernel circuit, and an output buffer coupled to an output port of a first kernel circuit in the first die and configured to temporarily store data output from the first kernel circuit. The stream traffic manager circuit is configured to initiate a data transfer from the first kernel circuit to the second kernel circuit in response to the input buffer determining that the input buffer has available space and that the output buffer is storing data.

In another aspect, the IC includes an input buffer coupled to an input port of a first kernel circuit in a first die and configured to temporarily store data streamed to the first kernel circuit, and an output buffer coupled to an output port of a second kernel circuit in a second die and configured to temporarily store data output from the second kernel circuit. The stream traffic manager circuit is configured to initiate a data transfer from the second kernel circuit to the first kernel circuit in response to the input buffer determining that the input buffer has available space and that the output buffer is storing data.

In one or more embodiments, a system includes a first IC having a first plurality of kernel circuits, a stream traffic manager circuit configured to control data streams exchanged between different kernel circuits of the first plurality of kernel circuits, and a first transceiver, and a second IC having a second plurality of kernel circuits, a satellite stream traffic manager circuit configured to control data streams exchanged between different kernel circuits of the second plurality of kernel circuits, and a second transceiver coupled to the first transceiver. The stream traffic manager circuit and the satellite stream traffic manager circuit are configured to exchange data streams passed between selected kernel circuits of the first plurality of kernel circuits and selected kernel circuits of the second plurality of kernel circuits.

In one aspect, the first plurality of kernel circuits are located on different dies of the first IC and the second plurality of kernel circuits are located on different dies of the second IC.

In another aspect, the data stream exchanged between a selected kernel circuit of the first plurality of kernel circuits and a selected kernel circuit of the second plurality of kernel circuits includes in-band instructions for the second kernel circuit.

In another aspect, the stream traffic manager circuit is configured to provide a selected data stream directly from a host system coupled to the first IC to a selected kernel circuit of the first plurality of kernel circuits or a selected kernel circuit of the second plurality of kernel circuits, and provide a resulting data stream from the selected kernel circuit of the first plurality of kernel circuits or the selected kernel circuit of the second plurality of kernel circuits to the host system.

In another aspect, the selected data stream includes in-band instructions for the first kernel circuit or the second kernel circuit.

In another aspect, the first IC includes an interconnect coupled to the stream traffic manager and the first plurality of kernel circuits, and the second IC includes an interconnect coupled to the satellite stream traffic manager and the second plurality of kernel circuits.

In another aspect, the stream traffic manager circuit and the satellite stream traffic manager circuit are configured to exchange data streams in response to determining that the input buffer of the receiving kernel circuit has available space.

In another aspect, the first IC includes a first plurality of dies, with the first plurality of kernel circuits distributed across the first plurality of dies. Each die includes an interconnect coupled to a stream traffic manager and a particular kernel circuit of the first plurality of kernel circuits in the die.

In one or more embodiments, the method includes monitoring, by a stream traffic manager circuitry, an output buffer of a kernel circuit for a packet, the kernel circuit being implemented in a programmable circuit of at least one IC; determining, by the stream traffic manager circuitry, in response to detecting that the output buffer of the sending kernel circuitry stores the packet, a receiving kernel circuit for the packet; determining, by the stream traffic manager circuitry, whether an input buffer of the receiving kernel circuitry has space available to store the packet; and in response to determining that the input buffer has space available to store the packet, initiating, by the stream traffic manager circuitry, a stream data transfer from the output buffer of the sending kernel circuitry to the input buffer of the receiving kernel circuit.

In one aspect, the stream data transfer is performed without the involvement of the host system.

In another aspect, the stream data transfer includes in-band instructions that control the operation of the receiving kernel circuitry.

This summary section is provided merely to introduce some concepts and is not intended to identify key or essential features of the claimed subject matter. Other features of the inventive subject matter will be apparent from the accompanying drawings and the detailed description that follows.

Configurations of the present invention are illustrated by way of example in the accompanying drawings. However, the drawings should not be construed as limiting the configurations of the present invention to only the particular implementations shown. Various aspects and advantages will become apparent upon review of the following detailed description and upon reference to the drawings.

FIG. 1 illustrates an exemplary architecture for hardware acceleration. FIG. 2 illustrates another exemplary implementation of the architecture of FIG. 1 . FIG. 2 illustrates an exemplary method for transferring data between a host system and kernel circuitry of a hardware accelerator using a data stream. FIG. 1 illustrates an exemplary architecture for exchanging data between kernel circuits using data streams. FIG. 1 illustrates an exemplary method for exchanging data between kernel circuits using data streams. FIG. 1 illustrates an exemplary system for use with one or more embodiments described herein. FIG. 1 illustrates an exemplary architecture for an IC.

Although the present disclosure concludes with claims defining novel features, it is believed that the various features described within the present disclosure will be better understood by considering the description in conjunction with the drawings. The process(es), machine(s), product(s) and any variations thereof described herein are provided for illustrative purposes. The specific structural and functional details described within the present disclosure should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to employ the described features in various ways in nearly all appropriately detailed structures. Moreover, the terms and phrases used within the present disclosure are not limiting, but rather provide an understandable description of the described features.

The present disclosure relates to ICs, and more particularly to using data streams for communication between a host system and a hardware accelerated circuit and for communication between kernel circuits of the hardware accelerated circuit. The IC implements the hardware accelerated circuit as one or more kernel circuits. For example, each kernel circuit represents hardware accelerated program code. The host system can offload one or more tasks to the kernel circuits implemented in the IC. In offloading one or more tasks to the kernel circuits, the host system transfers data to be operated on by the kernel circuits using an architecture that supports data streams. The kernel circuits can exchange data with each other using a data stream-enabled architecture. The kernel circuits also transfer data, e.g., results, to the host system as a packetized data stream before sending them to the host system.

In conventional systems, when offloading a task to a kernel circuit, the host system initiates a data transfer to the kernel circuit via a random access memory (RAM) coupled to the IC that implements the kernel circuit. However, the RAM is located on the same circuit board (e.g., accelerator card) as the kernel circuit, but not in the same IC. Once the data is transferred to the RAM, the host system notifies the kernel circuit that the data is ready for use. This means that the kernel circuit cannot begin to operate on the data until the data transfer to the RAM is complete. Instructions provided from the host system to the kernel circuit are provided separately, e.g., out-of-band, to the data. For example, commands are provided to the kernel circuit over a different physical interface than that used to convey the data.

In a conventional system, when the kernel circuit is notified of data availability, it reads the data from RAM, processes the data, and writes the results to RAM. When the kernel circuit has finished writing the results to RAM, it notifies the host system of the availability of the results. The host system then retrieves the results from RAM.

According to the inventive configurations described within this disclosure, data is exchanged between the host system and the kernel circuit using data streams and packetization. Data originating from the host system is sent directly to the kernel circuit. Similarly, data originating from the kernel circuit is sent directly to the host system. As an illustrative and non-limiting example, data transfer from the host system to the kernel circuit flows directly from the host system to the kernel circuit. Data transferred from the host system is not first stored and accumulated in off-chip RAM and then read by the kernel circuit. Similarly, results transferred from the kernel circuit to the host system are not first stored and accumulated in off-chip RAM before being provided to the host system. Instead, data flows directly from the kernel circuit to the host system. Streaming is implemented over a data path within the IC that utilizes one or more smaller internal memory buffers. The memory buffers are, for example, smaller in size than the amount of data exchanged between the host system and the kernel circuit.

The streaming architecture described within this disclosure facilitates faster data transfers, less latency, and more efficient use of memory compared to conventional systems. For example, a kernel circuit can begin operating on less than the entire data immediately after receiving the data, rather than waiting for the entire data to first be transferred to off-chip RAM and then loaded into the kernel circuit. This improves the speed and latency of the overall system. Similar gains in speed and latency are obtained by streaming data from the kernel circuit to the host system. With a streaming architecture, commands from the host system to the kernel circuit are included within the data stream itself, e.g., in-band, which further reduces system latency. By utilizing the IC's internal memory more efficiently, less off-chip RAM is needed, which reduces the power requirements of the system and/or hardware accelerators.

In certain embodiments, the kernel circuits may also communicate with each other using data streams. Also, the benefits described with respect to host system-kernel circuit communication are achieved by using data streams for communication between the kernel circuits. Furthermore, by including a streaming infrastructure within the programmable IC(s), the kernel circuits may exchange data with each other using a less complex infrastructure, e.g., an infrastructure that does not require a direct point-to-point communication link between the kernel circuits that are intended to communicate with each other.

Further aspects of the configuration of the present invention are described in more detail below with reference to the figures. For simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, dimensions of some of the elements may be exaggerated relative to other elements for clarity. Furthermore, where considered appropriate, reference numerals have been repeated among the figures to indicate corresponding, similar or similar features.

FIG. 1 illustrates an exemplary architecture 100 for hardware acceleration. Architecture 100 includes a host system 102 and a hardware accelerator 103. Host system 102 is implemented as a computer system, such as a server or other data processing system. Hardware accelerator 103 is implemented as a circuit board having an IC 104 and memory 106 attached to IC 104. For example, hardware accelerator 103 may be implemented as an accelerator card having an edge connector that may be inserted into an available peripheral slot of host system 102.

1 is described using memory (e.g., RAM) external to the IC 104, the embodiments described herein relating to streaming data are also valid and applicable when the IC 104 includes sufficient on-chip memory such that memory 106 is not required. When the IC 104 includes sufficient on-chip or same-die memory, problems similar to those involving external memory arise when the entire data must be transferred to memory before the kernel circuitry is permitted to operate on the data. Using internal memory is faster than using external memory, but introduces problems such as increased latency, increased storage capacity (memory) requirements, and synchronization that are overcome by the streaming-enabled embodiments described herein.

In one or more embodiments, the IC 104 is implemented as a programmable IC. In certain embodiments, the IC 104 is implemented using the same or similar architecture as described with respect to FIG. 7. In the example of FIG. 1, the IC 104 includes an endpoint 108, a direct memory access circuit (DMA) 110, a kernel circuit 112, and a memory controller 114. The endpoint 108 is an interface capable of communicating with the host system 102 over a communication bus. As an illustrative and non-limiting example, the communication bus may be implemented as a Peripheral Component Interconnect Express (PCIe) bus. Thus, the endpoint 108 may be implemented as a PCIe endpoint. However, it should be appreciated that other communication buses may be used and that the examples provided are not limiting. Thus, the endpoint 108 may be implemented as any of a variety of suitable interfaces for communicating over a communication bus.

The endpoint 108 is coupled to the DMA 110. The DMA 110 is also coupled to the kernel circuit 112 and to the memory controller 114 (abbreviated as "MC" in FIG. 1). In a particular embodiment, the DMA 110 includes two independent channels supporting bidirectional communication with the endpoint 108 and with the kernel circuit 112. In the example of FIG. 1, the DMA 110 is coupled to the kernel circuit 112 through one or more interfaces 116. Thus, the host system 102 can transfer data to the kernel circuit 112 via the endpoint 108 and the DMA 110 as packetized data that is converted into one or more data streams before being provided to the kernel circuit 112. Similarly, the kernel circuit 112 can transfer data to the host system 102 by outputting a data stream that is packetized before being provided to the host system 102 via the DMA 110 and the endpoint 108. Further details related to the transfer of data are described in more detail with respect to FIG. 2. Generally, a data stream, whether originating at the host system 102 or from the kernel circuit 112, is converted into multiple packets, although depending on the size of the data stream (e.g., if the data stream conveys a smaller amount of data), there may be cases where the data stream is converted into a single packet.

One example of the interface 116 is a stream-enabled on-chip interconnect, such as the Advanced Microcontroller Bus Architecture (AMBA®) Advanced Extensible Interface (AXI). The AXI stream interconnect allows for the connection of heterogeneous master/slave AMBA® AXI stream protocol compliant circuit blocks. The interface 116 is capable of routing connections conveying packetized data from one or more masters to one or more slaves. AXI is provided for illustrative purposes and not for limitation. It should be appreciated that the interface 116 may be implemented as any of a variety of interconnects. For example, the interface 116 may be implemented as a bus, a network-on-chip (NoC), a crossbar, a switch, or other type of interconnect.

In one or more embodiments, the memory controller 114 is coupled to the memory 106. The memory 106 is implemented as a RAM. The memory controller 114 may be multi-ported and is coupled to the DMA 110 and the kernel circuit 112. The memory controller 114 is capable of accessing (e.g., reading and/or writing) the memory 106 under the control of the DMA 110 and/or the kernel circuit 112. For example, the DMA 110 is coupled to the memory controller 114 through a memory-mapped interface 118. Similarly, the kernel circuit 112 is coupled to the memory controller 114 through a memory-mapped interface 120. The DMA 110 is coupled to the kernel circuit 112 through a control interface 122. In one or more embodiments, the control interface 122 is implemented as an AXI-Lite interface configured to provide point-to-point bidirectional communication with the circuit block. AXI-Lite may be used as a control interface for the kernel circuit 112. As described, AXI is provided for purposes of illustration and not limitation.

Using the memory-mapped interfaces 118 and 120 and the control interface 122, the architecture shown in FIG. 1 can also support data transfer between the host system 102 and the kernel circuit 112 through the memory 106. For example, the host system 102 sends data to the memory 106. The data can be provided to the DMA 110, which uses the memory controller 114 to store the data in the memory 106. The data is accumulated and stored in the memory 106 until the data transfer is completed, as previously described. The host system 102 can notify the kernel circuit 112 through the control interface 122 of the availability of data in the memory 106. The kernel circuit 112 can access the memory controller 114 to read data from the memory 106. The kernel circuit 112 generates a result and stores the result in the memory 106. The kernel circuit 112 notifies the host system 102 through the control interface 122 of the availability of the result in the memory 106.

In examples where data is transferred to the kernel circuit 112 or multiple kernel circuits implemented in the IC 104 using memory 106, the host system 102 is responsible for allocating and sharing the memory 106 among the various kernel circuits. The host system 102 configures and starts the kernel circuits through a control interface 122. However, the control interface 122 tends to be a slower interface with significant latency. In addition to having to communicate with the kernel circuits through the control interface 122, the host system 102 must also manage and synchronize the kernel circuit operations, adding significant overhead to the host system 102. The host system 102 must synchronize data transfers with control signals, for example, to start and/or stop the kernel circuits at the appropriate time(s).

As described, in other embodiments, IC 104 includes sufficient memory resources such that memory 106 is implemented as internal memory within IC 104. In that case, the circuit blocks described in IC 104 can access the internal memory using interface circuitry within IC 104, and thus memory controller 114 can be omitted.

In one or more embodiments, the architecture 100 is implemented to support direct communication between the host system 102 and the kernel circuit 112 via packetized data and data streams. In that case, memory-mapped communication capabilities may be omitted. For example, the control interface 122, the memory-mapped interfaces 118 and 120, and the memory controller 114 may be omitted (as may the memory 106). However, in one or more other embodiments, the architecture 100 is implemented to support both memory-mapped communication involving the memory 106 and direct communication using packetized data and data streams. For example, the DMA 110 may support both types of data transfer. Additionally, although a single kernel circuit is shown in the example of FIG. 1, multiple kernel circuits may be implemented, with some kernel circuits utilizing direct communication with the host system 102 via data streams and other kernel circuits utilizing the memory 106 for data transfer with the host system 102. In yet other embodiments, the kernel circuit may be implemented to utilize either direct communication via a data stream or memory 106 for data transfer, depending on the particular application being executed by the host system 102 that is invoking the kernel circuit, or the particular function being invoked by the application to communicate with the kernel circuit.

Architecture 100 and other streaming architectures described herein provide a more efficient way to configure and manage kernel circuits. In certain embodiments, instructions may be provided to the kernel circuits in-band along with the data payload of the data stream. Including instructions with the data, e.g., "putting instructions in-band," eliminates the need for control interface 122 when data streams are used and provides more efficient host system-kernel circuit communication.

The host system 102 may execute a software framework that includes one or more user applications, such as a memory-mapped user application 124 and/or a stream user application 126. The memory-mapped user application 124 is an application executed by the host system 102 that is configured to call kernel circuits, such as the kernel circuit 112, and exchange data with the kernel circuit 112 using the memory-mapped interfaces 118 and 120, the control interface 122, and the memory 106. The stream user application 126 is an application executed by the host system 102 that is configured to call kernel circuits, such as the kernel circuit 112, and exchange data with the kernel circuit 112 using the streaming interface 116.

The software framework also includes a runtime 128. The runtime 128 provides functionality, e.g., an application programming interface (API), for communicating with the IC 104. For example, the runtime 128 may provide functionality for implementing DMA transfers over PCIe. In one or more embodiments, the runtime 128 may provide support for streaming data between the kernel circuit 112 and the host system 102 using the interface 116. In one or more other embodiments, the runtime 128 may provide support for transferring data between the kernel circuit 112 and the host system 102 using the memory 106, the memory-mapped interfaces 118 and 120, and the control interface 122. As an illustrative example, the runtime 128 may support the execution of a memory-mapped user application 124 and the transfer of data to and from the kernel circuit 112 via the memory 106, and/or the execution of a streamed user application 126 and the transfer of data to and from the kernel circuit 112 via the interface 116.

The driver 130 may control an endpoint (not shown) within the host system 102. In the case of a PCIe connection, for example, the endpoint within the host system 102 is implemented as a root complex. The driver 130 may therefore implement and manage multiple read and write queues for storing descriptors that control data transfer between the host system 102 and the IC 104.

In one or more embodiments, the driver 130 can split requests for large data transfers (e.g., streamed data transfers) to a kernel circuit into multiple stream transfers of smaller chunks of data called packets. This splitting of data, or "packetization of data into packets," performed by the driver 130, is largely hidden from the kernel circuit 112. The packetization enables the interconnect fabric implemented in the IC 104 to service multiple kernel circuits concurrently by interleaving packets addressed to and/or from different kernel circuits. The driver 130 can determine the packet size to be large enough to efficiently amortize the packetization overhead, but not so large that the packets cause kernel circuits to stall while waiting their turn to send and/or receive streamed data while other kernel circuits are transferring streamed data.

As generally described, the control interface 122 tends to be a slow connection that requires synchronization between the control signals communicated to the kernel circuit 112 and the data delivered to the kernel circuit 112. For example, if the control interface 122 is used for out-of-band signaling with a data stream, speed and/or synchronization requirements often result in stopping the data stream(s) to the kernel circuit in order to change the control signals before resuming the data stream(s).

As an illustrative and non-limiting example, consider the case where the kernel circuit 112 implements an encryption operation. Different data payloads provided to the kernel circuit 112 generally require different keys for encryption. If the control interface 122 is to be used, the data stream to the kernel circuit 112 would be stopped, the key would be updated via the control interface 112, and then the data stream(s) would be resumed. Such an operation would be coordinated by the host system 102, which increases the overhead of the host system 102. In one or more embodiments described herein, one or more instructions to the kernel circuit 112 are provided in-band. Thus, new and/or updated keys may be included in the in-band data stream provided to the kernel circuit 112. The instructions may be included with or immediately before the payload. In certain embodiments, the instructions may be specified in a custom-defined header for each packet. In this example, the host system 102 may send an encryption key as part of a packet header for the plaintext payload(s) of the packet(s) on which the kernel circuit 112 should operate. Thus, the kernel circuit 112 operates efficiently, in that the kernel circuit 112 does not need to be stopped and/or synchronized with the control interface 122, allowing the host system 102 to switch encryption keys for different payloads without incurring synchronization overhead and with reduced latency compared to conventional techniques for data transfer.

Figure 2 illustrates another exemplary implementation of the architecture 100 of Figure 1. Figure 2 illustrates additional aspects of the architecture 100 that are not shown in the higher level view described with respect to Figure 1. However, for illustrative purposes, some elements shown in Figure 1 are not shown in Figure 2, such as selected elements of the software framework executed by the host system 102, the endpoints 108 and memory controller 114 in the IC 104, and the memory 106.

In the example of FIG. 2, a driver 130 of a software framework executed by the host system 102 is shown. The driver 130 can implement multiple queues 202-1 through 202-8. The driver 130 can create a read queue and a write queue for each kernel circuit implemented in the IC 104. For illustrative purposes, the queues 202 configured as write queues are shaded and the queues 202 configured as read queues are not shaded. Since the IC 104 implements four kernel circuits 234-1, 234-2, 234-3, and 234-4, the driver 130 implements four write queues (e.g., 202-1, 202-3, 202-5, and 202-7) and four read queues (e.g., 202-2, 202-4, 202-6, and 202-8). Each of the queues 202 may store one or more descriptors, each describing a data transfer to be performed. Each descriptor stored in a write queue describes a data transfer from the host system 102 to the kernel circuitry 234, and each descriptor stored in a read queue describes a data transfer from the kernel circuitry 234 to the host system 102.

DMA 110, as mentioned, includes two channels. The write channel supports the transfer of data from host system 102 to kernel circuit 234. The write channel includes write circuit 204 and arbitration circuit 206. Write circuit 204 can store commands and/or data received from host system 102 before forwarding the commands and/or data to kernel circuit 234. The read channel supports the transfer of data from kernel circuit 234 to host system 102. The read channel includes read circuit 208 and arbitration circuit 210. Read circuit 208 can store data received from kernel circuit 234 before forwarding the data to host system 102.

DMA 110 moves data between host memory (not shown) of host system 102 and buffers 218, 220, 222, 224, 226, 228, 230, and 232. DMA 110 fetches and maintains a list of addresses, e.g., descriptors, for every packet to be transferred and forms a sequence of commands and addresses for endpoint 108. In one or more embodiments, DMA 110 is highly configurable. Thus, traffic management and flow control for DMA 110 is implemented through stream traffic manager 212. Stream traffic manager 212 effectively ensures that all kernel circuits 234 have fair access to DMA 110 for data transfer to and from host system 102.

Stream traffic manager 212 is coupled to DMA 110 and interconnects 214 and 216. Stream traffic manager 212 can regulate the flow of data streams/packets between host system 102 and kernel circuitry 234. In the example of FIG. 2, stream traffic manager 212 includes controller 236, one or more buffers 238, one or more data mover engines 240, flow-to-pipe map (map) 242, and pipe-to-route map (map) 244.

In certain embodiments, interconnect 214 and interconnect 216 implement interface 116 of FIG. 1. In the example of FIG. 2, interconnect 214 is configured to receive packetized data from stream traffic manager 212 and route the packetized data to the appropriate kernel circuit 234. Interconnect 216 is configured to receive packetized data from kernel circuit 234 and provide the packetized data to stream traffic manager 212.

In the example of FIG. 2, kernel circuits 234 are connected to interconnects 214 and 216 through buffers. Each of kernel circuits 234 has an input port configured to receive a data stream through a corresponding input buffer and an output port configured to send a data stream through a corresponding output buffer. For illustrative purposes, the input buffers (e.g., buffers 218, 222, 226, and 230) are shaded. The output buffers (e.g., buffers 220, 224, 228, and 232) are not shaded.

Kernel circuit 234-1 is connected to interconnect 214 through buffer 218 and to interconnect 216 through buffer 220. Kernel circuit 234-2 is connected to interconnect 214 through buffer 222 and to interconnect 216 through buffer 224. Kernel circuit 234-3 is connected to interconnect 214 through buffer 226 and to interconnect 216 through buffer 228. Kernel circuit 234-4 is connected to interconnect 214 through buffer 230 and to interconnect 216 through buffer 232.

As mentioned, interconnects 214 and 216 may be implemented as AXI stream interconnects, although the inventive configurations are not so limited. Any of a variety of circuit architectures for delivering packetized data may be used. Other exemplary circuit architectures that may be used to implement interconnects 214 and 216 include, but are not limited to, a crossbar, a multiplexed bus, a mesh network, and/or a network-on-chip (NoC).

Each of the input buffers 218, 222, 226, and 230 is coupled to the interconnect 214 and to an input port of a kernel circuit 234-1, 234-2, 234-3, and 234-4, respectively. Each input buffer can temporarily store packetized data from the host system 102 that is directed to the corresponding kernel circuit 234 if the kernel circuit is not able to immediately accept or process the received data. Additionally, each input buffer can also convert the packetized data received from the host system 102 into a data stream that is provided to the corresponding kernel circuit 234. For example, each input buffer can combine a sequence of one or more packets to generate a data stream that can be provided to the corresponding kernel circuit.

Each of the output buffers 220, 224, 228, and 232 is coupled to the interconnect 216 and to an output port of the kernel circuit 234-1, 234-2, 234-3, and 234-4, respectively. Each output buffer can temporarily hold the data stream output from the corresponding kernel circuit 234, convert the data stream into packetized data, and send the packetized data to the host system 102 via the interconnect 216. Each output buffer can store data when the kernel circuit cannot keep pace with the streaming infrastructure. Each output buffer can, for example, separate the data stream output from the corresponding kernel circuit into one or more packets.

In one or more embodiments, output buffers 220, 224, 228, and 232 can provide kernel tagging information to identify the source kernel circuit and/or the destination kernel circuit. For example, the output buffers can add the tagging information as a pre-pended header. The tagging performed by the output buffers allows the data in the packets to be placed or routed to the correct location in host memory or to the appropriate kernel circuit. For example, each output buffer corresponding to kernel circuit 234 can tag each packet with a source kernel identifier and send those packets out to interconnect 216. Interconnect 216 delivers the packets to stream traffic manager 212 and DMA engine 110. DMA engine 110 moves the packetized data to host memory.

For illustrative purposes, kernel circuit 234-1 will be described. It should be appreciated that kernel circuits 234-2, 234-3, and 234-4 may operate in the same or similar manner. In the example of FIG. 2, an input port of kernel circuit 234-1 is connected to interconnect 214 through buffer 218. An output port of kernel circuit 234-1 is connected to interconnect 216 through buffer 220. For illustrative purposes, write queue 202-1 is mapped to input buffer 218 and read queue 202-2 is mapped to output buffer 220. Generally, each of queues 202 is mapped to one of buffers 218-232. However, buffers 218-232 may be mapped to more than one of queues 202. For purposes of illustration, queues 202-1 and 202-2 correspond to buffers 218 and 220, queues 202-3 and 202-4 correspond to buffers 222 and 224, queues 202-5 and 202-6 correspond to buffers 226 and 228, and queues 202-7 and 202-8 correspond to buffers 230 and 232.

In the example of FIG. 2, the host system 102 executes a user application configured for data streaming. To establish a connection to the kernel circuit 234, the host system 102 creates a pair of queues 202. As an illustrative example, the user application may call a function provided by the runtime 128 that causes the driver 130 to create a pair of queues 202-1 and 202-2 that correspond to the buffers 218 and 220, respectively. Once the pair of queues 202 is created, in this example, the host processor may call further functions to configure control registers (not shown) in the DMA 110 and maps 242 and 244 of the stream traffic manager 212 so that data may be streamed between the host system 102 and the kernel circuit 234-1.

When executing a user application, the host system 102 places descriptors in the queue 202-1 that specify instructions for sending (e.g., writing) data to the kernel circuit 234-1, and appropriately places descriptors in the read queue 202-2 that specify instructions for receiving (e.g., reading) data from the kernel circuit 234-1. In a particular embodiment, the driver 130 can packetize the data to be sent to the IC 104 and inform the DMA 110 of the number of descriptors available in the queue 202 to be fetched. The DMA 110 communicates that information to the stream traffic manager 212.

Stream traffic manager 212 maintains a mapping of queues 202 to buffers 218-232 using maps 242 and 244. Using the stored mappings, stream traffic manager 212 determines that queue 202-1 corresponds to buffer 218 and queue 202-2 corresponds to buffer 220. Controller 236, aware of the available descriptors in queue 202-1, can access buffer 218 for the input port of kernel circuit 234-1. Controller 236 determines whether buffer 218 has space available to receive data, and if so, determines the amount of data that can be received and stored in buffer 218.

In one or more embodiments, the DMA 110 can determine how full each of the queues 202 is and inform the controller 236. The write circuit 204 can, for example, determine the number of descriptors in each of the queues 202-1, 202-3, 202-5, and 202-7. The read circuit 208 can determine the number of descriptors in each of the queues 202-2, 202-4, 202-6, and 202-8. The read circuit 204 and the write circuit 208 can inform the stream traffic manager 212 of the number of descriptors in each queue 202. Additionally, the write circuit 204 and the read circuit 208 can remove descriptors from the queues 202 under the control of the stream traffic manager 212.

Within the stream traffic manager 212, the buffer(s) 238 store the descriptors retrieved from the queues 202 via the DMA 110. For example, the controller 236 can request that the DMA 110 retrieve a certain number of descriptors depending on the amount of space available in the buffer(s) 238. The DMA 110 provides the retrieved descriptors to the stream traffic manager 212. Thus, the stream traffic manager 212 can internally store a subset of the descriptors stored in each of the queues 202 in the buffer(s) 238.

In one or more embodiments, the format or syntax of the descriptor indicates how many descriptors are needed to form a packet and the number of bytes in the packet. In response to determining that the buffer 218 has available space to receive data, the controller 236 evaluates the stored descriptors in the buffer(s) 238 that correspond to the kernel circuit 234-1 (e.g., when a descriptor is retrieved from the queue 202-1) and determines, based on the data in the descriptor(s) themselves, how many descriptors to perform to retrieve a sufficient amount of data (e.g., packet(s)) to store in the buffer 218 without exceeding the available space of the buffer 218.

In one or more embodiments, each of the data mover engines 240 can retrieve data from the host system 102 and send data to the host system 102 via the DMA 110. The data mover engines 240 can operate concurrently. The controller 236 can assign descriptors to be executed to available ones of the data mover engines 240 from the buffer(s) 238. Each data mover engine 240 processes the assigned descriptors by fetching data specified by each of the respective descriptors. For example, the data mover engine 240 can send retrieved packetized data specified by the descriptor(s) to the buffer 218 via the interconnect 214. As mentioned, the input buffer 218 can store the packetized data, convert the packetized data into a data stream, and provide the data stream to the kernel circuit 234-1.

The packet handling capabilities of the stream traffic manager 212 allow packets that may correspond to different data streams to be retrieved in an interleaved manner. Packets may be retrieved from (or sent to) the host system 102 in an interleaved manner for N different data streams.

The stream traffic manager 212 can perform the operations described for each of the kernel circuits 234. Thus, the stream traffic manager 212 can continuously monitor the input buffer for each kernel circuit 234 and initiate data transfer to the buffer only in response to first determining that the input buffer has space to receive and store data. In other words, the controller 236 can continuously determine which descriptors in the queue 202 have corresponding buffers in the IC 104 with sufficient space available and then execute such descriptors.

At a given time, the communication bus connecting the IC 104 and the host system 102 can simultaneously carry multiple descriptors and/or data being fetched. Each of the interconnects 214 and 216 can carry a single packet at a time. In a particular embodiment, the arbitration circuit 206 can implement a round-robin arbitration scheme to pass one packet at a time to the different kernel circuits. In other embodiments, the arbitration circuit 206 may use a different arbitration scheme. Because the stream traffic manager 212 executes the descriptors (initiates a read request) only for kernel circuits 234 that have available space in their input buffers, packets received from the stream arbitration 206 are guaranteed to be passed on the intended input buffer of the target kernel circuit 234 and not have back pressure. Because the space in the input buffer was pre-allocated, space is guaranteed to receive the packetized data.

The stream traffic manager 212 may further instruct the DMA 110 to fetch data in an interleaved fashion. As an illustrative example, the controller 236 requests the DMA 110 to fetch one or more packets for kernel circuit 234-1, then one or more packets for kernel circuit 234-2, and so on, based on which kernel circuits are busy and the available space in the input buffer. Once the stream traffic manager 212 knows how busy each of the kernel circuits 234 is and how much data storage is available in each respective input buffer of each kernel circuit 234, it arbitrates between the kernel circuits 234. In a particular embodiment, the controller 236 implements a round-robin arbitration scheme that stores the first "N" descriptors for each of the write queues 202 locally in the buffer(s) 238 and checks each input buffer of each kernel circuit for available space.

The architecture 100 may operate in a similar manner when transferring data from the kernel circuit 234 to the host system 102. For example, the stream traffic manager 212 may store the first "N" descriptors for each of the read queues 202-2, 202-4, 202-6, and 202-8. The stream traffic manager 212 may determine when result data is available in the output queue for the kernel circuit 234. In response to determining that a descriptor corresponding to an output buffer is available, i.e., contains stored results, the controller 236 initiates a data transfer from the output buffer to the host system 102 using the available data mover engine 240. The availability of a descriptor indicates that the host system 102 has available space to receive results from the kernel circuit.

For illustrative purposes, kernel circuit 234-1 may operate on data from input buffer 218. Kernel circuit 234-1 outputs the resulting data as a data stream to output buffer 220. Stream traffic manager 212, e.g., controller 236, may monitor the output buffer to determine when data is available, e.g., when at least a complete packet of data is available in the output buffer and the corresponding read queue has sufficient space available to store that data (e.g., at least a complete packet). In response to determining that output buffer 220 has available data and that a descriptor (which may be retrieved and cached in buffer 238 in stream traffic manager 212) is available in the corresponding read queue 202-2, controller 236 initiates a data transfer from output buffer 220 through interconnect 216 to DMA 110 and host system 102. The output buffer 216 converts the data stream into packetized data before sending the data out onto the interconnect 216 and onto the host system 102. In one or more embodiments, the arbitration 210 may implement round robin arbitration. In other embodiments, the arbitration 210 may implement other arbitration techniques. The arbitration technique, whether round robin or otherwise, implements interleaving or alternation of the data stream and/or packets from the kernel circuit 234.

In embodiments in which multiple streaming-enabled kernel circuits are implemented within an IC, each active kernel circuit receives a portion of the IC's data transfer bandwidth. Concurrent operation of multiple streaming-enabled kernel circuits generally means that such kernel circuits are designed to operate on fragments of data as they arrive at each respective kernel circuit, rather than operating on the entire completed data transfer before computation begins. This ability to operate on smaller fragments of data gives the streaming-enabled kernel circuits described herein more rapid access to data, which facilitates lower latency, higher performance, lower data storage requirements, lower overall cost, and lower power consumption.

When interleaving (or alternating) between different kernel circuits sending data to and/or receiving data from the DMA 110, the stream traffic manager 212 can ensure that the interconnect fabric, e.g., the interconnects 214, 216, is not blocked by slow kernel circuits. This is accomplished, at least in part, through the use of buffers 218-232. In one embodiment, each of the buffers 218-232 is sized to store at least one complete packet of data. As described, data destined for a kernel circuit is not sent out unless buffer space is available at the kernel circuit's input buffer. When a burst of packets arrives at the input buffer, the kernel circuit can empty the buffer on its own timetable without adversely affecting traffic on the interconnect 214, thereby preventing a congestion condition known as "head of line blocking." Similarly, data destined for the host system 102 from the kernel circuit is not sent out across the interconnect 216 from the kernel circuit until the entire packet has been transferred to the output buffer.

The output buffers of a kernel circuit have exclusive use of the interconnect 216 once a data transfer begins. If a kernel circuit falls behind or stops sending data mid-packet, the interconnect 216 cannot switch to servicing another kernel circuit until the interconnect 216 receives the entire packet, thereby locking the interconnect 216 and preventing the other kernel circuit from sending data to the host system. If an output buffer were to be omitted, one kernel circuit could adversely affect the performance of the other kernel circuit. In accordance with the inventive configurations described herein, each output buffer is capable of receiving and storing a minimum of an entire packet before attempting to send data to the interconnect 216. This feature ensures that once a packet begins to be transmitted, it is completed quickly enough that the interconnect 216 and the upstream infrastructure can accommodate the transfer, regardless of kernel circuit behavior or kernel circuit output data rate.

In one or more embodiments, the kernel circuits and buffers are implemented using programmable circuits. Thus, buffers are created only for kernel circuits that are actually implemented in the IC 104. Circuit resources of the IC 104 are not wasted on input and/or output buffers when a small number of kernel circuits are deployed. Resource usage scales with the number of kernel circuits implemented in the IC 104. In certain embodiments, data transfers across the interconnects 214, 216 are regulated through a system of buffer credits managed by the stream traffic manager 212.

In one or more embodiments, runtime 128 may provide various application programming interfaces (APIs) that can be called by user applications to support direct communication with kernel circuitry using data streams. Below is a list of exemplary APIs provided by runtime 128:
clCreateHostPipe - An OpenCL API that creates a read or write type data buffer for streaming data, also called a "streaming pipe."
clEnqueueWritePipeBuffer--Queues a packet directly onto a streaming pipe for writing (transferring data to the kernel circuit).
clEnqueueReadPipeBuffer--Queues a packet directly onto a streaming pipe for reading (data transfer from the kernel circuit).

The runtime 128 may further provide APIs for creating, destroying, starting, stopping, and modifying read and/or write queue pairs.
struct xclQueueContext
xclCreateWriteQueue--Creates a write queue in the host system. Allocates resources in the host system and initializes the write queue for the DMA 110 to issue device requests. A queue handle for the created write queue is returned for future access.
xclCreateReadQueue--Creates a read queue in the host system. Allocates resources in the host system and initializes a read queue for the DMA 110 to issue "from device" requests. A queue handle for the created read queue is returned for future access.
xclDestroyQueue--Destroys the specified read queue/write queue and reclaims the resources used to implement the destroyed read queue/write queue.
xclModifyQueue--Modifies the parameters of the specified read/write queue.
xclStartQueue--Puts the specified read/write queue into an active state where the queue can begin accepting and processing DMA requests.
xclStopQueue--Puts the specified read/write queue into an initialized state. All pending DMA requests are flushed.

The runtime 128 may further provide APIs for issuing writes and reads to and from kernel circuits, such as the following:
struct xclQueueRequest
struct xclWRBuffer
xclWriteQueue--Writes to a specified queue.
xclReadQueue--Reads from the specified queue.

The driver 130 may further provide APIs to support the operations of the DMA 110, such as:
streamq_create()
streamq_destroy()
streamq_write()/streamq_read

In one or more embodiments, runtime 128 provides input/output control (IOCTL) system calls for input/output operations involving IC 104 that may be invoked to create, destroy, start, stop, and modify read and/or write requests. In certain embodiments, these system calls are not available to user space applications executing in host system 102. Runtime 128 may further provide portable operating system interface (POSIX) read/write functions and asynchronous I/O (AIO) read/write functions that are available to user space applications executing within host system 102.

A system running an electronic design automation (EDA) application, including a hardware compiler/system linker, can map kernel arguments to queues during the design flow (e.g., high-level synthesis, synthesis, placement, routing, and/or configuration bitstream generation) that implements the kernel. The mapping information is generated and stored in a container file along with a configuration bitstream (e.g., a partial configuration bitstream) that specifies the kernel circuit. The container file is stored in the host system 102 for use and implementation in the IC 104.

When the host system 102 retrieves the container file and implements the configuration bitstream from the container file with the IC 104, the host system 102 can further extract metadata including mapping information generated during compilation. Once implemented in the IC 104, the mapping information is provided to the runtime 128 for use in setting up communication paths to route data streams between the host system 102 and the kernel circuitry.

Based on the utilization of a "pipe" data construct in the program code for the kernel, the EDA application can generate kernel circuitry (e.g., the configuration bitstream specifies the kernel circuitry) that is configured to use data streams for data transfers instead of memory-mapped transactions involving either off-chip or internal RAM. For example, in response to detecting the pipe data structure, the EDA application can generate the necessary hardware infrastructure and/or circuitry to support data transfers using data streams, as described with respect to Figures 1 and/or 2. An example of a kernel specified in OpenCL is provided below as Example 1.

When compiling the above exemplary kernel, the EDA application generates mapping information for p1 and p2. The mapping information includes register settings that, when implemented in IC 104, configure stream traffic manager 212 (e.g., by storing register settings in maps 242 and 244) and DMA 110 (by storing in control registers in DMA 110) to properly route data streams between host system 102 and a particular kernel circuit, such as kernel circuit 234-1. In one example, the mapping information specifies the particular route_id and flow_id to which each pipe is bound, and/or static information related to pipes p1 and p2. This mapping data is stored in the container file as metadata (e.g., program code) for the configuration bitstream that specifies the kernel circuit generated from the kernel.

For example, to send data from the host system's 102 memory to kernel circuit 234-1, the runtime 128 and/or driver 130 assigns the operation to p1 and binds p1 to queue structure 202-1. The host system 102 looks up the route_id for kernel circuit 234-1 from an internal table. The route_id specifies the location of kernel circuit 234-1. The host system 102 sets the control registers of the DMA 110 with pipe p1 and the associated queue 202-1. The host system 102 creates an entry correlating the route_id for kernel circuit 234-1 with queue 202-1 and pipe p1. In one or more embodiments, in response to receiving data corresponding to pipe p1, the stream traffic manager 212 can tag the kernel circuit bound data belonging to p1 with the correct route_id. Given the data tagged with this route_id, the stream traffic manager 212 and interconnect 214 can deliver the data to the kernel circuit 234-1 via the buffer 218.

Similarly, to transfer data from kernel circuit 234-1 to memory in host system 102, runtime 128 and/or driver 130 can assign the operation to p2 and bind p2 to queue 202-1. Host system 102 looks up the flow_id used to tag host bound data from kernel circuit 234-1. In one or more embodiments, kernel circuit 234-1 can tag outbound data with the appropriate flow_id. In one or more other embodiments, buffer 220 includes circuitry that can tag outbound data with the appropriate flow_id. Host system 102 configures DMA 110 on pipe p2 and associates pipe p2 with queue 202-2. The host system 102 further creates an entry correlating the flow_id (e.g., buffer 220) for the kernel circuit 234-1 with the queue 202-2 and pipe p2 for data transfer. The stream traffic manager 212 can further bind host-bound traffic tagged with the flow_id to pipe p2 when forwarding the data to the DMA 110.

Once both the DMA 110 and the stream traffic manager 212 are configured for data transfer, the DMA 110 is instructed to begin operating according to example 1 above.

FIG. 3 illustrates an example method 300 for transferring data between a host system and kernel circuits of a hardware accelerator using a data stream. Method 300 can begin with the host system storing one or more container files in memory. Each container file includes one or more configuration bitstreams and corresponding metadata. Each of the configuration bitstreams, which may be partial configuration bitstreams, specifies one or more kernel circuits.

At block 305, the host system selects a container file. For illustrative purposes, the container file includes a configuration bitstream and metadata about the configuration bitstream. The configuration bitstream may be a partial configuration bitstream. In one or more embodiments, the host system selects the container file in response to a user application requesting hardware accelerated functionality implemented by a kernel circuit specified by the configuration bitstream in the container file. The user application may specify a particular container file to be selected or retrieved from memory and implemented in the hardware accelerator.

In block 310, the host system extracts the configuration bitstream from the container file. The host system loads the configuration bitstream into the hardware accelerator's IC, e.g., IC 104. By loading the configuration bitstream into the hardware accelerator's IC, the kernel circuitry specified by the configuration bitstream is physically implemented within the IC and is available to perform tasks requested by the host system.

At block 315, the host system determines one or more pipe properties from the metadata. For example, the host system extracts metadata about the configuration bitstream from the selected container file. The metadata includes mapping information generated when the kernel was compiled. The mapping data includes one or more pipe properties that can be used to configure the DMA 110 and the stream traffic manager 212. For example, the pipe properties can include settings, e.g., register settings such as route_id and/or flow_id that can be loaded into the DMA 110 and/or stream traffic manager to establish a route for exchanging data between the host system and one or more kernel circuits implemented by the configuration bitstream extracted from the selected container file.

In one or more embodiments, the metadata for the configuration bitstream includes additional information generated during the design flow that allows the stream traffic manager to operate more efficiently. For example, the metadata can specify information, e.g., settings, that are specific to each kernel. Thus, using the metadata, the stream traffic manager can adjust, for each kernel circuit, how data is streamed to and/or from the kernel circuit to the host system. The metadata can specify, for example, the size of the kernel circuit's working data set (corresponding to a packet size), the computation time required for the kernel circuit per data set, the amount of prefetching desired for the kernel circuit, etc. During operation, the stream traffic manager can adjust the amount of data fetched for a kernel and the amount of prefetching according to the metadata for that particular kernel circuit.

As part of block 315, the host system can send settings (e.g., pipe properties and/or other information as described) to the stream traffic manager and/or DMA to set up a data path for streaming data between the implemented kernel circuitry and the host system. For example, the host system calls one or more functions available in the driver and/or runtime to set up the data path. The functions, for example, write settings to control registers of the DMA and maps of the stream traffic manager. The stream traffic manager can include additional control registers that can be written with settings as described herein.

In block 320, the host system uses the setting to implement a direct data transfer from the host system to the kernel circuit as a data stream. For example, the host system adds one or more descriptors to a write queue in the driver that corresponds to the input buffer of the target kernel circuit. The DMA can retrieve one or more of the descriptors and provide the retrieved descriptors to the stream traffic manager. The stream traffic manager temporarily stores the descriptors in an internal buffer. As described, the stream traffic manager can monitor the status of the input buffer for the target kernel circuit and, when space is available in the input buffer, can execute one or more of the descriptors that correspond to the input buffer of the target kernel circuit using an available data mover engine included in the stream traffic manager. Thus, the DMA 110 retrieves data in packetized form from the host memory. The stream traffic manager streams the data to the input buffer of the target kernel circuit. As described, the input buffer can convert the packetized data into streamed data.

In one or more embodiments, the data transferred to the target kernel circuit includes one or more instructions for the kernel circuit embedded therein. In this regard, the commands are said to be "in-band" with or to the data. By including instructions for the kernel circuit in the data stream, no separate signaling needs to be provided for the kernel circuit to start and/or stop operation of the kernel circuit. Such operation may be initiated by an in-band instruction included in the data stream(s). In certain embodiments, the kernel circuit and/or the host system are capable of exchanging a continuous data stream, or a data stream optionally interspersed with instructions (e.g., commands or status information).

In one or more embodiments, the host system may determine that a data transfer should be implemented as a data stream based on the data type used by the user application requesting the data transfer and/or the particular API called by the user application.

In block 325, the host system uses the pipe property to implement further data transfer from the kernel circuit directly to the host system as a data stream. For example, the host system adds one or more descriptors to the driver's read queue corresponding to the output buffer of the target kernel circuit. As mentioned, the DMA can retrieve one or more of the descriptors and provide the retrieved descriptors to the stream traffic manager. The stream traffic manager temporarily stores the descriptors in an internal buffer. The stream traffic manager monitors the status of the output buffer for the kernel circuit and, when a data stream is available in the output buffer, can execute one or more of the descriptors corresponding to the output buffer of the target kernel circuit using an available data mover engine included in the stream traffic manager. Thus, the data mover engine of the stream traffic manager retrieves the packetized data from the output buffer of the target kernel circuit and provides the packetized data to the DMA. As mentioned, the output buffer converts the data stream to packetized data. The DMA provides the packetized data to the host memory over the communication bus.

Figure 4 illustrates an exemplary architecture 400 for exchanging data between kernel circuits using data streams. The architecture 400 supports use cases where an application requires multiple large and complex kernel circuits, and additional ICs are used to augment the programmable circuitry provided by the primary IC. The primary IC is configured to support communication with a host system via endpoints and DMA. The primary IC also includes a stream traffic manager. In one or more embodiments, the stream traffic manager is capable of routing packetized data for the kernel circuits to one of several different ports, each connected to an independent interconnect. Partitioning the kernel circuits to different interconnects allows the kernel circuits to be located in different physical regions of the IC, e.g., different dies in the case of a multi-die IC. Furthermore, the different interconnects isolate the kernel circuits in the different regions so that they do not interfere with each other. This partitioning allows multi-die ICs to be used and also allows secondary ICs to be used.

The architecture 400 includes IC 104 and IC 402. In one or more embodiments, IC 104 and 402 are coupled to the same circuit board, e.g., a hardware accelerator, that may also include RAM (not shown). In the example of FIG. 4, each of IC 104 and 402 is implemented as a multi-die IC. IC 104 includes dies 404 and 406. IC 402 includes dies 408 and 410. Each of dies 404, 406, 408, and 410 is implemented to include programmable circuitry, which is described in more detail herein with respect to FIG. 7. In a particular embodiment, one or more of dies 404, 406, 408, and 410 include one or more hardwired circuit blocks. In one example, each of dies 404, 406, 408, and 410 is implemented as a field programmable gate array (FPGA).

In the example of FIG. 4, die 404 and die 406 are included in the same package, but die 408 and die 410 are included in a different package. IC 104 and IC 402 may be implemented using any of a variety of available multi-die technologies. In one or more embodiments, die 404 and 406 are mounted on an interposer that includes wires capable of transmitting signals between die 404 and die 406. Similarly, die 408 and 410 are mounted on an interposer that includes wires capable of transmitting signals between die 408 and die 410. The dies may be attached using multiple solder bumps or another connection technique. The interposer includes multiple through vias that allow, for example, selected signals to cross over to the outside of the multi-die IC package and to a substrate.

For illustrative purposes, dies 404 and 408 are shaded to better show the different circuit blocks included in each respective die. In the example of FIG. 4, dies 404 and 408 each include additional circuit blocks not included in dies 406 and 410. For example, die 404 includes an endpoint 108, a DMA 110, a stream traffic manager 212, and a transceiver 442, while die 406 does not. In one or more embodiments, one or more of the endpoint 108, DMA 110, and/or transceiver 442 are implemented as hardwired circuit blocks. In certain embodiments, the endpoint 108, DMA 110, and/or transceiver 442 are implemented in programmable circuitry. These circuit structures are not repeated in die 406. Similarly, die 408 includes a transceiver 444 and a satellite stream traffic manager 412, while die 410 does not. These structures are not repeated in die 410.

In the example of FIG. 4, the endpoints 108, the DMA 110, and the stream traffic manager 212 are implemented substantially as described with respect to FIGS. 1 and 2. However, in the example of FIG. 4, the stream traffic manager 212 includes additional I/O ports. For example, the stream traffic manager 212 includes an additional I/O port that connects to a transceiver 442. Additionally, one or more I/O ports of the stream traffic manager 212 couple to the die 406, and in particular to an interconnect 416. In one or more embodiments, the interconnect 414 and the interconnect 416 represent instances of the interconnect 214 and the interconnect 216, respectively. Thus, each of the dies 404 and 406 includes instances of the interconnect 214 and the interconnect 216. As shown, the kernel circuit 234 and corresponding buffers are spread across the dies 404 and 406.

The die 408 includes a transceiver 444, a satellite stream traffic manager 412, an interconnect 418, buffers 422, 424, 426, and 428, and kernel circuits 440-1 and 440-2. In a particular embodiment, the interconnect 418 represents another instance of the interconnect 214 and another instance of the interconnect 216. The die 410 includes an interconnect 420, buffers 432, 434, 436, and 438, and kernel circuits 440-3 and 440-4. Similarly, the interconnect 420 represents another instance of the interconnect 214 and another instance of the interconnect 216. In one or more embodiments, the transceiver 444 is implemented as a hardwired circuit block. In a particular embodiment, the transceiver 444 is implemented in a programmable circuit.

In the example of FIG. 4, the IC 104 can act as a master in that the die 404 includes an endpoint 108 for communicating with the host system 102. Additionally, the stream traffic manager 212 can communicate with the satellite stream traffic manager 412 via transceivers 442 and 444. In one or more embodiments, the transceivers 442 and 444 implement a high-speed point-to-point interconnect including multiple serial data lanes. The connection formed by the transceivers 442 and 444 exchanges data between the stream traffic manager 212 and the satellite stream traffic manager 412. Additionally, the transceivers 442 and 444 can provide an additional layer of buffering to hide the additional latency due to crossing the IC boundary. In the example of FIG. 4, the stream traffic manager 212 and the satellite stream traffic manager 412 send and receive packetized data. In one or more embodiments, the transceivers 442 and 444 are capable of serializing streaming packets exchanged between the stream traffic manager 212 and the satellite stream traffic manager 412 for transmission from one transceiver to the other, and deserializing the transmitted data for sending and/or handling within the ICs 104 and 402. Similarly, the transceivers 442 and 444 are capable of serializing credit messages exchanged between the stream traffic manager 212 and the satellite stream traffic manager 412 for transmission from one transceiver to the other, and deserializing such messages for sending and/or handling within the ICs 104 and/or 402.

Using architecture 400, host system 102 can configure DMA 110, stream traffic manager 212, and satellite stream traffic manager 412 to route packetized data. For example, stream traffic manager 212 can pass necessary mapping data and/or settings onto satellite stream traffic manager 412. Once configured, host system 102 can offload tasks to IC 104 and/or IC 402. Additionally, host system 102 can direct tasks to one or more of kernel circuits 234 and/or one or more of kernel circuits 440.

While the kernel circuit 234 was included in a single die in the example of FIG. 2, in the example of FIG. 4, the kernel circuit 234 is distributed across dies 404 and 406. Similarly, the kernel circuit 440 is distributed across dies 408 and 410. Although the stream traffic manager 212 allows data to be provided to multiple kernel circuits concurrently, the stream traffic manager 212 can also establish connections between the kernel circuits 234 (e.g., from 234-1 to 234-2 or vice versa, from 234-1 or 234-2 to 234-3 or 234-4, from 234-3 to 234-4 or vice versa, from 234-3 or 234-4 to 234-2 or 234-1). For example, if the kernel circuits are not initially configured to communicate directly with each other, the stream traffic manager 212 can enable a kernel circuit to stream data to another kernel circuit, whether in the same die or in a different die of the same IC. Similarly, the satellite stream traffic manager 418 can enable a kernel circuit to stream data to another kernel circuit, whether in the same die or in a different die of the same IC (e.g., from 440-1 or 440-2 to 440-3 or 440-4, from 440-1 to 440-2 or vice versa, from 440-3 to 440-4 or vice versa, from 440-3 or 440-4 to 440-1 or 440-2). Data exchanged between kernel circuits located on different dies and/or different ICs must flow through, and be controlled by, the stream traffic manager 212 and/or the satellite stream traffic manager 412, as the case may be.

In another embodiment, when data is exchanged between kernel circuits located on the same die, the data may possibly flow from the sending kernel circuit to the interconnect and from the interconnect to the receiving kernel circuit, bypassing the stream traffic manager 212 and/or the satellite stream traffic manager 412, but under the control of the stream traffic manager 212 and/or the satellite stream traffic manager 412. In either case, the output buffer of the sending kernel circuit converts the data stream output from the sending kernel circuit into packetized data, and the input buffer of the receiving kernel circuit converts the packetized data into a data stream for consumption by the receiving kernel circuit.

The stream traffic manager 212 may also communicate with a satellite stream traffic manager 412. The satellite stream traffic manager 412 is implemented substantially similarly to the stream traffic manager 212. The communication between the stream traffic manager 212 and the satellite stream traffic manager 412 via the transceivers 442 and 444 may be implemented in a manner that allows a kernel circuit in one IC to communicate with a kernel circuit in a different IC (e.g., 234-1 or 234-2 to 440-1 or 440-2, 234-1 or 234-2 to 440-3 or 440-4, 234-3 or 234-4 to 440-1 or 440-2 ... or 440-2, 234-3 or 234-4 to 440-3 or 440-4, 440-1 or 440-2 to 234-1 or 234-2, 440-1 or 440-2 to 234-3 or 234-4, 440-3 or 440-4 to 234-1 or 234-2, 440-3 or 440-4 to 234-3 or 234-4).

Nevertheless, kernel circuits may be implemented to communicate directly with each other. In that case, the kernel circuits are created and implemented within a programmable circuit to incorporate this capability. Such a connection is shown in FIG. 4, where kernel circuit 234-3 can communicate directly with kernel 234-4 to provide data results to kernel 234-4 without using stream traffic manager 212. If the kernel circuits are located on different dies and/or different ICs, stream traffic manager 212 and/or satellite stream traffic manager 412 are required.

In many cases, kernel circuits (e.g., the operations performed by such kernel circuits) are serially linked together. Data may be passed from one kernel circuit to another in steps where each different kernel circuit is customized to perform a different operation. In other implementations that use memory-mapped interfaces, whether the memory is local within the IC or external to the IC, the progress of the upstream kernel circuit(s) must be tracked by the host system in order to initiate the downstream kernel circuit(s) in a timely manner, e.g., when the operation of the upstream kernel circuit(s) is detected. In some cases, the host system must also copy data from the upstream kernel circuit to the downstream kernel if the downstream kernel circuit does not have access to the same memory as the upstream kernel circuit. This type of architecture creates significant overhead in the software in the host system 102 and often results in under-utilization of the hardware (kernel circuits).

The streaming architecture described in this disclosure, which uses in-band instructions in a data stream passed from kernel circuit to kernel circuit, allows one kernel circuit to pass data directly to another kernel circuit along with instructions contained in the data stream, thereby implementing concatenated processing of data through multiple kernel circuits without the involvement of the host system 102. The streaming architecture reduces the overhead imposed on the host system and uses hardware resources more efficiently.

It should be appreciated that the stream traffic manager circuitry can provide data from the host system 102 to either the kernel circuitry implemented in IC 102 or IC 402. Packetized data provided from the host system 102 to the kernel circuitry in IC 104 passes through the endpoint 108, DMA 110, and stream traffic manager 212. A data stream output from the kernel circuitry in IC 104 (e.g., a result data stream) passes to the host system 102 via the stream traffic manager 212, DMA 110, and endpoint 108. Packetized data provided from the host system 102 to the kernel circuitry in IC 402 passes through the endpoint 108, DMA 110, stream traffic manager 212, transceivers 442 and 444, and satellite stream traffic manager 412. The data streams output from the kernel circuits in IC 402 (e.g., result data streams) pass through satellite stream traffic manager 412, transceivers 444 and 442, stream traffic manager 212, DMA 110, and endpoint 108. In sending and/or receiving packetized data to the kernel circuits in IC 402, host system 102 may operate substantially as described with respect to FIG. 2, with input driver 130 generating read and write queues for each kernel circuit, whether implemented in IC 102 or IC 402.

The architecture shown in Figures 1, 2, and 4 allows an upstream kernel circuit to stream data to any available downstream kernel circuit without requiring more complex interconnect circuitry supporting direct connections between each possible pair of kernel circuits. The architecture of Figures 1, 2, and 4 implements this capability by having the upstream kernel circuit output data to a stream traffic manager circuit (for purposes of explanation, "stream traffic manager circuit" refers to a stream traffic manager, a satellite stream traffic manager, or both operating in a cooperative manner). The stream traffic manager circuit routes the data to the downstream kernels. Since the data is regulated by the stream traffic manager circuit using credits, large store and forward buffers are not required. Furthermore, the host system 102 is not involved in the data transfer. As an illustrative and non-limiting example, the upstream kernel circuit, e.g., the send kernel circuit, performs compression and the downstream kernel circuit performs encryption. The upstream kernel circuit sends the resulting compressed data to a stream traffic manager circuit, which routes the packetized data by an output buffer of the sending kernel circuit to a downstream kernel circuit, e.g., a receiving kernel circuit. An input buffer of the receiving kernel circuit converts the packetized data into a data stream. The downstream kernel circuit can provide the resulting encrypted data to a stream traffic manager circuit, which can then route the encrypted data to yet another kernel circuit or provide the encrypted data to the host system 102.

The streaming architecture described within this disclosure also allows the place and route functions of an EDA application (executed by a data processing system) to operate more efficiently (require less time to complete) because the place and route tools do not need to consider the relative placement of upstream and downstream kernel circuits. This is particularly important when two or more kernel circuits that exchange data via a data stream are located on different dies and/or different ICs.

Without the exemplary streaming architecture described within this disclosure, direct kernel-to-kernel routing would need to be implemented between each pair of kernel circuits that are intended to communicate. This type of connectivity imposes significant constraints on the placement and routing of kernel circuits to meet timing requirements, and becomes even more difficult when crossing die and/or IC boundaries. Furthermore, using the architecture described herein also achieves greater clock speeds for the implemented kernel circuits while providing the described flexibility. An architecture using point-to-point connections between each possible pair of kernel circuits would require significantly more resources of the programmable circuit, so the resulting implementation would operate at a slower clock frequency than is achievable using the exemplary streaming architecture described herein.

In the example of FIG. 4, both IC104 and IC402 are implemented as multi-die ICs. In one or more other embodiments, one or both of IC104 and IC402 are implemented as single-die ICs that include a transceiver.

FIG. 5 illustrates an exemplary method 500 for exchanging data between kernel circuits using a data stream. Method 500 can begin with a host system offloading tasks to a kernel circuit in a hardware accelerator. In one or more embodiments, method 500 begins after blocks 305-320 of FIG. 3 are performed and/or for each IC involved in the data transfer. In the example of FIG. 5, a kernel circuit, referred to herein as a sending kernel circuit, performs an operation in a sequence of operations, each operation being performed by a different kernel circuit.

At block 505, the sending kernel circuit outputs or stores the data streaming in an output buffer attached to the output port. At block 510, the stream traffic manager circuit detects the data stream stored in the output buffer of the sending kernel circuit. The stream traffic manager circuit can monitor the status of the buffer as described with respect to FIG. 2. In one or more embodiments, the data stream includes information specifying a destination of the data. The destination is not the host system in this example, but rather another kernel circuit called a receiving kernel circuit. In one or more other embodiments, the stream traffic manager circuit is configured to route the data from the sending kernel circuit to another destination, such as the receiving kernel circuit and/or the host system, for example, using the mapping data previously described. At block 515, the stream traffic manager circuit determines the receiving kernel circuit. The stream traffic manager circuit can, for example, read the data stream stored in the output buffer of the sending kernel circuit and determine the designated receiving kernel circuit. In another example, the stream traffic manager determines the receiving kernel circuit based on mapping data (e.g., mapping of a particular kernel circuit output to a destination) stored in the stream traffic manager.

At block 520, the stream traffic manager circuit determines whether the input buffer of the receiving kernel circuit has enough space available to store the data stream from the sending kernel circuit. At block 525, in response to determining that the input buffer of the receiving kernel circuit has enough space, the stream traffic manager circuit initiates a data transfer from the sending kernel circuit to the receiving kernel circuit. The stream traffic manager circuit transfers data from the output buffer of the sending kernel circuit to the input buffer of the receiving kernel circuit through the interconnect(s) and/or through a transceiver if a cross-IC data transfer is performed. In one or more embodiments, when transferring data between kernel circuits in the same die, the data may be sent through the associated interconnect under the control of the stream traffic manager circuit without passing through the stream traffic manager circuit.

In a particular embodiment, the data stream from the sending kernel circuit includes one or more instructions in-band within the data stream. In one example, the instructions are included in the payload portion of the data stream (or packetized data) from the sending kernel circuit to the receiving kernel circuit. As described, the output buffer of the sending kernel circuit converts the data stream into packetized data for sending to the receiving kernel circuit. The input buffer of the receiving kernel circuit converts the received packetized data into a data stream that is provided to the receiving kernel circuit.

In the example of FIG. 5, it should be appreciated that a data stream may be sent from a kernel circuit in die 404 to a kernel circuit in die 406, or from a kernel circuit in die 406 to a kernel circuit in die 404. Similarly, a data stream may be sent from a kernel circuit in die 408 to a kernel circuit in die 410, or from a kernel circuit in die 410 to a kernel circuit in die 408.

5 references a stream traffic manager circuit. In this regard, method 500 may be implemented with the stream traffic manager performing the described operations (e.g., both the send kernel circuitry and the receive kernel circuitry are in IC 104), the satellite stream traffic manager performing the described operations (e.g., both the send kernel circuitry and the receive kernel circuitry are in IC 402), or both the stream traffic manager and the satellite stream traffic manager performing the operations (e.g., the send kernel circuitry and the receive kernel circuitry are in different ICs). In the latter case, it should be appreciated that the stream traffic manager and the satellite stream traffic manager each interact with kernel circuits located in the same IC.

For example, if the transmit kernel circuit and the receive kernel circuit are located on different ICs, the stream traffic manager and the satellite stream traffic manager can communicate via the transceivers 442 and 444 to determine the status of the input and output buffers of the kernel circuits. For example, the stream traffic manager can determine the status of the buffers in IC 104, and the satellite stream traffic manager can determine the status of the buffers in IC 402. The stream traffic manager can request the status of any buffer in IC 402 from the satellite stream traffic manager, and the satellite stream traffic manager responds with the requested status(es). Similarly, the satellite stream traffic manager can request the status of any buffer in IC 104 from the stream traffic manager, and the stream traffic manager responds with the requested status(es). The communication between the stream traffic manager and the satellite stream traffic manager supports transmit kernel circuit and receive kernel circuit located on the same die of IC 104 or on different dies of IC 104, on the same die of IC 402 or on different dies of IC 402, or on different ICs.

FIG. 6 illustrates an exemplary system 600 for use with one or more embodiments described herein. System 600 is an example of computer hardware that may be used to implement a computer, a server, a portable computer such as a laptop or tablet computer, or other data processing system. For example, system 600 is an exemplary implementation of host system 102 and/or another system that executes an EDA application to generate container files as described herein.

In the example of FIG. 6, system 600 includes at least one processor 605. Processor 605 is coupled to memory 610 through interface circuitry 615. System 600 is capable of storing computer-readable instructions (also referred to as "program code") in memory 610. Memory 610 is an example of a computer-readable storage medium. Processor 605 is capable of executing program code accessed from memory 610 via interface circuitry 615.

Memory 610 may include one or more physical memory devices, such as, for example, local memory and bulk storage devices. Local memory generally refers to the non-persistent memory device(s) used during the actual execution of the program code. Examples of local memory include RAM and/or any of the various types of RAM suitable for use by the processor during execution of the program code (e.g., dynamic RAM or "DRAM" or static RAM or "SRAM"). Bulk storage devices refer to persistent data storage devices. Examples of bulk storage devices include, but are not limited to, hard disk drives (HDDs), solid state drives (SSDs), flash memory, read only memory (ROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), or other suitable memory. System 600 may also include one or more cache memories (not shown) that provide temporary storage of at least some program code to reduce the number of times the program code must be retrieved from a bulk storage device during execution.

Memory 610 may store program code and/or data. In one or more embodiments, when system 600 implements a system such as host system 102, memory 610 may store and execute a framework the same or similar to that described with respect to FIG. 1. The framework may also include an operating system. One or more containers may also be stored in memory 610 for implementation within a hardware accelerator 625 attached to system 600 through interface circuit 615. Hardware accelerator 625 includes one or more ICs having the same or similar architecture as that described with respect to FIG. 7.

In one or more other embodiments, system 600 implements an EDA system that executes EDA applications. Thus, system 600 can process program code that specifies a kernel to generate a kernel circuit specified as a configuration bitstream or possibly a partial configuration bitstream. System 600 includes the configuration bitstream(s) in a container file. Additionally, system 600 can generate mapping information and include the mapping information as metadata in the container file. In embodiments in which system 600 implements an EDA system, hardware accelerator 625 may or may not be included.

System 600, e.g., processor 605, is capable of executing the operating system, applications, and/or frameworks described herein to perform the operations described within this disclosure. Thus, the instructions and/or data stored in memory 610 may be considered an integral part of system 600. Furthermore, it should be appreciated that the data used, generated, and/or acted upon by system 600 (e.g., processor 605) are functional data structures that provide functionality when employed as part of the system.

Examples of interface circuitry 615 include, but are not limited to, a system bus and an input/output (I/O) bus. Interface circuitry 615 may be implemented using any of a variety of bus architectures. Examples of bus architectures may include, but are not limited to, an Extended Industry Standard Architecture (EISA) bus, an Accelerated Graphics Port (AGP), a Video Electronics Standards Association (VESA) local bus, a Universal Serial Bus (USB), and a PCIe bus.

The system 600 may further include one or more I/O devices 620 coupled to the interface circuit 615. The I/O devices 620 may be coupled to the system 600, e.g., the interface circuit 615, either directly or through an intervening I/O controller. Examples of I/O devices 620 include, but are not limited to, a keyboard, a display device, a pointing device, one or more communication ports, and a network adapter. A network adapter refers to a circuit that allows the system 600 to become coupled to other systems, computer systems, remote printers, and/or remote storage devices through an intervening private or public network. Modems, cable modems, Ethernet cards, and wireless transceivers are examples of different types of network adapters that may be used with the system 600.

System 600 may include fewer components than those shown, or additional components not shown in FIG. 6, depending on the particular type of device and/or system being implemented. Additionally, the particular operating system, application(s), and/or I/O devices included may vary based on the system type. Additionally, one or more of the illustrated components may be incorporated into or otherwise form a portion of another component. For example, a processor may include at least some memory. System 600 may be used to implement a single computer or multiple networked or interconnected computers, each implemented using the architecture of FIG. 6 or a similar architecture.

Some ICs, called programmable ICs, can be programmed to perform a specified function. One example of a programmable IC is an FPGA. An FPGA typically contains an array of programmable tiles. These programmable tiles can include, for example, input/output blocks (IOBs), configurable logic blocks (CLBs), dedicated RAM blocks (BRAMs), multipliers, digital signal processing blocks (DSPs), processors, clock managers, delay-locked loops (DLLs), etc.

Each programmable tile typically includes both programmable interconnect circuitry and programmable logic circuitry. The programmable interconnect circuitry typically includes a number of interconnect lines of varying lengths interconnected by programmable interconnect points (PIPs). The programmable logic circuitry implements the logic of a user design using programmable elements, which may include, for example, function generators, registers, arithmetic logic, etc.

Programmable interconnect circuits and programmable logic circuits are typically programmed by loading a stream of configuration data into internal configuration memory cells that define how the programmable elements are configured. The configuration data can be read from memory (e.g., from an external PROM) or written to the FPGA by an external device. The overall state of the individual memory cells then determines the functionality of the FPGA.

Another type of programmable IC is the Complex Programmable Logic Device, or CPLD. A CPLD contains two or more "functional blocks" connected together by an interconnect switch matrix and to input/output (I/O) resources. Each functional block in a CPLD contains a two-level AND/OR structure similar to those used in programmable logic array (PLA) and programmable array logic (PAL) devices. In CPLDs, configuration data is generally stored on-chip in non-volatile memory. In some CPLDs, configuration data is stored on-chip in non-volatile memory and then downloaded to volatile memory as part of an initialization (programming) sequence.

For all of these programmable ICs, the functionality of the device is controlled by data bits provided to the device for that purpose. The data bits may be stored in volatile memory (e.g., static memory cells, as in FPGAs and some CPLDs), non-volatile memory (e.g., flash memory, as in some CPLDs), or any other type of memory cell.

Other programmable ICs are programmed by applying processing layers, such as metal layers, that programmably interconnect various elements on the device. These programmable ICs are known as mask-programmable devices. Programmable ICs may also be implemented in other ways, for example, using fuse or anti-fuse technology. The phrase "programmable IC" may include, but is not limited to, these devices, and may further encompass devices that are only partially programmable. For example, one type of programmable IC includes a combination of hard-coded transistor logic and a programmable switch fabric that programmably interconnects the hard-coded transistor logic.

Figure 7 illustrates an example architecture 700 for an IC. In one aspect, the architecture 700 may be implemented within a programmable IC. For example, the architecture 700 may be used to implement an FPGA. The architecture 700 may also represent a system-on-chip (SoC) type of IC. A SoC is an IC that includes a processor that executes program code and one or more other circuits. The other circuits may be implemented as hardwired circuits, programmable circuits, and/or combinations thereof. The circuits may operate in cooperation with each other and/or with the processor.

As shown, architecture 700 includes several different types of programmable circuits, e.g., logic, blocks. For example, architecture 700 may include a number of different programmable tiles, including multi-gigabit transceivers (MGTs) 701, configurable logic blocks (CLBs) 702, BRAMs 703, input/output blocks (IOBs) 704, configuration and clocking logic (CONFIG/CLOCKS) 705, digital signal processing blocks (DSPs) 706, specialized I/O blocks 707 (e.g., configuration and clock ports), and other programmable logic 708, such as digital clock managers, analog-to-digital converters, system monitoring logic, etc.

In some ICs, each programmable tile includes a programmable interconnect element (INT) 711 with standardized connections between corresponding INTs 711 in each adjacent tile. Thus, the INTs 711 collectively implement the programmable interconnect structure for the IC shown. Each INT 711 also includes connections between programmable logic elements in the same tile, as shown by the example included at the top of FIG. 7.

For example, CLB 702 may include a configurable logic element (CLE) 712 that may be programmed to implement user logic, and a single INT 711. BRAM 703 may include one or more INTs 711 plus a BRAM logic element (BRL) 713. In general, the number of INTs 711 included in a tile depends on the height of the tile. As depicted, the BRAM tile has the same height as five CLBs, although other numbers (e.g., four) may also be used. DSP tile 706 may include an appropriate number of INTs 711 plus a DSP logic element (DSPL) 714. IOB 704 may include, for example, one instance of an INT 711 plus two instances of an I/O logic element (IOL) 715. The actual I/O pads connected to the IOL 715 may not be limited to the area of the IOL 715.

In the example depicted in FIG. 7, a columnar area near the center of the die, for example formed from regions 705, 707, and 708, may be used for configuration, clocks, and other control logic. A horizontal area 709 extending from this column may be used to distribute clock and configuration signals across the width of the programmable IC.

Some ICs utilizing the architecture shown in FIG. 7 include additional logic blocks that disrupt the regular columnar structure that makes up most of the IC. The additional logic blocks may be programmable blocks and/or dedicated circuits. For example, the processor block shown as PROC 710 spans several columns of CLBs and BRAMs.

In one aspect, PROC 710 may be implemented as a dedicated circuit fabricated as part of a die that implements the programmable circuitry of an IC, e.g., as a hardwired processor. PROC 710 may represent any of a variety of different processor types and/or systems ranging in complexity from an individual processor, e.g., a single core capable of executing program code, to an entire processor system having one or more cores, modules, coprocessors, interfaces, etc.

In another aspect, PROC 710 may be omitted from architecture 700 and replaced with one or more of the other types of programmable blocks described. Furthermore, such blocks may be utilized to form a "soft processor" in that various blocks of programmable circuitry may be used to form a processor capable of executing program code, as in the case of PROC 710.

The phrase "programmable circuitry" refers to programmable circuit elements within an IC, such as the various programmable or configurable circuit blocks or tiles described herein, as well as the interconnect circuitry that selectively couples the various circuit blocks, tiles, and/or elements according to configuration data loaded into the IC. For example, the circuit blocks shown in FIG. 7 that are external to PROC 710, such as CLB 702, are considered programmable circuitry of the IC.

Generally, the functionality of a programmable circuit is not established until configuration data is loaded into the IC. A set of configuration bits may be used to program the programmable circuit of an IC, such as an FPGA. The configuration bit(s) are generally referred to as a "configuration bitstream." Generally, a programmable circuit is not operable or functional without first loading a configuration bitstream into the IC. The configuration bitstream effectively implements a particular circuit design within the programmable circuit. The circuit design specifies, for example, the functional aspects of the programmable circuit blocks and the physical connectivity between the various programmable circuit blocks.

"Hardwired" or "hardened" circuits, i.e., non-programmable circuits, are fabricated as part of an IC. Unlike programmable circuits, hardwired circuits or circuit blocks are not implemented after the fabrication of an IC through the loading of a configuration bitstream. Hardwired circuits are generally considered to have dedicated circuit blocks and interconnects that are functional without first loading, for example, a configuration bitstream into an IC, e.g., PROC 710.

In some cases, a hardwired circuit may have one or more operating modes that can be set or selected according to register settings or values stored in one or more memory elements within the IC. The operating mode may be set, for example, through the loading of a configuration bitstream into the IC. Despite this capability, a hardwired circuit is not considered a programmable circuit because it is operational and has a specific function when fabricated as part of an IC.

In the case of a SoC, the configuration bitstream may specify the circuitry to be implemented in the programmable circuitry and the program code to be executed by the PROC 710 or a soft processor. In some cases, the architecture 700 includes a dedicated configuration processor that loads the configuration bitstream into appropriate configuration memory and/or processor memory. The dedicated configuration processor does not execute user-specified program code. In other cases, the architecture 700 may utilize the PROC 710 to receive the configuration bitstream, load the configuration bitstream into the appropriate configuration memory, and/or extract the program code for execution.

7 is intended to illustrate an exemplary architecture that may be used to implement a programmable circuit, e.g., an IC including a programmable fabric. For example, the number of logic blocks in a column, the relative width of the columns, the number and order of columns, the types of logic blocks included in the columns, the relative sizes of the logic blocks, and the interconnect/logic implementations included in the upper portion of FIG. 7 are merely exemplary. In an actual IC, for example, two or more adjacent columns of CLBs are typically included wherever CLBs appear to facilitate efficient implementation of the user circuit design. However, the number of adjacent CLB columns may vary with the overall size of the IC. Additionally, the size and/or positioning of blocks such as PROC 710 within the IC are for illustrative purposes only and are not intended as limitations.

As described, ICs implemented using architecture 700 or an architecture similar to architecture 700 may be used to implement the streaming architectures described herein. In one or more embodiments, the endpoints 108, DMA 110, stream traffic manager 212, satellite stream traffic manager 412, interconnects 214 and 216, buffers 218-232, and kernel circuit 234 may be implemented using programmable circuits. In one or more other embodiments, selected ones of the circuit blocks, such as the endpoints 108, DMA 110, and/or interconnects, may be implemented as hardened or hardwired circuit blocks. In one or more embodiments, the input buffers and/or output buffers may be implemented as AXI4-Stream data FIFOs.

In particular embodiments, any buffers or queues described as being located on the IC 104 may be implemented using available memory resources (e.g., BRAM) or other similar circuit blocks available within the IC 104, as opposed to using slower off-chip RAM. For example, buffers 218-232, queues in the traffic stream manager 212, and/or queues in the DMA 110 may be implemented using memory resources available on the IC.

The architecture described herein is provided for purposes of illustration and not limitation. For example, an IC may include fewer or more kernel circuits than those shown in the figures. Furthermore, based on the number of kernel circuits implemented using the programmable circuitry of the IC, the number of queues in the driver and the number of queues in the buffer implemented within the IC will vary.

For purposes of explanation, specific nomenclature is set forth to provide a thorough understanding of the various inventive concepts disclosed herein. However, the terminology used herein is intended to be merely for the purpose of describing particular aspects of the inventive configurations and is not intended to be limiting.

As defined herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

As defined herein, the term "about" means close to, but not exactly, the approximate or exact value or amount. For example, the term "about" can mean that a stated property, parameter, or value is within a given amount of the exact property, parameter, or value.

The terms "at least one," "one or more," and "and/or," as defined herein, are open-ended expressions that are both conjunctive and disjunctive in operation, unless otherwise specified. For example, each of the expressions "at least one of A, B, and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C," and "A, B, and/or C" means A only, B only, C only, A and B together, A and C together, B and C together, or A, B, and C together.

As used herein, the term "automatically" means without user intervention. As used herein, the term "user" means a human being.

The term "computer-readable storage medium" as defined herein means a storage medium that contains or stores program code for use by or with an instruction execution system, apparatus, or device. A "computer-readable storage medium" as defined herein is not itself a transitory propagating signal. A computer-readable storage medium may be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. The various forms of memory described herein are examples of computer-readable storage media. A non-exhaustive list of more specific examples of computer-readable storage media may include portable computer diskettes, hard disks, RAM, read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electronically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, and the like.

The term "if" as defined herein means "when" or "upon" or "in response to" or "responsive to," depending on the context. Thus, the phrase "when it is determined" or "when the stated condition or event is detected" may be interpreted to mean "upon determining" or "in response to determining" or "upon detecting the stated condition or event" or "in response to detecting the stated condition or event" or "in response to detecting the stated condition or event," depending on the context.

The term "in response to" as defined herein and similar phrases as explained above, e.g., "when," "when," or "upon," means to readily respond or react to an action or event. The response or reaction is performed automatically. Thus, when a second action is performed "in response to" a first action, there is a causal relationship between the occurrence of the first action and the occurrence of the second action. The term "in response to" indicates a causal relationship.

As defined herein, the terms "one embodiment," "an embodiment," "one or more embodiments," "particular embodiment," or similar phrases mean that a particular feature, structure, or characteristic described with respect to an embodiment is included in at least one embodiment described within the disclosure. Thus, appearances of the phrases "in one embodiment," "in an embodiment," "in one or more embodiments," "in a particular embodiment," and similar phrases throughout this disclosure may, but do not necessarily, all refer to the same embodiment. The terms "embodiment" and "configuration" are used interchangeably within this disclosure.

As defined herein, the term "processor" means at least one hardware circuit capable of executing instructions contained in program code. The hardware circuit may be an integrated circuit. Examples of processors include, but are not limited to, central processing units (CPUs), array processors, vector processors, digital signal processors (DSPs), and controllers.

As defined herein, the term "output" means storing in a physical memory element, e.g., a device, writing to a display or other peripheral output device, sending or transmitting to another system, exporting, etc.

As defined herein, the term "substantially" means that the recited characteristic, parameter, or value need not be achieved exactly, but that deviations or variations, including, for example, tolerances, measurement errors, measurement accuracy limits, and other factors known to those skilled in the art, may occur in amounts that do not interfere with the effect that the characteristic is intended to provide.

Terms such as first, second, etc. may be used herein to describe various elements. These terms are only used to distinguish one element from another, unless otherwise stated or the context clearly indicates otherwise, and therefore these elements should not be limited by these terms.

A computer program product may include a computer-readable storage medium(s) having computer-readable program instructions thereon for causing a processor to perform aspects of the inventive arrangements described herein. Within this disclosure, the term "program code" is used interchangeably with the term "computer-readable program instructions." The computer-readable program instructions described herein may be downloaded from the computer-readable storage medium to the respective computing/processing device or to an external computer or external storage device via a network, e.g., the Internet, a LAN, a WAN, and/or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, edge devices including routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium within the respective computing/processing device.

The computer readable program instructions for performing the operations for the inventive arrangements described herein may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, or source or object code written in any combination of one or more programming languages, including object-oriented and/or procedural programming languages. The computer readable program instructions may include state setting data. The computer readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a LAN or WAN, or the connection may be to an external computer (e.g., through the Internet using an Internet Service Provider). In some cases, for example, an electronic circuit including a programmable logic circuit, an FPGA, or a PLA may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuit to implement aspects of the inventive arrangements described herein.

Several aspects of the inventive arrangements have been described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions, e.g., program code.

These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to manufacture a machine, such that the instructions executing through the processor of the computer or other programmable data processing apparatus create means for implementing the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium capable of directing a computer, a programmable data processing apparatus, and/or other device to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored comprises an article of manufacture that includes instructions that implement aspects of the operations specified in one or more blocks of the flowcharts and/or block diagrams.

The computer-readable program instructions may also be loaded into a computer, other programmable apparatus, or other device to cause a sequence of operations to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various aspects of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions that comprises one or more executable instructions for implementing the specified operations.

In some alternative implementations, the actions noted in the blocks may be performed out of the order noted in the figures. For example, two blocks shown in succession may be performed substantially concurrently, or the blocks may sometimes be performed in the reverse order, depending on the functionality involved. In other examples, the blocks may be performed generally in ascending numerical order, and in still other examples, one or more blocks may be performed in a varying order, with the results stored and utilized in other blocks that follow or do not immediately follow. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, may be implemented by dedicated hardware-based systems that perform the specified functions or acts or a combination of dedicated hardware and computer instructions.

The corresponding structures, materials, acts, and equivalents of all means or step and functional elements that may appear in the following claims are intended to include any structure, material, or act for performing a function in combination with other claimed elements as specifically asserted.

The description of the inventive configurations provided herein is for illustrative purposes and is not exhaustive or limited to the forms and examples disclosed. The terminology used herein is chosen to explain the principles of the inventive configurations, practical applications, or technical improvements over the art found in the marketplace, and/or to enable others skilled in the art to understand the inventive configurations disclosed herein. Modifications and variations may become apparent to those skilled in the art without departing from the scope and spirit of the inventive configurations described. Accordingly, reference should be made to the following claims, rather than to the above disclosure, as indicating the scope of such features and implementations.

Claims (15)

a communications interface configured to communicate with a host system;
a direct memory access circuit coupled to the communication interface;
A kernel circuit implemented using a programmable circuit;
a stream traffic manager circuit coupled to the direct memory access circuit and to the kernel circuit, the stream traffic manager circuit configured to control data streams exchanged between the host system and the kernel circuit;
The integrated circuit, wherein the stream traffic manager circuit is configured to set at least one of a size of a data set or an amount of prefetch data for the kernel circuits on a per kernel circuit basis. a first interconnect configured to receive packetized data from the stream traffic manager circuitry and to deliver the packetized data to the kernel circuitry;
a second interconnect configured to receive data from the kernel circuit and provide the data to the stream traffic manager circuit. an input buffer coupled to an output port of the first interconnect and to an input port of the kernel circuit, the input buffer configured to temporarily store the packetized data, convert the packetized data into a data stream, and provide the data stream to the kernel circuit;
3. The integrated circuit of claim 2, wherein the stream traffic manager circuit initiates a data transfer to the kernel circuit in response to determining that the input buffer has available space. an output buffer coupled to an output port of the kernel circuit and to an input port of the stream traffic manager circuit, the output buffer configured to temporarily store a data stream output from the kernel circuit, convert the data stream into further packetized data, and provide the further packetized data to the second interconnect;
4. The integrated circuit of claim 3, wherein the stream traffic manager circuit initiates further data transfers from the kernel circuit to the host system in response to determining that a buffer in the host system corresponding to the output buffer has available space and that the output buffer contains at least one complete packet. 5. The integrated circuit of claim 4, wherein the output buffer is configured to add tagging information to the further packetized data to identify the kernel circuit as a source kernel circuit. the kernel circuit is one of a plurality of kernel circuits implemented in the programmable circuit;
the stream traffic manager circuitry is configured to interleave data streams exchanged with the plurality of kernel circuits;
2. The integrated circuit of claim 1, wherein each kernel circuit of the plurality of kernel circuits is coupled to the stream traffic manager circuit through a buffer and an interconnect, the stream traffic manager circuit implementing a round robin arbitration scheme for streaming data to each kernel circuit of the plurality of kernel circuits based on space availability in the buffer corresponding to each kernel circuit of the plurality of kernel circuits. a first kernel circuit implemented in a programmable circuit;
a second kernel circuit implemented in the programmable circuit;
a stream traffic manager circuit coupled to the first kernel circuit and the second kernel circuit, the stream traffic manager circuit configured to control data streams exchanged between the first kernel circuit and the second kernel circuit;
the stream traffic manager circuitry is configured to set at least one of a size of a data set or an amount of prefetch data for the first kernel circuit and the second kernel circuit for each of the first kernel circuit and the second kernel circuit . The integrated circuit of claim 7, wherein the selected data stream sent from the first kernel circuit to the second kernel circuit includes in-band instructions for the second kernel circuit. the first kernel circuit is coupled to a first interconnect through a first input buffer and a first output buffer;
the second kernel circuit is coupled to a second interconnect through a second input buffer and a second output buffer;
The integrated circuit of claim 7 , wherein the first interconnect and the second interconnect are coupled to the stream traffic manager circuit. 10. The integrated circuit of claim 9, wherein the first output buffer and the second output buffer are configured to temporarily store data streams output from the first kernel circuit and the second kernel circuit, respectively, convert the data streams into packetized data, and add tagging information to the packetized data to identify at least one of a source kernel circuit or a destination kernel circuit. The integrated circuit of claim 10, wherein the selected data stream provided to the first kernel circuit or the second kernel circuit includes in-band instructions for the first kernel circuit or the second kernel circuit. The integrated circuit of claim 9, wherein the first kernel circuit and the first interconnect are located on a first die of the integrated circuit, and the second kernel circuit and the second interconnect are located on a second die of the integrated circuit. The integrated circuit of claim 12, wherein the stream traffic manager circuitry is located on the first die. the second input buffer is coupled to an input port of the second kernel circuit in the second die and configured to temporarily store data to be streamed to the second kernel circuit;
the first output buffer is coupled to an output port of the first kernel circuit in the first die and configured to temporarily store data output from the first kernel circuit;
13. The integrated circuit of claim 12, wherein the stream traffic manager circuit is configured to initiate a data transfer from the first kernel circuit to the second kernel circuit in response to determining that the second input buffer has available space and the first output buffer is storing data. a first transceiver coupled to an input/output port of the stream traffic manager circuit;
the first transceiver is configured to communicate with a second transceiver implemented in a different integrated circuit;
8. The integrated circuit of claim 7 , wherein the stream traffic manager circuit is configured to control data streams exchanged with a satellite stream traffic manager circuit coupled to a further kernel circuit implemented in the different integrated circuit via communication via the first transceiver and the second transceiver with the further kernel circuit, the satellite stream traffic manager circuit being implemented in the different integrated circuit .

Images