DE112013007842B3 - SYSTEM, DEVICE AND METHOD FOR GENERATION AND TRANSMISSION FLITS OF DIFFERENT PROTOCOLS IN DIFFERENT MODES - Google Patents

Abstract

An apparatus comprising:a circuit board comprising Peripheral Component Interconnect Express, PCIe, electrical,wherein the PCIe electrical is to implement a PCIe physical layer;first protocol stack logic to implement at least a portion of a PCIe stack,wherein the first protocol stack logic is to generate Peripheral Component Interconnect Express based packets, PCIe based packets and cause the PCIe based packets to be sent over the PCIe physical layer; andsecond protocol stack logic to implement at least a portion of another protocol stack,wherein the other protocol stack comprises a storage protocol, and the second protocol stack logic is to generate flits of a different protocol and cause the flits to be sent over the PCIe physical layer.

Description

AREA AND TASK

This disclosure relates to a system, apparatus and method for generating and sending flits of different protocols in different modes.
The object on which the present invention is based is to form a data stream from sequences which are assigned either to a PCIe protocol or to a memory protocol. This object is solved by the subject matter of independent claims 1, 10 and 15. The dependent claims describe preferred embodiments.
The pamphlet US 2013/0 163 605 A1 discloses a multi-switched network that allows data associated with multiple different protocols to be transmitted over a single, non-protocol specific connection.

BACKGROUND

Advances in semiconductor processing and logic design have enabled an increase in the amount of logic that can reside on integrated circuit devices. As a direct result, computer system configurations have evolved from a single or multiple integrated circuits to a system of multiple cores, multiple hardware threads, and multiple logical processors residing on single integrated circuits, as well as other interfaces built into such processors. A processor or integrated circuit typically has a single physical processor die, where the processor die may contain any number of cores, hardware threads, logical processors, interfaces, memory, controller hubs, and so on.

Smaller computing devices have become more popular due to the greater ability to fit more processing power into smaller packages. Smartphones, tablets, ultra-thin notebooks and other user devices have grown exponentially. However, these smaller devices rely on servers for both data storage and complex processing, exceeding the form factor. Consequently, the demand for the high performance computing market (i.e., server space) has also increased. For example, in modern servers, not only is there typically a single processor with multiple cores to increase computing power, but there are also multiple physical processors (also referred to as multiple sockets). However, as processing power increases with the number of devices in a computing system, communication between sockets and other devices becomes more critical.

In fact, interconnects have evolved from more traditional multi-drop buses that primarily handled electrical communications to fully-fledged interconnect architectures that facilitate high-speed communications. Unfortunately, as the consumption of future processors will be at even higher rates, there is a corresponding need in the capabilities of existing interconnect architectures.

character list

• 1 Figure 1 illustrates one embodiment of a computing system that includes an interconnect architecture.
• 2 Figure 1 illustrates one embodiment of an interconnect architecture that includes a layered stack.
• 3 Figure 12 illustrates one embodiment of a request or packet to be generated in an interconnect architecture.
• 4 Figure 11 illustrates one embodiment of a transmitter and receiver pair for an interconnect architecture.
• 5A Figure 12 illustrates a simplified block diagram of one embodiment of an example node.
• 5B Figure 12 illustrates a simplified block diagram of one embodiment of an example system that includes multiple nodes.
• 6 12 is an illustration of data being transferred according to an example shared memory connection.
• 7A 12 is an illustration of data being transferred according to another example shared memory connection.
• 7B Figure 12 is an illustration of an example beginning of a data framing token.
• 8th 12 is an illustration of data being transferred according to another example shared memory connection.
• 9A-9D 12 are flow charts showing example techniques for memory access messaging.
• 10 FIG. 11 illustrates one embodiment of a block diagram for a computing system that includes a multi-core processor.
• 11 Figure 11 illustrates another embodiment of a block diagram for a computing system that includes a multi-core processor.
• 12 Figure 1 illustrates one embodiment of a block diagram for a processor.
• 13 12 illustrates another embodiment of a block diagram for a computer system that includes a processor.
• 14 Figure 1 illustrates one embodiment of a block for a computing system that includes multiple processors.
• 15 illustrates an example system implemented as a system-on-chip (SoC).

Like reference numerals and symbols in the different drawings indicate like elements.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, such as specific types of processors and system configurations, specific hardware structures, specific architectural and microarchitectural details, specific register configurations, specific instruction types, specific system components, specific dimensions/heights , specific processor pipeline stages and operation, etc. However, it will be apparent to those skilled in the art that these specific details need not be used in the practice of the present invention. In other examples, well-known components or methods, such as specific and alternative processor architectures, specific logic circuits/code for described algorithms, specific firmware code, specific interconnect operations, specific logic configurations, specific manufacturing techniques and materials, specific compiler implementations, specific expression of algorithms in code, specific shutdown and control techniques/logic, and other specific operational details of a computer system have not been described in detail in order not to unnecessarily obscure the present invention.

Although the following embodiments may be described in terms of power conservation and energy efficiency in specific integrated circuits, such as computing platforms or microprocessors, other embodiments are applicable to other types of integrated circuits and logic devices. Similar techniques and teachings of embodiments described herein may be applied to other types of circuits or semiconductor devices that may also benefit from better power efficiency and power conservation. For example, the disclosed embodiments are not limited to desktop computer systems or Ultrabooks™. And can also be used in other devices such as handheld devices, tablets, other thin notebooks, system-on-chip (SOC) devices, and embedded applications. Some examples of handheld devices include cellular phones, Internet protocol devices, digital cameras, personal digital assistants (PDAs), and handheld PCs. Embedded applications typically contain a microcontroller, digital signal processor (DSP), system on a chip, network computer (NetPC), set-top boxes, network hubs, wide area network (WAN) switches, or any other system that requires the following functions and operations taught. Furthermore, the apparatus, methods, and systems described herein are not limited to physical computing devices, but may also relate to software optimizations for energy conservation and efficiency. As will be immediately apparent in the following description, the embodiments of methods, apparatus and systems (whether in terms of hardware, firmware, software or a combination thereof) are essential to a 'green technology' future balanced with performance considerations.

With the further development of computer systems, the components they contain are becoming more complex. As a result, the complexity of the interconnect architecture for coupling and communication between the components to ensure that bandwidth requirements are met for optimal component operation also increases. Furthermore, different market segments require different aspects of interconnect architectures to meet market needs. For example, servers require higher performance, while the mobile ecosystem can sometimes trade off overall performance for the sake of power savings. Nevertheless, it is a singular task of most matrices to offer the highest possible performance with a maximum power saving. Various intermediates are discussed below that would potentially benefit from aspects of the invention described herein.

An interconnect matrix architecture includes the Peripheral Component Interconnect (PCI) Express (PCIe) architecture. A primary goal of PCIe is to allow operation between components and devices from different vendors in an open architecture, spanning multiple market segments; Clients (desktops and mobile), servers (standard and proprietary) and embedded and communication devices. PCI Express is a high-performance, general-purpose, I/O interconnect defined for a wide range of future computing and communications platforms. Some PCI attributes, such as its utility model, load-memory architecture, and software interfaces, were retained through its revisions, while previous parallel bus implementations were replaced with a highly scalable, fully serial interface. Recent versions of PCI Express take advantage of advances in point-to-point interconnects, switch-based technology, and packetized protocol to provide new levels of performance and features. Performance management, quality of service (QoS), hot-plug/hot-swap support, data integrity and fault management are among the advanced features supported by PCI Express.

With reference to 1 Figure 1 shows an embodiment of a matrix composed of point-to-point links connecting a set of components. System 100 includes a processor 105 and system memory 110 coupled to a controller hub 115 . Processor 105 includes any processing element such as a microprocessor, host processor, embedded processor, co-processor, or other processor. The processor 105 interfaces to the controller hub 115 through a front-side bus (FSB) 106. In one embodiment, the FSB 106 is a point-to-point serial interconnect, as described below. In another embodiment, link 106 includes a serial, differential interconnect architecture that conforms to various interconnect standards.

System memory 110 includes any storage device, such as random access memory (RAM), non-volatile (NV) memory, or other memory accessible to devices in system 100 . The system memory 110 is coupled to the controller hub 115 through a memory interface 116 . Examples of a memory interface include a double data rate (DDR) memory interface, a dual channel DDR memory interface, and a dynamic RAM (DRAM) memory interface.

In one embodiment, controller hub 115 is a root hub, root complex, or root controller in a Peripheral Component Interconnect Express (PCIe or PCIE) interconnect hierarchy. Examples of the controller hub 115 include a chipset, a memory controller hub (MCH), a northbridge, an interconnect controller hub (ICH), a southbridge, and a root controller/hub. Often the term chipset refers to two physically separate controller hubs, i.e., a memory controller hub (MCH) coupled to an interconnect controller hub (ICH). Note that current systems often include the MCH integrated with the processor 105, while the controller 115 can communicate with I/O devices in the same manner as described below. In some embodiments, peer-to-peer routing through root complex 115 is optionally supported.

Here the controller hub 115 is coupled to a switch/bridge 120 by a serial link 119 . Input/output modules 117 and 121, which may also be referred to as interfaces/ports 117 and 121, contain/implement a layered protocol stack for providing communication between controller hub 115 and switch 120. In one embodiment, multiple devices can connect to the switch 120 are coupled.

The switch/bridge 120 routes packets/messages from the device 125 upstream, ie up one hierarchy towards a root complex, to the controller hub 115 and downstream, ie down one hierarchy away from the root complex, from processor 105 or system memory 110 to device 125. Switch 120 is referred to in one embodiment as a logical assembly of multiple virtual PCI-to-PCI bridging devices. The device 125 includes any internal or external device or component that is to be coupled to an electrical system, such as an I / O device, a network interface controller (NIC), an expansion card, an audio processor, a network processor, a hard disk drive, a Storage device, CD/DVD ROM, monitor, printer, mouse, keyboard, router, portable storage device, Firewire device, Universal Serial Bus (USB) device, scanner and other input/output devices. Such a device is often referred to as an endpoint in PCIe jargon. Although not specifically shown, device 125 may include a PCIe to PCI/PCI-X bridge to support legacy or other version PCI devices. endpoint devices in PCIe are often classified as legacy, PCIe, or root complex integrated endpoints.

A graphics accelerator 130 is also coupled to controller hub 115 through serial link 132 . In one embodiment, graphics accelerator 130 is coupled to an MCH that is coupled to an ICH. The switch 120, and thus the I/O device 125, is then coupled to the ICH. I/O modules 131 and 118 are also intended to implement a layered protocol stack for graphics accelerator 130 and controller hub 115 communication. Similar to the previous discussion of the MCH, a graphics controller or graphics accelerator 130 may be integrated into the processor 105 itself.

With regard now to 2 An embodiment of a layered protocol stack is shown. Layered protocol stack 200 includes any form of layered communications stack, such as a Quick Path Interconnect (QPI) stack, a PCIe stack, a next generation high performance computing interconnect stack, or other layered stack. Although the discussion immediately following may refer to 1-4 refers to a PCIe stack, the same concepts can be applied to other interconnect stacks. In one embodiment, protocol stack 200 is a PCIe protocol stack that includes transaction layer 205 , link layer 210 , and physical layer 220 . An interface, such as interfaces 117, 118, 121, 122, 126, and 131 in 1 , can be represented as a communication protocol stack 200 . The communications protocol stack representation may also be referred to as a module or interface that implements/contains a protocol stack.

PCI Express uses packets to communicate information between components. In the transaction layer 205 and data link layer 210, packets are formed to carry the information from the sending component to the receiving component. As the packets sent pass through the other layers, they are augmented with additional information needed to handle packets in those layers. At the receiving end, the reverse process occurs and the packets are converted from their physical layer 220 representation to the data link layer 210 representation and ultimately (for transaction layer packets) to the form that can be processed by the transaction layer 205 of the receiving device.

transaction layer

In one embodiment, the transaction layer 205 serves to provide an interface between a processing core of a device and the interconnect architecture, such as the data link layer 210 and the physical layer 220. In this regard, a primary responsibility of the transaction layer 205 is the assembly and disassembly of packets (i.e., transaction layer packets or TLP's). The translation layer 205 typically manages the credit-based flow control for TLPs. PCIe implements split transactions, i.e., transactions in which request and response are separated in time, allowing one connection to carry other traffic while the target device collects data for the response.

In addition, the PCIe uses the Credit-Based Flow Control. In this scheme, a device advertises an initial amount of credits to each of the receive buffers in the transaction layer 205. An external device at the opposite end of the connection, such as the controller hub 115 in 1 , counts the number of credits consumed by each TLP. A transaction can be sent if the transaction does not exceed a credit limit. Upon receipt of a response, the credit amount is restored. An advantage of a credit scheme is that the latency of credit return does not affect performance provided the credit limit is not reached.

In one embodiment, four transaction address spaces include a configuration address space, a memory address space, an input/output address space, and a message address space. Memory space transactions include one or more read requests and write requests to transfer data to/from a memory mapped location. In one embodiment, memory space transactions are capable of using two different address formats, e.g., a short address format, such as a 32-bit address, or a long address format, such as a 64-bit address. Configuration space transactions are used to access the configuration space of PCIe devices. Transactions to the configuration space include read requests and write requests. Message space transactions (or simply messages) are defined to support in-band communication between PCIe agents.

Therefore, in one embodiment, transaction layer 205 assembles packet header/payload 206 . The format for the current packet header rows/payloads can be found in the PCIe specification at the PCIe specifications website.

With reference to 3 1 briefly illustrates one embodiment of a PCIe transaction descriptor. In one embodiment, transaction descriptor 300 is a mechanism for conveying transaction information. In this regard, transaction descriptor 300 assists in identifying transactions in a system. Other possible applications include tracking modifications to a given order of transactions and associating a transaction with channels.

The transaction descriptor 300 includes a global identifier field 302, an attribute field 304, and a channel identifier field 306. In the example shown, the global identifier field 302 is shown with a local transaction identifier field 308 and source identifier field 310. FIG. In one embodiment, the global transaction identifier 302 is unique for all outstanding requests.

According to one implementation, the local transaction identifier field 308 is a field generated by a requesting agent and is unique to all outstanding requests that need to be completed for that requesting agent. Also in this example, the source identifier 310 uniquely identifies the requesting agent within a PCIe hierarchy. Therefore, the local transaction identifier 308 field, together with the source ID 310, provides a global identification of a transaction within a hierarchy domain.

Attribute field 304 specifies characteristics and relationships of the transaction. In this regard, attribute field 304 may be used to provide additional information that allows modification of the default handling of transactions. In one embodiment, attribute field 304 includes a priority field 312, a reserved field 314, an enqueue field 316, and a no-snoop field 318. Here, a priority subfield 312 may be modified by an initiator to prioritize the transaction. The reserved attribute field 314 remains reserved for future or seller-defined use. Possible usage models using priority or security attributes can be implemented using the reserved attribute field.

In this example, a ranking attribute field 316 is used to pass optional information specifying the type of ranking that may modify default ranking rules. According to an example implementation, a sequencing attribute of "0" indicates that predetermined sequencing rules apply, with a sequencing attribute of "1" indicating relaxed sequencing, where writes can traverse writes in the same direction and read completions can traverse writes in the same direction. The snoop attribute field 318 is used to determine whether transactions are snooped. As illustrated, the channel ID field 306 identifies a channel with which a transaction is associated.

connection layer

Link layer 210, also referred to as data link layer 210, serves as an intermediary between transaction layer 205 and physical layer 220. In one embodiment, a responsibility of data link layer 210 is to provide a reliable mechanism for exchanging transaction layer packets (TLPs) between two components of a link. One side of the data link layer 210 accepts TLPs assembled by the transaction layer 205, gives a packet sequence identifier 211, i.e., an identification number or packet number, calculates and applies an error detection code, i.e. CRC 212, and submits the modified TLPs to the physical layer 220 for transmission via a physical to an external device.

physical layer

In one embodiment, the physical layer 220 includes a logical sub-block 221 and an electrical sub-block 222 for physically sending a packet to an external device. Here the logical sub-block 221 is responsible for the “digital” functions of the physical layer 221. In this regard, the logical sub-block includes a transmit portion for preparing outbound information to be transmitted by physical sub-block 222 and a receive portion for identifying and preparing received information before passing it to link layer 210 .

The physical block 222 contains a transmitter and a receiver. The sender is supplied symbols from logic sub-block 221, which the sender serializes and sends to an external device. The receiver receives the serialized symbols from an external device and converts the received signals into a bit stream. The bit stream is deserialized and passed to logic sub-block 221. In one embodiment, an 8b/10b broadcast code is used, with ten-bit symbols being transmitted/received. Special symbols are used here to add frames 223 to a packet. Additionally, in one example, the receiver also provides a symbol clock obtained from the incoming serial stream.

As noted above, although transaction layer 205, link layer 210, and physical layer 220 are discussed in terms of a specific embodiment of a PCIe protocol stack, a layered protocol stack is not so limited. In fact, any layered protocol can be included/implemented. As an example, a port/interface represented as a layered protocol includes: (1) a first layer for assembling packets, i.e., a transaction layer; a second layer for queuing packets, i.e., a link layer; and a third layer for sending the packets, i.e. a physical layer. As a specific example, a layered Common Standard Interface (CSI) protocol is used.

With reference to below 4 Figure 1 shows one embodiment of a serial point-to-point PCIe matrix. Although one embodiment of a PCIe point-to-point serial link is illustrated, a point-to-point serial link is not so limited as it includes any transmit path for sending serial data. In the illustrated embodiment, a basic PCIe link includes two distinct low voltage driven signal pairs: a transmit pair 406/411 and a receive pair 412/407. Therefore, device 405 includes transmit logic 406 for transmitting data to device 410 and receive logic 407 for receiving data from device 410. In other words, in a PCIe connection there are two transmit paths, ie, paths 416 and 417, and two Receive paths, ie, paths 418 and 419 included.

A transmission path refers to any path for sending data, such as a transmission line, copper line, optical line, wireless communication channel, infrared communication link, or other communication path. A link between two devices, such as device 405 and device 410, is referred to as a link, such as link 415. A link may support one lane - each lane representing a set of distinct signal pairs (one pair to transmit, one pair to receive). To scale bandwidth, a link may aggregate multiple lanes, denoted by xN, where N is any supported link width, such as 1, 2, 4, 8, 12, 16, 32, 64, or wider.

A different pair refers to two transmission paths, such as lines 416 and 417, for transmitting different signals. As an example, when line 416 transitions from a low voltage level to a high voltage level, i.e., a rising edge, line 417 transitions from a high logic level to a low logic level, i.e., a falling edge. Different signals may exhibit better electrical properties, such as better signal integrity, i.e., cross-coupling, voltage overshoot/undershoot, ringing, etc. This allows for a better timing window that enables faster transmit frequencies.

Physical layers of existing interconnect and communications architectures, including PCIe, can be built to provide shared storage and I/O services in a system. Typically, cacheable memory cannot be shared between independent systems using conventional load/store (LD/ST) memory semantics. An independent system, or "node," can be independent in the sense that it functions as a single logical entity, is controlled by a single operating system (and/or a single BIOS or Virtual Machine Monitor (VMM)) and /or has an independent fault domain. A single node may contain one or more processor devices, may be implemented on a single board or multiple boards, and may contain local memory, including cacheable memory, accessed using LD/ST semantics by the devices on it node can be accessed. In a node, shared memory may include one or more blocks of memory, such as random access memory (RAM) that can be accessed by multiple different processors (e.g., central processing units (CPUs)) in a node. A shared memory can also include the local memory of the processors or other devices in the node. The multiple devices in a shared memory node can share a single view of data in the shared memory. The I/O communication involving the shared memory can have very low latency and allow rapid access to the memory by the multiple processors.

Traditionally, memory sharing between different nodes has not enabled memory sharing according to a load/memory paradigm. For example, in some systems, memory sharing between different nodes has been facilitated by distributed memory architectures. In In conventional solutions, computing tasks operate on local data and when data from another node is desired, the computing task (e.g. being executed by another CPU node) communicates with the other node via, for example, a communication channel using a communication protocol stack such as Ethernet, InfiniBand or a different layered protocol used. In conventional multi-node systems, the processors of different nodes do not need to be aware of where the data resides. Data sharing using traditional methods, such as via a protocol stack, may have significantly higher latency than memory sharing in a node using a load/store paradigm. Instead of directly addressing and manipulating data in shared memory, one node can, among other examples, request data from another using an existing protocol handshake such as Ethernet (or Infiniband) and the source node can provide the data so that the Data can be stored and edited by the requesting node.

In some implementations, a shared memory architecture may be provided that enables memory sharing between independent nodes for exclusive or shared access using load/store (LD/ST) memory semantics. In one example, memory semantics (and directory information, if applicable) and I/O semantics (for protocols like PCIe) can be exported to either a common set of pins or a separate set of pins. In such a system, each of multiple nodes in a system can use the enhanced shared memory architecture to maintain its own independent fault domain (and local memory) while using a shared memory pool for node-by-node access and low-latency messaging between Node runs using memory that allows LD/ST semantics. In some implementations, such a shared memory pool can be allocated dynamically (or statically) between different nodes. Therefore, the various nodes of a system can be configured as dynamically changing groups of nodes that work together and flexibly on different tasks as needed, using, for example, the shared storage infrastructure.

With regard now to 5A Illustrated is a simplified block diagram 500a showing an example system that includes a shared memory 505 that can be accessed using load/store techniques from each of multiple independent nodes 510a-510n. For example, a shared memory controller 515 can be provided that can accept load/memory access requests from the various nodes 510a-510n on the system. The shared memory 505 may be implemented using synchronous dynamic random access memory (SDRAM), dual in-line memory modules (DIMM), and other non-volatile (or volatile) memory.

Each node may have its own CPU socket(s) and may also contain local memory that remains isolated from LD/ST access by other nodes in the system. The node may communicate with other devices on the system (e.g. shared memory controller 515, network controller 520, another node, etc.) using one or more protocols including PCIe, QPI, Ethernet, among other examples. In some implementations, a shared memory link (SML) protocol may be provided through which low latency LD/ST memory semantics may be supported. For example, SML can be used in the communication of reads and writes of a shared memory 505 (through the shared memory controller 515) from the various nodes 510a-510n of a system.

In one example, the SML may be based on a memory access protocol such as 3rd Generation (SM13) Scalable Memory Interconnect (SMI). Alternatively, other memory access protocols may be used, such as transactional memory access protocols such as fully buffered DIMM (FB-DIMM), DDR transactional (DDR-T), among other examples. In other cases, the SML may be based on native PCIe memory read/write semantics with additional directory extensions. A memory protocol based implementation of the SML can provide bandwidth efficiency benefits as it is tailored for cache line memory accesses. While high-performance communication protocols exist between devices, such as PCIe, upper layers (e.g., transaction and link layers) of such protocols can introduce latency that degrades the application of the full protocol for use in LD/ST memory transactions, including transactions involving shared memory 505 include. A storage protocol, such as SMI3, can provide a potential additional benefit by providing low-latency approaches be offered as it can bypass most of another protocol stack, such as PCIe. Therefore, SML implementations can leverage SMI3 or another storage protocol running on a different protocol logical and physical PHY, such as SMI3 on PCIe.

As noted, in some implementations a shared memory controller (SMC) 515 may be provided that includes logic to handle load/memory requests from nodes 510a-510n in the system. Load/store requests may be received by the SMC 515 over links using the SML and connecting nodes 510a-510n to the SMC 515. In some implementations, SMC 515 may be implemented as a device, such as an application specific integrated circuit (ASIC), including logic for servicing nodes 510a-510n's access requests for shared memory resources. In other cases, SMC 515 (as well as shared memory 505) may reside on a device, chip, or board separate from one or more (or even all) of nodes 510a-510n. The SMC 515 may also include logic for coordinating various transactions of the nodes that include the shared memory 505 . In addition, the SMC can maintain directory tracking access to various data resources, such as each cache line, contained in shared memory 505. For example, a data resource may, among other possible examples, be in a shared access state (e.g., capable of being accessed (e.g., loaded or read) by multiple processing and/or I/O devices in a node concurrently), an access-only state (e.g., exclusively, unless temporarily reserved (e.g., for a store or write) by a single processing and/or I/O device in a node, can be a non-cache state. Furthermore, while each node may have direct access to one or more sections of shared memory 505, different addressing schemes and values may be used by the different nodes (e.g. 510a-510n), resulting in the same being shared used memory is referenced (e.g. in an instruction) by a first node according to a first address value and a second node referencing the same data by a second address value. The SMC 515 may contain logic that includes data structures that map addresses of the nodes to shared memory resources so that the SMC 515 can interpret the different access requests of the different nodes.

Additionally, in some cases, some portion of the shared memory (e.g., certain separations, memory blocks, records, files, etc.) may be subject to certain permissions, rules, and allocations such that only a portion of nodes 510a-510n (e.g., by the SMC 515 ) can access corresponding data. In fact, each shared memory resource may be associated with a corresponding (and in some cases different) subset of the system's nodes 510a-510n. These allocations may be dynamic, and SMC 515 may modify such rules and permissions (e.g., on demand, dynamically, etc.) to accommodate new or changed rules, permissions, node allocations, and ownership applicable to a particular portion of shared memory 505 .

An example SMC 515 may also track various transactions involving nodes (e.g., 510a-510n) in the system that access one or more shared memory resources. For example, the SMC 515 may track information for each shared memory 505 transaction, including an identification of the node(s) involved in the transaction, the progression of the transaction (e.g., whether it completed), along with other transaction information. This may allow some of the transaction-oriented aspects of traditional distributed memory architectures to be applied to the enhanced multi-node shared memory architecture described herein. Additionally, transaction tracking (e.g., from the SMC) may be used to help maintain or enforce each respective node's separate and independent fault domains. For example, the SMC can maintain the appropriate node ID for each ongoing transaction in its internal data structures, including in memory, and use this information to enforce access rights and manage individual fault domains for each node. Therefore, if one of the nodes goes down (e.g. due to a critical error, a triggered recovery sequence or some other error or event), only that node and its transactions involving the shared memory 505 will be interrupted (e.g. dropped by the SMC) - Transactions of the remaining nodes that hold the shared memory 505 continue regardless of the failure in the other node.

A system can contain multiple nodes. Additionally, some example systems may include multiple SMCs. In some cases, a node may be able to share one to access used memory beyond a remote SMC to which it is not directly connected (ie, the node's local SMC is connected to the remote SMC by one or more SML connection hops). The remote SMC can be on the same board or could be on a different board. In some cases, some of the nodes may be off-system (eg, off-board or off-chip) but still access shared memory 505 . For example, one or more off-system nodes may be directly connected to the SMC using an SML-compliant connection, among other examples. Additionally, other systems containing their own SMC and shared memory may also be connected to the SMC 510 to extend memory 505 sharing to nodes contained, for example, on another board that interfaces with of the other SMC that is connected to the SMC via an SML connection. Furthermore, network connections may be tunneled through to extend access to the other nodes off-board or off-chip. For example, the SML may tunnel over an Ethernet connection (e.g., provided by the network controller 520) that the example system of FIG 5A communicatively coupled to another system that also includes one or more other nodes, and also allow those nodes access to the SMC 515 and thereby to the shared memory 505, among other examples.

As another example, as shown in simplified block diagram 500b of FIG 5B 1, an improved shared memory architecture that allows shared access by multiple independent nodes according to LD/ST memory semantics can flexibly allow for a variety of different multi-node system designs. Various combinations of the multiple nodes may be allocated to share portions of a shared memory block or blocks provided in an example system. For example, another example system shown in the example of FIG 5B illustrated, contain multiple devices 550a-550d, implemented, for example, as separate dies, boards, chips, etc., with each device containing one or more independent CPU nodes (eg, 510a-510h). Each node can contain its own local storage. One or more of the multiple devices 550a-550d may further include shared memory accessible by two or more of the nodes 5 10a-5 10h of the system.

This in 5B The system illustrated is an example intended to illustrate some of the variability that can be achieved through an improved shared memory architecture as illustrated and described herein. For example, each of device A 550a and device C 550c may include a corresponding shared memory element (eg, 505a, 505b). Therefore, in some implementations, each shared memory element on a distinct device may further include a corresponding shared memory controller (SMC) 515a, 515b. Various combinations of nodes 510a-510h may be communicatively coupled to each SMC (eg, 515a, 515b), allowing the nodes to access the corresponding shared memory (eg, 505a, 505b). As an example, the SMC 515a of device A 550a may be connected to the nodes 510a, 510b on device A using a direct data link that supports SML. Additionally, other nodes 510c on another device (e.g., Device C 550c) may also have access to shared memory 505a through a direct, hardwired connection (that supports SML) from node 510c (and/or its device 550c) to SMC 515a. Indirect, network-based, and other such connections may also be used to allow nodes (e.g., 510f-510h) of a remote device or off-board (e.g., Device D 550d) to use a conventional protocol stack to interface with the SMC 515a to access shared memory 505a as well. For example, an SML tunnel 555 may be established over an Ethernet, InfiniBand, or other connection that couples device A and device D. During establishment and maintenance, the tunnel may introduce some additional overhead and latency compared to SML running on other, less software-managed, physical connections, with SML tunnel 555, when established, like other SML channels and allow node 510f-5 10h to interface with SMC 515a over SML and access shared memory 505a as any other node that communicates with SMC over an SML connection can. For example, reliability and ordering of the packets in the SML channels can either be enforced by the networking components in the system or can be enforced end-to-end between the SMCs.

In yet another example, nodes (e.g., 515d, 515e) may reside on a device other than the one that has a particular portion of shared memory (e.g., 505a) may be indirectly connected to the corresponding SMC (e.g. SMC 515a) through a direct connection to another SMC (e.g. 515b) which is itself coupled (e.g. using an SML connection) to the corresponding SMC (e.g. 515a). Linking two or more SMCs (eg, 515a, 515b) can effectively increase the amount of available shared memory available to nodes 510a-510h on the system. For example, due to a connection between SMCs 515a, 515b in the example of 5B in some implementations, each of the nodes (e.g., 510a-510c, 510f-510h) that can access a shared memory 505a through the SMC 515a may also access a shared memory 505b due to the connection between the SMC 515a and the Access SMC 515b. Likewise, in some implementations, each of the nodes that can directly access the SMC 515b can also access a shared memory 505a due to the connection between the SMCs 515a, 515b, among other possible examples.

As noted above, an improved shared memory architecture may include and provide a low latency connection protocol (i.e., SML) based memory access protocol such as SMI3 to facilitate load/memory requests involving the shared memory. While the traditional SMI3 and other memory access protocols can be configured for use in memory sharing within a single node, the SML can extend the memory access semantics to multiple nodes to enable memory sharing between the multiple nodes. Furthermore, the SML can potentially be used on any physical communication link. The SML may use a memory access protocol that supports LD/ST memory semantics overlaid on a physical layer (and corresponding physical layer logic) adapted to interconnect different devices (and nodes). Additionally, SML physical layer logic may not provide packet loss and error retry functionality, among other features.

In some implementations, the SML can be implemented by overlaying the SMI3 on a PCIe PHY. An SML link layer may be provided (e.g., in place of a traditional PCIe link layer) to eliminate flow control and other features and facilitate low-latency memory access that would be characteristic of traditional CPU memory access architectures. In one example, the SML link layer logic may multiplex between shared memory transactions and other transactions. For example, the SML link layer logic can multiplex between SMI3 and PCIe transactions. For example, the SMI3 (or other storage protocol) may reside on a PCIe (or other interconnect protocol) so that the link can dynamically switch between SMI3 and PCIe transactions. This allows traditional PCIe traffic to coexist on the same link as SML traffic in some cases.

With regard now to 6 A diagram 600 is shown that illustrates a first implementation of SML. For example, the SML can be implemented by overlaying an SMI3 on a PCIe PHY. The physical layer can use a standard PCIe 128b/130b that encodes all physical layer activities, including link training as well as PCIe data blocks. The SML can provide traffic on the lanes (eg Lane0 - Lane7) of the connection to be multiplexed between PCIe packets and SMI3 flits. For example, in the in 6 implementation shown, the sync header of the PCIe 128b/130b encoding will be modified and used to indicate that SMI3 flits and not PCIe packets are sent on the lanes of the link. In traditional PCIe 128b/130b encoding, valid sync headers (e.g. 610) can be sending either a 10b pattern on all lanes of the link (indicating that the payload type of the block should be the PCIe data block) or a 01b pattern on all lanes of the connection (to indicate that the type of payload of the block should be the PCIe queued block set). In an example of an SML, an alternate sync header may be defined to distinguish SMI3 flit traffic from PCIe frames and queued records. In an example found in 6 As shown, the PCIe 128b/130b sync header (eg, 605a, 605b) may be encoded with alternating 01b, 10b patterns on odd/even tracks to indicate that SMI3 flits should be sent. In another alternative implementation, the 128b/130b sync header encoding SMI3 traffic may be defined by alternating 10b, 01b patterns on odd/even tracks, among other exemplary encodings. In some cases, SMI3 flits can be sent immediately after the SMI3 Sync header on a per byte basis, with the transition between PCIe and SMI3 protocols occurring at the block boundary.

In some implementations, like those shown in the example of 6 , the transition between protocols can be defined to occur at the block boundary, whether it is an SMI3 flit or PCIe packet boundary is equivalent to. For example, a block can be defined to contain a predefined amount of data (eg, 16 symbols, 128 bytes, etc.). In such implementations, if the block boundary does not conform to an SMI3 flit or PCIe packet boundary, transmission of an entire SMI3 flit may be interrupted. An interrupted SMI3 flit can be resumed in the next SMI3 block, indicated by sending another Sync Header encoded for SMI3.

With regard now to 7A A diagram 700 is shown that illustrates another example implementation of an SML. In the example of 7A Physical layer framing tokens can be used instead of using a specialized sync header that encodes signal transitions between memory access and interconnect protocol traffic. A framing token (or "token") may be a physical layer data encapsulation that specifies or implies the number of symbols to be contained in a data stream associated with the token. Thus, the framing token can indicate that a stream begins as well as imply where it will end, and therefore can also be used to indicate the location of the next framing token. A framing token of a data stream can be located in the first symbol (symbol 0) of the first track (eg track 0) of the first data block of the data stream. In the example of PCIs, five framing tokens may be defined, which are the TLP Start of Traffic (STP) token, End of Data Stream (EDS) token, Bad End (EDB) token, DLLP Start (SDP) token, and Logical Idle (IDL) tokens included.

In the example of 7A the SML can be implemented by overlaying an SMI3 or other data access protocol on a PCIe and the standard PCIe STP token can be modified to define a new STP token indicating that SMI3 (instead of TLP traffic) on the trail of the compound will begin. In some examples, spare bit values of the standard PCIe STP token may be modified to define the SMI3 STP token in SML. As also in 7B As illustrated, an STP token 705 may contain multiple fields, including a 710 field specifying the length of the SMI3 payload (in terms of the number of flits) that will follow. In some implementations, one or more default payload lengths may be defined for TLP data. SMI3 data may be defined to contain a fixed number of flits in some implementations, or contain a variable number of flits in other cases, in which case the length field for the number of SMI3 flits becomes a field that is neglected can be. Furthermore, the length field for an SMI3 STP can be defined as a length different from that of the defined TLP payload lengths. Therefore, as an example, an SMI3 STP may be specified based on a non-TLP length value present in the STP length field. For example, in one implementation, the upper 3-bits of the 11-bit STP length field may be set to 111b to indicate the SMI3 packet (e.g., based on the assumption that no spec-compliant PCIe TLP can be long enough to have a length where the upper 3 bits of the length field would each result in "1"). Other implementations may alter or encode other fields of the STP token to distinguish a PCIe STP token that identifies a traditional PCIe TLP data payload from an SMI3 STP token that indicates SMI3 data encapsulated in TLP data are.

Referring again to the example of 7A sync header data follows the encoding specified for traditional PCIe 128b/130b encoding. For example, at 715a-c, sync headers are received with a value of 10b indicating that data blocks are forthcoming. When a PCIe STP (e.g. 720) is received, a PCIe TLP payload is expected and the data stream is processed accordingly. In accordance with the payload length identified in the PCIe STP 720, the PCIe TLP payload can use the full allocated payload length. Another STP token may be received at substantially the same time within a data block following the end of the TLP payload. For example, at 725, an SMI3 STP may be received signaling a transition from PCIe TLP data to SMI3 flit data. For example, the SMI3 STP can be sent once an end of the PCIe packet data is identified.

Continuing with the example of 7A As with the PCIe TLP data, the SMI3 STP 725 can define a length of the SMI3 flit payload to follow. For example, the payload length of the SMI3 data may correspond to the number of SMI3 flits in terms of DW to follow. A window (e.g. ending at symbol 15 of track 3) corresponding to the payload length can thereby be defined on the tracks in which only SMI3 data is to be sent during the window. When the window closes, other data may be sent, such as another PCIe STP that starts sending TLP data again, or other data, such as a queued record. As for example in the example of 7A As shown, an EDS token is sent after the end of the SMI3 data window defined by the SMI3 STP token 725 . The EDS token can signal the end of the data stream and imply that a lined block set will follow, as in the case of the example from 7A . A sync header 740 encoded 01b is sent to indicate that a queued block set is to be sent. In this case, a queued PCIe SKP record is sent. Such queued sets can be sent periodically or at set intervals or windows so that various PHY level tasks and coordination can be performed, including initializing bit sufficiency, initializing symbol alignment, exchanging PHY parameters, one Compensating for different bit rates for two communicating ports, among other examples. In some cases, an ordered queued sentence may be sent to break a defined window or data block specified for SMI3 flit data by a corresponding SMI3 STP token.

Although in the example of 7A not explicitly shown, an STP token can also be used for a transition from SMI3 flit data on the link to PCIe TLP data. For example, after the end of a defined SMI3 window, a PCIe STP token (eg, similar to token 720) may be sent to indicate that the next window is for sending a specified amount of PCIe TLP data.

Memory access flits (eg, SMI3 flits) may vary in size in some embodiments, making it difficult to predict how much data in the corresponding STP token (eg, SMI3 STP token) to reserve for the memory access payload. As an example, as in 7 shown, the SMI3 STP 725 may have a length field indicating that 244 bytes of SMI3 data are expected to follow the SMI3 STP 725. In this example, however, only ten flits (eg, SMI3 flits 0-9) are ready to be sent during the window, and these ten SMI3 flits use only 240 of the 244 bytes. Therefore, four (4) bytes of empty bandwidth are left and these are filled with IDL tokens. This can be suboptimal especially when PCIe TLP data is queued waiting for the SMI3 window to close. In other cases, the window allotted for sending SMI3 flits may be insufficient to send the amount of SMI3 data that is ready for the trace. Arbitration techniques can be used to determine how to arbitrate between SMI3 and PCIe TLP data coexisting on the link. Furthermore, in some implementations, the length of the SMI3 windows can be dynamically modified to support more efficient use of the connection. For example, decision or other logic may monitor how well the defined SMI3 windows are being used to determine whether the defined window length can be better optimized given the amount of SMI3 (and competing PCIe TLP traffic) expected for the lane. Therefore, in such implementations, the length field values of SMI3 STP tokens may vary depending on the amount of link bandwidth that SMI3 flit data should be allocated to (e.g., relative to other PCIe data including TLP, DLLP data, and a queued record). other examples, can be adjusted dynamically (e.g. between different values).

With regard now to 8th An illustration 800 of another example implementation of an SML is presented. In this alternate embodiment, the SML may provide for interleaved SMI3 and PCIe protocols through a modified PCIe framing token. As noted above, an EDS token in PCIe can be used to indicate an end of a data stream and to indicate that the next block will be a queued block set. In the example of 8th the SML can define an SMI3 EDS token (eg 805) that indicates the end of a TLP data stream and the transition to SMI3 flit transmissions. An SMI3 EDS (e.g. 805) can be defined by encoding part of the reserved bits of the traditional EDS token to indicate that SMI3 data will follow and not unqueued PCIe sets or other data following a PCIe EDS. Unlike the traditional EDS token, the SMI3 EDS can be sent essentially anywhere in a PCIe data block. This may allow additional flexibility in sending SMI3 data and ingesting corresponding low latency shared memory transactions. For example, a transition from PCIe to SMI3 can be done with a single double word (DW) of overhead. Furthermore, as with traditional EDS tokens, an example SMI3 EDS may not specify a length associated with the SMI3 data to follow the token. After an SMI3 EDS, PCIe TLP data can end and SMI3 flits can continue on the link. SMI3 traffic can continue until the SMI3 logic returns control to the PCIe logic. In some implementations, sending an SMI3 EDS causes control to pass from PCIe logic to SMI3 logic provided on devices attached to the link, for example.

In one example, SMI3 (or another protocol) may define its own link control signaling for use in performing link layer control. For example, in one implementation, the SML may define a specialized version of an SMI3 link layer control (LLCTRL) flit (eg, 810) that indicates a transition from SMI3 back to PCIe protocol. As with the SMI3 EDS the defined LLCTRL flit (eg, 810) causes control to flow back from the SMI3 logic to the PCIe logic. In some cases, as in the example of 8th As shown, the LLCTRL flit (eg, 810) may be provisioned with a predefined number of LLCTRL idle (LLCTRL-IDLE) flits (eg, 815) before the transition to PCIe is completed. For example, the number of LLCTRL-IDLE flits 815 sent to equip the SMI3 LLCTRL flit 810 may depend on the latency to decode the defined SMI3 LLCTRL flit 810 signaling the transition. After completing the transition back to PCIe, an STP packet can be sent and TLP packet data can begin again on the link under PCIe control.

It should be understood that the implementations described herein are provided as examples to demonstrate certain principles and features that are disclosed in the specification. It should be understood that alternative configurations, protocols, and architectures (besides those specifically discussed in the examples) may use and apply such principles and features. As an example of an alternative, a PCIe memory read/write (e.g. instead of an SMI3 protocol) augmented with distortion information can be used. The directory information can be implemented by spare bits of the PCIe packet. In another example, the CPU node may use a cache controller (e.g., as an alternative to a shared memory controller) to send memory read/write transactions on a PCIe link, e.g. based on a remote address range check, among others possible examples and alternatives.

With regard now to 9A-9D Flowcharts 900a-d are shown showing example techniques for communicating using an MCPL. For example, in 9A a load/store memory access message is received 905 from a first node, the message requesting particular shared memory data. Access to the particular data may be provided 910 for the first node. A second load/memory access message may be received from a second independent node 915. The second message may be a request for access to the same particular shared memory data and access to the particular data may be provided 920 for the second node. Data in shared memory may thus be shared and accessible by multiple different independent nodes.

In the example of 9B a first sync header (such as a PCIe sync header) with a first encoding may be received 925. The encoding may indicate a transition from an interconnect protocol to a memory access protocol and the transition may be identified from the first sync header 930. Data of the memory access protocol may be received after the first sync header and the data may be processed 935 (eg, in accordance with the memory access protocol). In some examples, the memory access log data may include transactions involving shared memory shared by multiple independent nodes. A second sync header may be received 940 containing a second, different encoding indicating a transition from the interconnect protocol. The transition from memory access protocol back to interconnect protocol can be identified from the second sync header 945.

With regard now to 9C In some cases, a first data start token (eg, a PCIe STP token) may be received 950 that includes one or more values encoded to identify a transition from an interconnect protocol to a memory access protocol. Memory access protocol data may arrive after the first data start token and may be identified 955. The memory access protocol data may be processed 960. A length field may be included in the first data start token that indicates when data returns to interconnect protocol data. Indeed, in some implementations, the length field of a data start token may be encoded to indicate a length corresponding to memory access protocol data. A second, different data start framing token may also be defined to be interpreted as corresponding to an arrival of interconnect protocol data. Each of the first and second data start framing tokens may be defined according to the interconnect protocol (eg, PCIe), among other examples.

In the example of 9D an end-of-stream token (eg, a specialized PCIe EDS token) may be received 965 encoded to indicate a transition to memory access protocol data. The received end-of-stream token may cause a transition 970 from link-layer logic for processing interconnect protocol data to link-layer logic for processing memory access protocol data. Memory access protocol data may be received 975 and stored using the memory's link layer logic be processed. Memory access protocol link layer control data may be received 980 (eg, at the end of the memory access protocol data) to indicate a transition to interconnect protocol data. Reception 980 of the link layer control data may cause a transition 985 from the link layer logic of the memory access protocol to the link layer logic of the interconnect protocol. Interconnect protocol data may be received after the link layer control data and may be processed by the interconnect protocol link layer logic after transition 985, among other examples.

It should be noted that although much of the principles and examples are described in the context of PCIe and particularly revisions to the PCIe specification, the principles, solutions and features described herein may be equally applicable to other protocols and systems. For example, analog tracking errors in other links using other protocols may be based on analog symbols, data streams, and tokens, as well as rules specified for the use, arrangement, and formatting of such structures within data sent over those other links become, be recorded. Furthermore, alternative mechanisms and structures (e.g., in addition to a PCIe LES register or SKP OS) for providing track error detection and reporting functionality in a system may be used. Furthermore, combinations of the above solutions may be applied in systems including, among other examples, combinations of logical and physical reinforcements on a link and its corresponding logic as described herein.

It should be noted that the apparatus, methods and systems described above may be implemented in any electronic device or system as mentioned above. As a specific illustration, the following figures provide example systems for utilizing the invention as described herein. As the following systems are described in more detail, a number of different interconnects are disclosed, described, and recalled from the previous discussion. And it is immediately apparent that the advancements described above can be applied to any of these interconnects, matrices or architectures.

With reference to 10 Illustrated is one embodiment of a block diagram for a computing system including a multi-core processor. Processor 1000 includes any processor or processing device such as a microprocessor, an embedded processor, a digital signal processor (DSP), a network processor, a handheld processor, an application processor, a co-processor, a system on a chip ( SOC) or other device for executing code. Processor 1000, in one embodiment, includes at least two cores - cores 1001 and 1002, which may include asymmetric cores or symmetric cores (the illustrated embodiment). However, processor 1000 may include any number of processing elements, which may be symmetric or asymmetric.

In one embodiment, a processing element refers to hardware or logic to support a software thread. Examples of hardware processing elements include: a thread unit, a thread slot, a thread, a process unit, a context, a context unit, a logical processor, a hardware thread, a core, and/or any other element capable to hold state for a processor, such as execution state or architectural state. In other words, a processing element, in one embodiment, refers to any hardware capable of being independently interfaced with code, such as a software thread, operating system, application, or other code. A physical processor (or processor socket) typically refers to an integrated circuit that may contain any number of other processing elements, such as cores or hardware threads.

A core often refers to logic residing on an integrated circuit capable of maintaining an independent architectural state, with each independently maintained architectural state being associated with at least some dedicated execution resources. In contrast to cores, a hardware thread typically refers to any logic residing on an integrated circuit capable of maintaining an independent architectural state, where the independently maintained architectural states share access to execution resources. As can be seen, when certain resources are shared and others are dedicated to an architectural state, the line between hardware thread and core nomenclature overlaps. However, a core and a hardware thread are often used by an operating system as separate logi logical processors where the operating system is able to schedule operation on each logical processor individually.

The physical processor 1000, as in 10 1, contains two cores - core 1001 and 1002. Here, core 1001 and 1002 are considered symmetric cores, ie, cores with the same configurations, functional units, and/or logic. In another embodiment, core 1001 includes an out-of-order processor core, while core 1002 includes an in-order processor core. However, the cores 1001 and 1002 can be individually selected from any type of core, such as a native core, a software managed core, a core adapted to execute a native instruction set architecture (ISA), a core adapted to do so configured to execute a translated instruction set architecture (ISA), a co-designed kernel, or another known kernel. In a heterogeneous core environment (ie, asymmetric cores), some form of translation, such as binary translation, may be used to schedule or execute code on one or both cores. For further discussion, the functional units represented in core 1001 are described in detail below, since the units in core 1002 operate in a similar manner in the illustrated embodiment.

As illustrated, core 1001 includes two hardware threads 1001a and 1001b, which may also be referred to as hardware thread slots 1001a and 1001b. Therefore, in one embodiment, software entities, such as an operating system, may view processor 1000 as four separate processors, i.e., four logical processors or processing elements capable of executing four software threads concurrently. As indicated above, a first thread is associated with architecture state registers 1001a, a second thread is associated with architecture state registers 1001b, a third thread may be associated with architecture state registers 1002a, and a fourth thread may be associated with architecture state registers 1002b. Here, each of the architecture state registers (1001a, 1001b, 1002a, and 1002b) may be referred to as processing elements, thread slots, or thread units, as described above. As shown, architecture state registers 1001a are replicated in architecture state registers 1001b so that individual architecture states/contexts can be stored for logical processor 1001a and logical processor 1001b. Smaller resources such as instruction pointers and renaming logic in the allocator and renamer block 1030 level for the threads 1001a and 1001b can also be replicated in the core 1001. Some resources such as retirement buffers in retirement/retirement unit 1035, ILTB 1020, load/store buffers, and queues can be shared through separation. Other resources such as internal general purpose registers, page table base registers, low level data cache and data TLB 1015, execution unit(s) 1040, and portions of an out-of-order unit 1035 may be shared entirely.

Processor 1000 often includes other resources that may be shared entirely, shared through separation, or dedicated by/to processing elements. In 10 1 is an embodiment of a purely exemplary processor with illustrative logical units/resources of a processor. It should be noted that a processor may or may not include any of these functional units, as well as any other known functional units, logic, or firmware, not shown. As shown, core 1001 includes a simplified representative out-of-order (OOO) processor core. However, an in-order processor can also be used in various embodiments. The OOO core includes a branch target buffer 1020 for predicting branches to be taken/taken and an instruction translation buffer (I-TLB) 1020 for storing address translation entries for instructions.

The core 1001 further includes a decode module 1025 coupled to a fetch unit 1020 for decoding fetched items. In one embodiment, fetch logic includes individual sequencers associated with thread slots 1001a and 1001b, respectively. Typically, core 1001 is associated with a first ISA that defines/specifies instructions executable on processor 1000 . Other machine code instructions that are part of the first ISA contain a portion of the instruction (referred to as an opcode) that indicates/specifies an instruction or operation to be performed. The decode logic 1025 includes circuitry that recognizes these instructions from their opcodes and passes the decoded instructions down the pipeline for processing as defined by the first ISA. For example, as discussed in more detail below, in one embodiment, decoders 1025 include logic designed or adapted to recognize specific instructions, such as a transaction instruction. Upon detection by decoders 1025, architecture or core 1001 takes specific, predefined actions to perform tasks associated with the appropriate instruction. It must be noted that all tasks, blocks, operations and methods can be performed in response to a single or multiple command; some of which may be new or old commands. Note that in one embodiment, decoders 1026 recognize the same ISA (or a subset thereof). Alternatively, in a heterogeneous core environment, the decoders 1026 recognize a second ISA (either a subset of the first ISA or a different ISA).

In one example, allocator and renamer block 1030 includes an allocator for reserving resources, such as register files for storing instruction processing results. However, threads 1001a and 1001b may be capable of out-of-order execution where allocator and renamer block 1030 also reserves other resources, such as retirement buffers for tracking instruction results. Unit 1030 may also include a register renamer for renaming program/instruction reference registers to other registers in processor 1000. Retirement/retirement unit 1035 includes components such as the aforementioned retirement buffers, load buffers, and store buffers to support out-of-order execution and later in-order deferral of instructions that are executed out of order.

The scheduler and execution unit(s) block 1040 includes, in one embodiment, a scheduler for scheduling instructions/operations onto execution units. For example, a floating point instruction is scheduled on a port of an execution unit that has an available floating point execution unit. Register files associated with the execution units are also included for storing information of instruction processing results. Exemplary execution units include a floating point execution unit, an integer execution unit, a branch execution unit, a load execution unit, a store execution unit, and other known execution units.

A low level data cache and a data translation buffer (D-TLB) 1050 are coupled to execution unit(s) 1040 . The data cache is used to store recently used/edited items, such as data operands, that may be held in memory coherency states. The D-TLB is used to store recent virtual/linear to physical address translations. As a specific example, a processor may include a page table structure to break up physical memory into multiple virtual pages.

Here the cores 1001 and 1002 share access to a higher level cache or a more external cache such as a second level cache associated with an on-chip interface 1010 . Note that "higher level" or "further out" refers to cache levels that are rising or further away from the execution unit(s). In one embodiment, a higher level cache is a last level data cache - the last cache in the memory hierarchy on processor 1000 - such as a second or third level data cache. However, a higher level cache is not so limited as it may be associated with or contain an instruction cache. A trace cache - a type of instruction cache - may instead be coupled after decoder 1025 to store recently decoded traces. Here, an instruction may refer to a macroinstruction (i.e., a generic instruction recognized by the decoders) that can be decoded into a number of microinstructions (microoperations).

In the illustrated configuration, processor 1000 also includes an on-chip interface module 1010. Historically, a memory controller, described in more detail below, was included external to processor 1000 in a computing system. In this scenario, an on-chip interface 1010 is intended to communicate with devices external to the processor 1000, such as the system memory 1075, a chipset (which often includes a memory controller hub for connecting to a memory 1075, and an I/O controller hub for connecting with peripheral devices), a memory controller hub, a northbridge, or other integrated circuit. And in this scenario, bus 1005 can be any known interconnect, such as a multi-drop bus, a point-to-point interconnect, a serial interconnect, a parallel bus, a coherent (e.g., cache-coherent) bus, a layered protocol architecture , a differential bus and a GTL bus.

Memory 1075 may be dedicated to processor 1000 or shared by other devices in a system. Common examples of types of memory 1075 include DRAM, SRAM, non-volatile memory (NV memory), and other known memory devices. Note that device 1080 may include a graphics accelerator, a processor or card coupled to a memory controller hub, a memory coupled to an I/O controller hub, a wireless transceiver, a Flash device, audio controller, network controller, or other known device.

However, as more and more logic and devices are recently integrated onto a single die, such as SOC, any of these devices may be included on processor 1000. For example, in one embodiment, a memory controller hub is on the same package and/or die as processor 1000. Here, part of core (part on core) 1010 contains one or more controllers for interfacing with other devices such as memory 1075 or a graphics device 1080. The configuration that includes interconnect and controls for interfacing with such devices is often referred to as an "on-core" (or UnCore) configuration. As an example, an on-chip interface 1010 includes a ring interconnect for on-chip communication and a high-speed, serial, point-to-point link 1005 for off-chip communication. However, in the SOC environment, even more devices such as the network interface, co-processors, memory 1075, graphics processor 1080 and any other known computing devices/interfaces can be integrated on a single die or integrated circuit to a small form factor with high functionality and low power consumption.

In one embodiment, processor 1000 is capable of executing compiler, optimizer, and/or translator code 1077 to compile, translate, and/or optimize application code 1076 to implement the apparatus, method, or interface described herein to support with this. A compiler often contains a program or a set of programs for translating source text/code into target text/code. Typically, compiling program/application code with a compiler occurs in multiple phases and proceeds to convert high-level programming language code to low-level machine assembly language code. However, single-pass compilers can still be used for simple compiling. A compiler can use any known compilation technique and perform any known compiler operation, such as lexical analysis, preprocessing, parsing, semantic analysis, code generation, code transformation, and code optimization.

Larger compilers often contain multiple phases, but most often these phases are contained in two general phases: (1) a front-end, i.e. where syntactic processing, semantic processing and some conversion/optimization can generally take place, and (2 ) a back-end, i.e. where analysis, transformations, optimizations and code generation generally take place. Some compilers refer to a middle, which illustrates the blurring of a boundary between a compiler's front end and back end. As a result, reference to deploying, linking, generating, or other operation of a compiler may occur in any of the above phases or passes, as well as any other known phases or passes of a compiler. As an illustrative example, a compiler may insert operations, calls, functions, etc. in one or more phases of compilation, such as inserting calls/operations in a front-end phase of compilation and then recasting the calls/operations into code lower down plane during a reshaping phase. It should be noted that during dynamic compiling the compiler code or dynamic optimization code may employ such operations/calls as well as optimize the code for execution at runtime. As a specific illustrative example, the binary code (code already compiled) can be dynamically optimized at runtime. Here, the program code may include the dynamic optimization code, the binary code, or a combination thereof.

Similar to a compiler, a translator, like a binary translator, translates the code either statically or dynamically to optimize and/or translate the code. Therefore, a reference to execution of code, application code, program code, or other software environment may refer to: (1) execution of one or more compiler programs, optimization code optimizers, or translators, either dynamically or statically, to translate the program code compile, to maintain the software structures, to perform other operations, to optimize the code or to translate the code; (2) execution of main program code containing operations/calls, such as application code that has been optimized/compiled; (3) executing other program code, such as libraries, linked to the main program code to maintain software structures, perform other software-related operations, or optimize the code; or (4) a combination thereof.

Referring now to 11 Illustrated is a block diagram of one embodiment of a multi-core processor. As in the embodiment of 11 As shown, processor 1100 includes multiple domains. Specifically, a core domain 1130 includes multiple cores 1130A-1130N, a graphics domain 1160 includes one or more graphics engines with a media engine 1165, and a system agent domain 1110.

In various embodiments, system agent domain 1110 handles power control events and power management such that individual entities of domains 1130 and 1160 (e.g., cores and/or graphics engines) are independently controllable to dynamically perform in a given activity (or inactivity) occurring in a particular entity to operate in the appropriate power mode/level (e.g., active, turbo, sleep, hibernate, deep sleep, or other Advanced Configuration Power Interface-like state). Each of the domains 1130 and 1160 may operate at a different voltage and/or power, and further, the individual units within the domain may each operate at an independent frequency and voltage. It should be noted that while only three domains are illustrated, an understanding of the scope of the present invention is not limited in this regard and additional domains may be present in other embodiments.

As illustrated, each core 1130 also includes low-level caches in addition to various execution units and additional processing elements. Here, the various cores are coupled to each other and to a shared cache memory made up of multiple units or slices of last level cache (LLC) 1140A-1140N; these LLCs often contain memory and cache control functionality and are shared between the cores as well as possibly the graphics engine.

As can be seen, a ring interconnect 1150 couples the cores together and provides an interconnection between the core domain 1130, the graphics domain 1160 and the system agent circuitry 1110 via multiple ring stops 1152A-1152N, each at a coupling between a core and LLC slice. As in 11 As can be seen, interconnect 1150 is used to convey various information, including address information, data information, acknowledgment information, and snoop/invalid information. Although a ring interconnect is shown, any known on-die interconnect or matrix may be used. As an illustrative example, some of the matrices discussed above (e.g., another on-die interconnect, an on-chip system matrix (OSF), an advanced microcontroller bus architecture (AMBA) interconnect, a multi-dimensional network matrix, or other interconnect architecture) on be used in the same way.

As further illustrated, system agent domain 1110 includes a display engine 1112 intended to provide control of and interface to an associated display. The system agent domain 1110 may include other entities such as: an integrated memory controller 1120 that provides an interface to system memory (e.g., a DRAM implemented with multiple DIMMs) coherency logic 1122 for performing memory coherency operations. Multiple interfaces may be present to allow interconnection between the processor and other circuitry. For example, in one embodiment, at least one direct media interface (DMI) 1116 interface is provided, as are one or more PCIe™ interfaces 1114. The display engine and these interfaces are typically coupled to memory via a PCIe™ bridge 1118. Further, one or more other interfaces may be provided to provide communications between other agents, such as additional processors or other circuitry.

Referring now to 12 a block diagram of a representative core is shown; specifically, logical blocks of a back-end of a core, such as core 1130 of FIG 11 . In general, the in 12 structure illustrated an out-of-order processor having a front-end unit 1270 configured to fetch incoming instructions, perform various processing (eg, caching, decoding, branch prediction, etc.), and forward instructions/operations to an out -of-order machine (OOO) 1280 is used. The OOO engine 1280 performs further processing on decoded instructions.

Specifically in the embodiment of 12 For example, an out-of-order engine 1280 includes a mapper 1282 for receiving decoded instructions, which may be in the form of one or more microinstructions or uops, from the front-end unit 1270 and mapping them to appropriate resources such as registers and so on. The instructions are then provided to a reservation station 1284, which reserves resources and schedules them for execution on one or more execution units 1286A-1286N. Various types of execution units may be present including, for example, arithmetic logic units (ALUs), load and store units, vector processing units (VPUs), floating point execution units, among others. Results from these different execution units are provided to a reorder buffer (ROB) 1288 that takes unordered results and puts them in the correct program order.

With further reference to 12 note that both the front-end unit 1270 and the out-of-order engine 1280 are coupled to different levels of a memory hierarchy. Specifically, an instruction level cache 1272 coupled to a mid-level cache 1276 coupled to a last-level cache 1295 is shown. In one embodiment, the last level cache 1295 is implemented in an on-chip unit 1290 (sometimes referred to as an UnCore). As an example, entity 1290 is associated with system agent 1110 of 11 similar. As discussed above, the UnCore 1290 communicates with the system memory 1299, which in the illustrated embodiment is implemented by ED RAM. It is also noted that the various execution units 1286 in an out-of-order engine 1280 are in communication with a first-level cache 1274 that is also in communication with the mid-level cache 1276 . It should also be noted that additional cores 1230N-2 - 1230N may be coupled to LLC 1295. Although in the embodiment of 12 Illustrated at this high level, it is understood that various changes and additional components may be present.

With regard now to 13 Illustrated is a block diagram of an exemplary computer system formed with a processor including execution units for executing an instruction where one or more of the interconnects implement one or more features, in accordance with an embodiment of the present invention. The system 1300 includes a component, such as a processor 1302, for use of execution units including logic to perform algorithms for processing data in accordance with the present invention, as in the embodiment described herein. The System 1300 is representative of processing systems based on PENTIUM III™, PENTIUM 4™, Xeon™, Itanium, XScale™ and/or StrongARM™ microprocessors, although other systems (including personal computers with other microprocessors, engineering workstations, set-top -Boxes and the like) can also be used. In one embodiment, a sample system 1300 runs a version of the WINDOWS™ operating system available from Microsoft Corporation of Redmond, Washington, although other operating systems (UNIX and Linux, for example), embedded software, and/or graphical user interfaces also use it can become. Thus, embodiments of the present invention are not limited to any particular combination of hardware circuitry and software.

Embodiments are not limited to computer systems. Alternative embodiments of the present invention may be used in other devices, such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, internet protocol devices, digital cameras, personal digital assistants (PDAs), and handheld PCs. Embedded applications may contain a microcontroller, digital signal processor (DSP), system on a chip, network computer (NetPC), set-top boxes, network hubs, wide area network (WAN) switches, or any other system that requires one or more command(s) according to at least one embodiment.

In this illustrated embodiment, the processor 1302 includes one or more execution units 1308 for implementing an algorithm to execute at least one instruction. An embodiment may be described in the context of a single processor desktop or server system, but alternative embodiments may be included in a multiprocessor system. The 1300 system is an example of a 'hub' system architecture. The computer system 1300 includes a processor 1302 for processing data signals. Processor 1302 includes, as an illustrative example, a complex instruction set computer (CISC) microprocessor, a reduced instruction set computer (RISC) microprocessor, a very long instruction word (Very Long Instruction Word, VLIW), a processor implementing a combination of instruction sets, or any other processing device such as a digital signal processor. The processor 1302 is coupled to a processor bus 1310 that sends data signals between the processor 1302 and other components in the system 1300. The elements of system 1300 (e.g., graphics accelerator 1312, memory controller hub 1316, memory 1320, I/O controller hub 1324, wireless transceiver 1326, flash BIOS 1328, network controller 1334, audio controller 1336, serial expansion port 1338, I/O Controller 1340, etc.) perform their conventional functions well known to those skilled in the art.

In one embodiment, the processor 1302 includes an internal level 1 (L1) cache memory 1304 . Depending on the architecture, processor 1302 may have a single internal cache or contain multiple levels of internal caches. Other embodiments include a combination of both internal and external caches, depending on the particular implementation and needs. Register file 1306 is used to store various types of data in various registers including integer registers, floating point registers, vector registers, banked registers, shadow registers, checkpoint registers, status registers, and instruction pointer registers.

Execution unit 1308, which includes logic for performing integer and floating point operations, also resides in processor 1302. Processor 1302, in one embodiment, includes microcode (ucode) ROM for storing microcode that, when executed, executes algorithms for performs certain macro commands, or to handle complex scenarios. Here the microcode may be updatable to handle logic bugs/fixes for the 1302 processor. For one embodiment, the execution unit 1308 includes logic for handling a packed instruction set 1309. By including the packed instruction set 1309 in the instruction set of a general purpose processor 1302, along with the associated circuitry for executing the instructions, the operations used by many multimedia applications can be executed in a general purpose processor 1302 using packed data. Thus, many multimedia applications are accelerated and run more efficiently using the full width of a processor data bus to perform operations on packed data. This potentially eliminates the need to transfer smaller units of data across the processor data bus to perform one or more operations, each on a data item.

Other embodiments of an execution unit 1308 may also be used in microcontrollers, embedded processors, graphics devices, DSPs, and other types of logic circuits. The system 1300 includes a memory 1320. The memory 1320 includes a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, or other storage device. Memory 1320 stores instructions and/or data represented by data signals to be executed by processor 1302.

It should be noted that any of the above features or aspects of the invention may be used on one or more interconnect(s) described in 13 are shown. For example, an on-die interconnect (ODI), not shown, for coupling internal units of processor 1302 implements one or more of the aspects of the invention described above. Or, the invention is implemented with a processor bus 1310 (e.g., other known high-performance computing interconnect), a high-bandwidth memory path 1318 to memory 1320, a point-to-point connection to the graphics accelerator 1312 (e.g., a Peripheral Component Interconnect Express (PCIe) compliant matrix) , a controller-hub interconnect 1322, an I/O, or other interconnect (eg, USB, PCI, PCIe) for coupling the other components shown. Some examples of such components include audio control 1336, firmware hub (Flash BIOS) 1328, wireless transceiver 1326, data storage 1324, legacy I/O control 1310 containing user input and keyboard interfaces 1342, a serial expansion port 1338 such as a Universal Serial Bus (USB) serial bus system and network controller 1334. Data storage device 1324 may include a hard disk drive, floppy disk drive, CD-ROM device, flash memory device, or other mass storage device.

Referring now to 14 1, a block diagram of a second system 1400 is shown in accordance with an embodiment of the present invention. As in 14 As illustrated, a multiprocessor system 1400 is a point-to-point interconnect system and includes a first processor 1470 and a second processor 1480 coupled via a point-to-point interconnect 1450 . Each of processors 1470 and 1480 may be a version of a processor. In one embodiment, 1452 and 1454 are part of a serial, point-to-point, coherent interconnect matrix, such as a high performance architecture. As a result, the invention can be implemented within the QPI architecture.

While only two processors 1470, 1480 are shown, it is understood that the scope of the present invention is not so limited. In other embodiments, one or more additional processor(s) may be present in a given processor.

Processors 1470 and 1480 are shown with integrated memory controllers 1472 and 1482, respectively. Processor 1470 also includes, as part of its bus control units, point-to-point (PP) interfaces 1476 and 1478; likewise, the second processor 1480 includes PP interfaces 1486 and 1488. The processors 1470, 1480 can transmit information over a point-to-point (PP) interface 1450 using PP Replace interface circuits 1478, 1488. As in 14 As illustrated, IMCs 1472 and 1482 couple the processors to respective memories, namely memory 1432 and memory 1434, which may be portions of main memory locally attached to the respective processors.

The processors 1470, 1480 each exchange information with a chipset 1490 via individual P-P interfaces 1452, 1454 using point-to-point interface circuits 1476, 1494, 1486, 1498. The chipset 1490 also exchanges information with a high performance graphics circuit 1438 via an interface circuit 1492 along a high performance graphics interconnect 1439 .

A shared cache (not shown) can be either internal to the processor or external to both processors; but still be connected to the processors via a P-P interconnect so that local cache information of either or both processors can be stored in the shared cache when a processor is placed in a low-power mode.

The chipset 1490 may be coupled to a first bus 1416 via an interface 1496 . In one embodiment, the first bus 1416 may be a Peripheral Component Interconnect (PCI) bus or a bus such as a PCI Express bus or other third generation I/O interconnect bus, although the scope of the present invention is not so limited.

As in 14 As shown, various I/O devices 1414 are coupled to the first bus 1416, along with a bus bridge 1418 that couples the first bus 1416 to a second bus 1420. In one embodiment, the second bus 1420 includes a Low Pin Count (LPC) bus. Various devices are coupled to the second bus 1420 in one embodiment, including, for example, a keyboard and/or mouse 1422, communication devices 1427, and a storage unit 1428 such as a hard drive or other mass storage device, which often contains instructions/code and data 1430. An audio I/O 1424 is also shown coupled to the second bus 1420 . Note that other architectures are possible where the components included and interconnect architectures vary. For example, instead of using the point-to-point architecture of 14 implement a multi-drop bus or other such architecture.

Referring now to 15 1 is shown an embodiment of a system-on-chip (SOC) design according to the invention. As a specific illustrative example, SOC 1500 is included in User Equipment (UE). In one embodiment, UE refers to any device used by an end user to communicate, such as a handheld phone, smartphone, tablet, ultra-thin notebook, notebook with broadband adapter, or any other similar communication device. Often a UE is connected to a base station or node, which may be similar in nature to a mobile station (MS) in a GSM network.

Here, the SOC 1500 contains 2 cores - 1506 and 1507. Similar to the previous discussion, the cores 1506 and 1507 can be an instruction set architecture such as an Intel® Architecture Core™-based processor, an Advanced Micro Devices, Inc. (AMD) processor, a MIPS-based processor, an ARM-based processor design, or a customer thereof, as well as their licensees or adopters. The cores 1506 and 1507 are coupled to a cache controller 1508 which interfaces with the bus interface unit 1509 and an L2 cache 1511 to communicate with other parts of the system 1500. Interconnect 1510 includes an on-chip interconnect, such as an IOSF, AMBA, or other interconnect as discussed above, that may implement one or more aspects described herein.

The interface 1510 provides communication channels to the other components, such as a Subscriber Identity Module (SIM) 1530 to interface with a SIM card, a boot ROM 1535 to boot code for execution by the cores 1506 and 1507 hold to initialize and boot the SOC 1500, an SDRAM controller 1540 for interfacing with the external memory (e.g. DRAM 1560), a flash controller 1545 for interfacing with the non-volatile memory (e.g. flash 1565), a peripheral Controller 1550 (e.g., serial peripheral port) for interfacing with peripheral devices, video codecs 1520, and a video interface 1525 for displaying and receiving input (e.g., touch-triggered input), a GPU 1515 for performing graphics-related computations, etc. Any of these Interfaces may include aspects of the invention described herein.

In addition, the system illustrates peripheral devices for communication, such as a Bluetooth module 1570, 3G modem 1575, GPS 1585, and WiFi 1585. It should be noted that a UE contains a radio for communication as stated above. As a result, not all of these peripheral communication modules are required. However, some form of functionality for external communication should be included in a UE.

While the present invention has been described with respect to a limited number of embodiments, numerous modifications and variations therefrom will be apparent to those skilled in the art. It is intended that the appended claims cover all such modifications and variations as fall within the true spirit and scope of this present invention.

A design can go through different stages, from creation to simulation to manufacture. Data presenting a theme can represent the theme in a number of ways. First, as is useful in simulations, the hardware can be represented using a hardware description language or another functional description language. Additionally, a circuit level model with logic and/or transistor gates may be generated at some stages of the design process. Furthermore, most designs, at some stage, arrive at a data plane that represents the physical arrangement of various devices in the hardware model. In the case where conventional semiconductor manufacturing techniques are used, the data representing the hardware model may be the data representing the presence or absence of various features on various mask layers for masks used to manufacture the integrated circuit. In any representation of the design, the data may be stored in any form of machine-readable medium. A memory, or magnetic or optical storage, such as a disk, may be a machine-readable medium for storing information transmitted via an optical or electrical wave that is modulated or otherwise generated to transmit such information. When an electrical carrier wave indicative or carrying the code is transmitted, a new copy is made to the extent that copying, buffering, or retransmission of the electrical signal is performed. Thus, a communications provider or a network provider may store on a tangible, machine-readable medium, at least temporarily, an item such as information encoded into a carrier wave embodying techniques of embodiments of the present invention.

A module, as used herein, refers to any combination of hardware, software, and/or firmware. As an example, a module includes hardware, such as a microcontroller, associated with a non-transitory medium to store code configured to be executed by the microcontroller. Therefore, in one embodiment, a reference to a module refers to hardware that is specifically configured to recognize and/or execute code held on a non-transitory medium. Furthermore, in another embodiment, use of a module refers to the non-transitory medium containing code specifically configured to be executed by the microcontroller to perform predetermined operations. And, as can be inferred, in another embodiment, the term module (in this example) can refer to the combination of the microcontroller and the non-transitory medium. Often, module boundaries shown as separate generally vary and may overlap. For example, a first and second module may share hardware, software, firmware, or a combination thereof while possibly retaining some independent hardware, software, or firmware. In one embodiment, use of the term "logic" includes hardware such as transistors, registers, or other hardware such as programmable logic devices.

Use of the phrase 'configured for', in one embodiment, refers to the arrangement, assembly, manufacture, offering for sale, import, and/or design of an apparatus, hardware, logic, or element to perform a specified or specific task to execute. In this example, an apparatus or element thereof that is not in operation is still 'configured' to perform a particular task if it is designed, coupled, and/or connected to perform the particular task. As a purely illustrative example, a logic gate may provide a 0 or a 1 during operation. But a logic gate that is 'configured' to provide an enable signal for a clock does not contain every possible logic gate that can provide a 1 or 0. Instead, the logic gate is one that is coupled in some way such that during operation the output of the 1 or 0 serves to enable the clock. Again, note that the use of the term 'configured for' does not imply operation, but rather focuses on the latent state of an apparatus, hardware and/or item, where the apparatus, hardware and/or item in the latent state is designed to perform a particular task when the apparatus, hardware, and/or element is operating.

Further, in one embodiment, use of the phrases 'to', 'capable of', and/or 'operable to' refers to some apparatus, logic, hardware, and/or element so configured that the use of the apparatus, logic, hardware and/or element is possible in a specified manner. It is noted as above that the use of 'capable of' or 'operable to' in one embodiment refers to the latent state of an apparatus, logic, hardware and/or element in which the apparatus, logic, the hardware and/or element is not operational but is designed to enable use of an apparatus in a specified manner.

A value, as used herein, includes any known representation of a number, state, logical state, or binary logical state. Often the use of logic levels, logic values, or logical values is also referred to as 1s and 0s, which simply represent binary logic states. For example, a 1 refers to a high logic level and a 0 refers to a low logic level. In one embodiment, a memory cell, such as a transistor or flash cell, may be capable of holding a single logical value or multiple logical values. However, other representations of values in computer systems have been used. For example, the decimal number ten can also be represented as a binary value 1010 and a hexadecimal letter A. Therefore, a value includes any representation of information capable of being held in a computer system.

Furthermore, states can be represented by values or parts of values. As an example, a first value, such as a logical one, may represent a default or initial state, while a second value, such as a logical zero, may represent a non-default state. Additionally, in one embodiment, the terms reset and set refer to a default and updated value or state, respectively. For example, a default value may contain a high logical value, i.e., reset, while an updated value may contain a low logical value, i.e., set. Note that any combination of values can be used to represent any number of states.

The method, hardware, software, firmware, or code embodiments noted above may be implemented by instructions or code stored on any machine-accessible, machine-readable, computer-accessible, or computer-readable medium, which may be transmitted by a computer Processing element are executable. A non-transitory machine-accessible/readable medium includes any mechanism that provides (i.e., stores and/or transmits) information in a form readable by a machine, such as a computer or electronic system. For example, a non-transitory machine-accessible medium includes random access memory (RAM), such as static RAM (SRAM) or dynamic RAM (DRAM); a ROM; a magnetic or optical storage medium; flash memory devices; electrical storage devices; optical storage devices; acoustic storage devices; another form of storage devices for holding information received from transient (propagated) signals (e.g., carrier waves, infrared signals, digital signals); etc., which are distinguished from the non-volatile media capable of receiving information therefrom.

Instructions used to program logic to perform embodiments of the invention may be stored in memory in the system, such as DRAM, cache, flash memory, or other memory. Furthermore, the instructions may be propagated over a network or through other computer-readable media. Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), such as, but not limited to, floppy disks, optical disks, read-only compact disks - Memories (CD-ROMs) and magneto-optical disks, read only memories (ROMs), random access memories (RAM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), magnetic or optical cards, flash memory, or tangible machine-readable memory used to transmit information over the Internet in an electrical, optical, acoustic, or other form of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Therefore, the computer-readable medium includes any type of tangible, machine-readable medium suitable for storing or transmitting electronic instructions or information in a form that can be read by a machine (e.g., a computer).

The following examples relate to embodiments according to this specification. One or more embodiment(s) may include an apparatus, system, machine-readable memory, machine-readable medium, hardware- and/or software-based logic, and a Provide methods for providing shared memory control to service load and store operations from multiple independent nodes to provide access to a shared memory resource, wherein each of the multiple independent nodes is granted access to a corresponding portion of the shared memory resource is.

In at least one example, the load and store operations are communicated using a shared memory connection protocol.

In at least one example, the shared memory interconnect protocol includes a memory access protocol that uses physical layer logic of another interconnect protocol.

In at least one example, the shared memory connection protocol provides multiplexing between transmission of memory access protocol data and transmission of interconnect protocol data.

In at least one example, the interconnect protocol data comprises at least one of link layer data and transaction layer data.

In at least one example, the memory access protocol includes SMI3 and the interconnect protocol includes Peripheral Component Interconnect (PCI) Express (PCIe).

In at least one example, transitions between interconnect protocol data and memory access protocol data are identified by a sync header encoded to identify the transitions.

In at least one example, transitions between interconnect protocol data and memory access protocol data are identified by a data start framing token encoded to identify the transitions.

In at least one example, transitions from interconnect protocol data to memory access protocol data are identified by an interconnect protocol end-of-stream framing token encoded to identify the transitions, and transitions from memory access protocol data to interconnect protocol data are identified by link layer control flits of the memory access protocol.

In at least one example, the shared memory connection protocol is tunneled over a network protocol stack.

In at least one example, the network protocol stack includes Ethernet.

In at least one example, a first of the plurality of CPU nodes is on a first board and a second of the plurality of CPU nodes is on a second board that is separate from the first board.

In at least one example, at least two of the multiple CPU nodes reside on the same device.

In at least one example, the shared memory controller is further used to track memory transactions that include the load and store operations.

In at least one example, the shared memory controller is further used to determine that a particular one of the plurality of CPU nodes is failing, identify a portion of the particular CPU node's memory transactions, and drop the portion of the particular CPU node's memory transactions while all other memory transactions are occurring be maintained.

In at least one example, the shared memory controller is further to manage permissions for access of the multiple CPU nodes to data on the shared memory resource.

In at least one example, at least a particular one of the plurality of CPU nodes is blocked from accessing at least a first portion of the shared memory and a second of the plurality of CPU nodes is allowed access to the first portion.

In at least one example, the shared memory controller is further operable to manage directory information for data on the shared memory resource.

In at least one example, the directory information determines, for each of multiple data resources stored in the shared storage resource, whether access to the corresponding data resource is exclusive to one of the multiple CPU nodes or shared by two or more of the multiple CPU nodes is used.

In at least one example, the shared memory controller is further operable to negotiate an access change for a particular one of the plurality of data resources, the change comprising at least one of an access change from shared to exclusive or an access change from exclusive to shared.

In at least one example, the shared memory controller is coupled to at least one other shared memory controller that manages at least one other shared memory resource, and the shared memory controller is further operable to communicate load/store operations to the other shared memory controller to allow the multiple CPU nodes to access the other shared memory.

In at least one example, the shared memory controller is further operable to map address information in the load and store operations to corresponding data resources stored in the shared memory resource.

One or more embodiments may provide an apparatus, system, machine-readable memory, machine-readable medium, hardware and/or software based logic and method for sending a memory access request to a shared memory controller, the memory access request including a load/ comprises a memory operation and is for identifying an address of a data resource to be included in a shared memory resource corresponding to the shared memory controller and granting each of the plurality of independent nodes access to a corresponding portion of the shared memory resource.

In at least one example, the memory access request comprises a load request, and the I/O logic is further operable to receive data corresponding to the data resource in response to the load request.

In at least one example, the memory access request comprises a memory request.

In at least one example, a memory access request is sent using a shared memory connection protocol, and the shared memory connection protocol includes a memory access protocol that uses physical layer logic of another interconnect protocol.

In at least one example, the shared memory connection protocol provides for multiplexing between transmission of memory access protocol data and transmission of interconnect protocol data.

In at least one example, transitions between interconnect protocol data and memory access protocol data are identified by at least one of the following: (a) a sync header encoded to identify the transitions; (b) a data start framing token encoded to identify the transitions; and (c) an end-of-stream framing token encoded to identify the transitions.

In at least one example, the memory access protocol comprises an SMI3 and the interconnect protocol comprises a PCIe based protocol.

In at least one example, a particular one of the multiple nodes has multiple CPU sockets and local memory. In at least one example, the shared memory resource resides on a device separate from the particular node.

One or more embodiment(s) may include an apparatus, system, machine-readable memory, machine-readable medium, hardware- and/or software-based logic, and method for receiving a first load/store message from a first independent CPU node identifying particular data in shared memory, providing access to the particular data for the first CPU node in response to the first load/store message, receiving a second load/store message from a second independent CPU node identifying particular data in a shared memory, and providing access to the particular data for the second CPU memory in response to the second load/store message.

In at least one example, each of the first and second first load/store messages are received over a data connection using a shared memory connection protocol.

At least some embodiments may provide for identifying that the first CPU node has access to the particular data and identifying that the second CPU node has access to the particular data.

At least some embodiments may provide tracking of transactions involving the shared memory for both the first and second CPU nodes.

At least some embodiments may provide for identifying directory information of the particular data, the directory information identifying whether the particular data is in a shared, uncached, or exclusive state.

In at least one example, the first load/store message identifies the particular data by a first address and the second first load/store message identifies the particular data by a second, different address.

At least some embodiments may provide for mapping the first address to the particular data and mapping the second address to the particular data.

At least some embodiments may provide a system that includes a first node that includes one or more processor devices, a second node that is independent of the first node and that includes one or more processor devices, and a shared memory that is accessible to each of the first and second nodes through a load/store memory access protocol.

In at least one example, the first node has a fault domain independent of the second node.

In at least one example, the first node is controlled by a first operating system and the second node is controlled by a second operating system.

In at least one example, the load/store memory access protocol is contained within a shared memory connection protocol and the shared memory connection protocol alternates between the memory access protocol and another interconnect protocol.

In at least one example, a shared memory controller may service load and memory operations from the first and second nodes and provide access to the shared memory.

One or more embodiment(s) may provide an apparatus, system, machine-readable memory, machine-readable medium, hardware and/or software-based logic, and method for sending a first sync header on lanes of a data link, the first sync header being encoded to identify a transition from interconnect protocol data to memory access protocol data and sending a second sync header on the lanes of the data link, the second sync header being encoded to identify a transition from memory access protocol data to data of the Interconnect protocol is encoded.

In at least one example, each sync header identifies a type of data block that follows the sync header.

In at least one example, each data block is a predefined length.

In at least one example, the memory access protocol comprises a protocol based on SMI3.

In at least one example, the interconnect protocol includes a protocol based on a PCIe-based protocol.

In at least one example, each sync header is encoded using a 128b/130b encoding.

In at least one example, the second sync header indicates an interconnect protocol data block and a third sync header is sent on the data link lanes to indicate a queued block set of the interconnect protocol.

In at least one example, the first sync header is encoded with alternating values on the tracks and the second sync header is encoded with the same value on all of the tracks.

In at least one example, the storage access protocol data comprises link layer data and the interconnect protocol data comprises one of transaction layer and data link layer packets.

In at least one example, the sync headers are defined according to the interconnect protocol.

In at least one example, the memory access protocol supports load/store memory access messaging.

In at least one example, the memory access log data includes memory access messaging for access to a shared memory resource, each of the plurality of independent nodes being granted access to a corresponding portion of the shared memory resource.

In at least one example, each of the multiple independent nodes has an independent fault domain.

In at least one example, the data link has at least four lanes.

One or more embodiment(s) may include an apparatus, system, machine-readable memory, machine-readable medium, hardware and/or software-based logic, and method for receiving a first sync header on lanes of a data link, wherein the first sync header is encoded with a first encoding, identifying, from the first encoding of the first sync header, a transition from data of an interconnection protocol to data of a memory access protocol, receiving a second sync header on the lanes of the data connection, the second sync header is encoded with a second encoding; and identifying, from the second encoding of the second sync header, a transition from memory access protocol data to interconnect protocol data.

In at least one example, each sync header identifies a type of data block that follows the sync header.

In at least one example, the interconnect protocol comprises a PCIe based protocol.

In at least one example, the memory access protocol is based on SMI3.

In at least one example, the sync header is encoded according to 128b/130b encoding.

In at least one example, the first encoding has values of 01b and 10b alternating on the lanes of the data link.

In at least one example, the memory access protocol data comprises load/store memory access messages.

In at least one example, the memory access messages include messages for access to a shared memory resource and each of the multiple independent nodes in a system is granted access to a corresponding portion of the shared memory resource.

One or more embodiment(s) may include an apparatus, system, machine-readable memory, machine-readable medium, hardware and/or software-based logic, and method for receiving a first sync header on tracks of a data link, wherein the first sync header is encoded with a first encoding, identifying from the first encoding of the first sync header a transition from interconnect protocol data to memory access protocol data, processing the memory access protocol data, receiving a second sync header on the lanes of the data link wherein the second sync header is encoded with a second encoding, and identifying, from the second encoding of the second sync header, a transition from memory access protocol data to interconnect protocol data.

In at least one example, the interconnect protocol comprises a PCIe based protocol and the memory access protocol is based on SMI3.

In at least one example, the sync headers are PCIe.

In at least one example, the memory access log data is processed to service a memory access request contained in the memory access log data.

In at least one example, the memory access request is a request for a shared memory resource shared by multiple independent CPU nodes.

In at least one example, the memory access request comprises a load/store message.

One or more embodiment(s) may include an apparatus, system, machine-readable memory, machine-readable medium, hardware and/or software-based logic, and method for sending a first data start framing token on lanes of a data link wherein the first data start framing token is encoded to identify a transition from interconnect protocol data to memory access protocol data, and transmitting a second data start framing token on the lanes of the data link, wherein the second data start framing token is encoded to identify a Transition of data from memory to encoded to data of the interconnect protocol.

In at least one example, the first data start framing token comprises a modified PCIe STP framing token and the second data start framing token comprises a PCIe STP framing token.

In at least one example, each data start framing token includes a length field.

In at least one example, the transition from interconnect protocol data to memory access protocol data in the first data start framing token is indicated by a value in the length field of the first data start framing token.

In at least one example, the memory access protocol data is to be sent in a window defined by the length field of the first data start framing token.

In at least one example, the memory access protocol is based on SMI3.

In at least one example, the interconnect protocol comprises a PCIe based protocol.

In at least one example, the storage access protocol data comprises link layer data and the interconnect protocol data comprises one of transaction layer and data link layer packets.

In at least one example, the physical layer logic is further to send the memory access protocol data and the memory access protocol data comprises load/store memory access messages.

In at least one example, the memory access log data includes memory access messages for access to a shared memory resource, and each of the plurality of independent nodes is granted access to a corresponding portion of the shared memory resource.

In at least one example, each of the multiple independent nodes has an independent fault domain.

In at least one example, the data connection includes one or more lanes.

One or more embodiment(s) may include an apparatus, system, machine-readable memory, machine-readable medium, hardware and/or software-based logic, and method for receiving a first data start framing token on lanes of a data link , identifying, from the first data start framing token, an arrival of data of a memory access protocol, receiving a second data start framing token on traces of the data connection, the second data start framing token differing from the first data start framing token, and identifying, from the second data start framing token, an arrival of data of an interconnect protocol.

In at least one example, the first data start framing token comprises a modified PCIe STP framing token and the second data start framing token comprises a PCIe STP framing token.

In at least one example, each data start framing token includes a length field.

In at least one example, the transition from interconnect protocol data to memory access protocol data in the first data start framing token is indicated by a value in the length field of the first data start framing token.

In at least one example, the memory access protocol is SMI3 based and the interconnect protocol is a PCIe based protocol.

In at least one example, the memory access protocol data is received and

the data of the interconnect protocol is received.

One or more embodiment(s) may include an apparatus, system, machine-readable memory, machine-readable medium, hardware and/or software-based logic, and method for transmitting a first end-of-stream framing token on lanes of a data link wherein the first end-of-stream framing token is encoded to identify a transition from an interconnect protocol to a memory access protocol, sending memory access protocol data after the transition to the memory access protocol, and sending link layer control data of the memory access protocol to identify a transition from the memory access protocol to the interconnect protocol.

In at least one example, the memory access protocol data is to be sent on the data link until the link layer control data is sent.

In at least one example, the transition to memory access protocol causes a transition from interconnect protocol logic handling data on the data link to memory access protocol logic handling data on the data link.

In at least one example, the memory access protocol comprises a protocol based on SMI3.

In at least one example, the interconnect protocol comprises a PCIe based protocol.

In at least one example, the first end-of-stream framing token comprises a modified PCIe EDS framing token.

In at least one example, a PCIe EDS is sent to indicate an end of a set of PCIe transaction layer packets and an arrival of a PCIe queued block set.

In at least one example, the storage access protocol data comprises link layer data and the interconnect protocol data comprises one of transaction layer and data link layer packets.

In at least one example, the memory access protocol data is broadcast and includes load/store memory access messages.

In at least one example, the memory access log data includes memory access messages for access to a shared memory resource, and each of the plurality of independent nodes is granted access to a corresponding portion of the shared memory resource.

In at least one example, each of the multiple independent nodes has an independent fault domain.

One or more embodiment(s) may include an apparatus, system, machine-readable memory, machine-readable medium, hardware and/or software-based logic, and method for receiving a first end-of-stream framing token on lanes of a data link coded to identify a transition from an interconnect protocol to a memory access protocol, transitioning to using memory access protocol link layer logic based on the first end-of-stream framing token, receiving memory access protocol link layer data, receiving memory access protocol link layer control data to identify a transition from memory access protocol to interconnect protocol, and transitioning to using link layer logic of the interconnect protocol based on the link layer control data.

In at least one example, the memory access protocol is based on SMI3.

In at least one example, the interconnect protocol comprises a PCIe based protocol.

In at least one example, the first end-of-stream framing token comprises a modified PCIe EDS framing token.

In at least one example, the storage access protocol data comprises link layer data and the interconnect protocol data comprises one of transaction layer and data link layer packets.

In at least one example, the memory access protocol data comprises load/store memory access messages.

Claims (22)

Device comprising: a circuit board containing Peripheral Component Interconnect Express electrical, PCIe electrical, wherein the PCIe electrical is to implement a PCIe physical layer; a first protocol stack logic to implement at least part of a PCIe stack, wherein the first protocol stack logic is to generate Peripheral Component Interconnect Express based packets, PCIe based packets and cause the PCIe based packets to be sent over the PCIe physical layer; and a second protocol stack logic to implement at least part of another protocol stack, wherein the other protocol stack comprises a storage protocol, and the second protocol stack logic is to generate flits of a different protocol and cause the flits to be sent over the PCIe physical layer. device after claim 1 , wherein the different protocol comprises a non-PCIe protocol. device after claim 1 or 2 , further comprising one or more cores to generate data to be included in the flits. Device according to one of Claims 1 until 3 , wherein the second protocol stack logic implements two or more of a transaction layer, a protocol layer, and a link layer. Device according to one of Claims 1 until 4 , wherein the different protocol comprises a memory access protocol. Device according to one of Claims 1 until 5 , wherein the different protocol comprises a cache-coherent protocol. Device according to one of Claims 1 until 6 , the different protocol realizing lower latency than PCIe. Device according to one of Claims 1 until 7 , further comprising a controller to multiplex the PCIe-based packets and the different protocol flits. Device according to one of Claims 1 until 8th , further comprising a connection for connecting to another device, the connection being implemented by PCIe electronics. Method comprising: generating PCIe-based packets in a first mode; Sending the PCIe-based packets over a board's PCIe electronics during the first mode, wherein the PCIe electrical implements a PCIe physical layer of an interconnect; generating flits of a different different protocol in a second mode, wherein the other protocol comprises a storage connection protocol; and sending the flits over the PCIe electronics of the board during the second mode. procedure after claim 10 , further comprising multiplexing the sending of the packets and the sending of the flits. procedure after claim 10 or 11 , where data sent in the flits has lower latency than data sent in the packets. Procedure according to one of Claims 10 until 12 , where the packets and flits are sent from a first device on the board to a second device on the board. Procedure according to one of Claims 10 until 13 , wherein the memory connection protocol comprises a cache-coherent protocol. System comprising: an interconnect that includes PCIe electrical equipment, wherein the PCIe electrical is to implement a PCIe physical layer; a first device; a second device coupled to the first device via the interconnect, wherein the second device comprises: a first protocol stack logic to implement at least part of a PCIe stack, wherein the first protocol stack logic is to generate PCIe-based packets and cause the PCIe-based packets to be sent over the PCIe physical layer; and a second protocol stack logic to implement at least part of another protocol stack, wherein the other protocol stack comprises a storage protocol, and the second protocol stack logic is to generate flits of a different protocol and cause the flits to be sent over the PCIe physical layer. system after claim 15 , where the first device and the second device are on the same package. system after claim 15 or 16 , where the first device and the second device are on the same circuit board. system according to one of the Claims 15 until 17 , wherein the different protocol comprises a non-PCIe protocol. system according to one of the Claims 15 until 18 , wherein the first device comprises a processor core. system according to one of the Claims 15 until 19 , wherein the second protocol stack logic implements two or more of a transaction layer, a protocol layer, and a link layer. system according to one of the Claims 15 until 20 , wherein the different protocol comprises a memory access protocol. system according to one of the Claims 15 until 21 , where the different protocol is a Enables data communication with a lower latency than the PCIe-based protocol.

Images