New! View global litigation for patent families

US20010027512A1 - Selective address translation in coherent memory replication - Google Patents

Selective address translation in coherent memory replication Download PDF

Info

Publication number
US20010027512A1
US20010027512A1 US09875233 US87523301A US2001027512A1 US 20010027512 A1 US20010027512 A1 US 20010027512A1 US 09875233 US09875233 US 09875233 US 87523301 A US87523301 A US 87523301A US 2001027512 A1 US2001027512 A1 US 2001027512A1
Authority
US
Grant status
Application
Patent type
Prior art keywords
address
node
local
entry
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US09875233
Other versions
US6446185B2 (en )
Inventor
Erik Hagersten
Original Assignee
Hagersten Erik E.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0706Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
    • G06F11/0712Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a virtual computing platform, e.g. logically partitioned systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0706Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
    • G06F11/0721Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment within a central processing unit [CPU]
    • G06F11/0724Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment within a central processing unit [CPU] in a multiprocessor or a multi-core unit
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0706Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
    • G06F11/073Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a memory management context, e.g. virtual memory or cache management
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F12/00Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02Addressing or allocation; Relocation
    • G06F12/0223User address space allocation, e.g. contiguous or non contiguous base addressing
    • G06F12/0284Multiple user address space allocation, e.g. using different base addresses
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F12/00Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02Addressing or allocation; Relocation
    • G06F12/08Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/10Address translation
    • G06F12/1009Address translation using page tables, e.g. page table structures
    • G06F12/1018Address translation using page tables, e.g. page table structures involving hashing techniques, e.g. inverted page tables
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F12/00Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02Addressing or allocation; Relocation
    • G06F12/08Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/10Address translation
    • G06F12/1072Decentralised address translation, e.g. in distributed shared memory systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0766Error or fault reporting or storing
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F12/00Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02Addressing or allocation; Relocation
    • G06F12/08Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/0802Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
    • G06F12/0864Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches using pseudo-associative means, e.g. set-associative or hashing
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F2212/00Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
    • G06F2212/25Using a specific main memory architecture
    • G06F2212/254Distributed memory
    • G06F2212/2542Non-uniform memory access [NUMA] architecture
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F2212/00Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
    • G06F2212/27Using a specific cache architecture
    • G06F2212/272Cache only memory architecture [COMA]

Abstract

A multiprocessing computer system employs local and global address spaces and multiple access modes. A portion of the global memory of the multiprocessing computer system is allocated to each node, called local memory space. Two logical address spaces are mapped to the local memory of each node. A coherent memory replication (CMR) address space stores shadow pages of data from remote nodes and a local address space stores local data. A bit within a local physical address identifies whether data is a shadow page, which is stored in CMR space, or local data, which is stored in local address space. When a transaction requiring a coherency operation is performed, the CMR bit indicates whether a local physical address to global address translation is required. In one embodiment, if the CMR bit is clear, the local physical address is the same as the global address and the local physical address is used for the coherency operation. Alternatively, if the CMR bit is set, the local physical address is translated to a global address during the coherency operation.

Description

    CROSS REFERENCE TO RELATED PATENT APPLICATIONS
  • [0001]
    This application is a continuation application of U.S. patent application Ser. No. 09/148,738, filed Sep. 4, 1998, which is a continuation-in-part application of U.S. patent application Ser. No. 08/924,385 filed Sep. 5, 1997.
  • BACKGROUND OF THE INVENTION
  • [0002]
    1. Field of the Invention
  • [0003]
    This invention relates to the field of multiprocessor computer systems and, more particularly, to address translations employed within multiprocessor computer systems having distributed shared memory architectures.
  • [0004]
    2. Description of the Relevant Art
  • [0005]
    Multiprocessing computer systems include two or more processors which may be employed to perform computing tasks. A particular computing task may be performed upon one processor while other processors perform unrelated computing tasks. Alternatively, components of a particular computing task may be distributed among multiple processors to decrease the time required to perform the computing task as a whole. Generally speaking, a processor is a device configured to perform an operation upon one or more operands to produce a result. The operation is performed in response to an instruction executed by the processor.
  • [0006]
    A popular architecture in commercial multiprocessing computer systems is the symmetric multiprocessor (SMP) architecture. Typically, an SMP computer system comprises multiple processors connected through a cache hierarchy to a shared bus. Additionally connected to the bus is a memory, which is shared among the processors in the system. Access to any particular memory location within the memory occurs in a similar amount of time as access to any other particular memory location. Since each location in the memory may be accessed in a uniform manner, this structure is often referred to as a uniform memory architecture (UMA).
  • [0007]
    Processors are often configured with internal caches, and one or more caches are typically included in the cache hierarchy between the processors and the shared bus in an SMP computer system. Multiple copies of data residing at a particular main memory address may be stored in these caches. In order to maintain the shared memory model, in which a particular address stores exactly one data value at any given time, shared bus computer systems employ cache coherency. Generally speaking, an operation is coherent if the effects of the operation upon data stored at a particular memory address are reflected in each copy of the data within the cache hierarchy. For example, when data stored at a particular memory address is updated, the update may be supplied to the caches which are storing copies of the previous data. Alternatively, the copies of the previous data may be invalidated in the caches such that a subsequent access to the particular memory address causes the updated copy to be transferred from main memory. For shared bus systems, a snoop bus protocol is typically employed. Each coherent transaction performed upon the shared bus is examined (or “snooped”) against data in the caches. If a copy of the affected data is found, the state of the cache line containing the data may be updated in response to the coherent transaction.
  • [0008]
    Unfortunately, shared bus architectures suffer from several drawbacks which limit their usefulness in multiprocessing computer systems. A bus is capable of a peak bandwidth (e.g. a number of bytes/second which may be transferred across the bus). As additional processors are attached to the bus, the bandwidth required to supply the processors with data and instructions may exceed the peak bus bandwidth. Since some processors are forced to wait for available bus bandwidth, performance of the computer system suffers when the bandwidth requirements of the processors exceeds available bus bandwidth.
  • [0009]
    Additionally, adding more processors to a shared bus increases the capacitive loading on the bus and may even cause the physical length of the bus to be increased. The increased capacitive loading and extended bus length increases the delay in propagating a signal across the bus. Due to the increased propagation delay, transactions may take longer to perform. Therefore, the peak bandwidth of the bus may decrease as more processors are added.
  • [0010]
    These problems are further magnified by the continued increase in operating frequency and performance of processors. The increased performance enabled by the higher frequencies and more advanced processor microarchitectures results in higher bandwidth requirements than previous processor generations, even for the same number of processors. Therefore, buses which previously provided sufficient bandwidth for a multiprocessing computer system may be insufficient for a similar computer system employing the higher performance processors.
  • [0011]
    Another structure for multiprocessing computer systems is a distributed shared memory architecture. A distributed shared memory architecture includes multiple nodes within which processors and memory reside. The multiple nodes communicate via a network coupled there between. When considered as a whole, the memory included within the multiple nodes forms the shared memory for the computer system. Typically, directories are used to identify which nodes have cached copies of data corresponding to a particular address. Coherency activities may be generated via examination of the directories.
  • [0012]
    Distributed shared memory systems are scaleable, overcoming the limitations of the shared bus architecture. Since many of the processor accesses are completed within a node, nodes typically have much lower bandwidth requirements upon the network than a shared bus architecture must provide upon its shared bus. The nodes may operate at high clock frequency and bandwidth, accessing the network when needed. Additional nodes may be added to the network without affecting the local bandwidth of the nodes. Instead, only the network bandwidth is affected.
  • [0013]
    Distributed shared memory systems may employ local and global address spaces. A portion of the global address space is assigned to each node within the distributed shared memory system. Accesses to the address space assigned to a requesting node (i.e. local address space) are typically local transactions. Accesses to portions of the address space not assigned to the requesting node are typically global transactions.
  • [0014]
    In some distributed shared memory systems, data corresponding to addresses of remote nodes may be copied to a requesting node's shared memory such that future accesses to that data may be performed via local transactions rather than global transactions. In such systems, CPU's local to the node may use the local physical address is assigned to the copied data. The copied data is referred to as a shadow page. Address translation tables are provided to translate between the global address and the local physical address assigned to the shadow copy.
  • [0015]
    During coherency operations, such as a request to obtain sufficient access rights to perform a transaction, the local physical address is translated to a global address. If the local physical address does not correspond to a shadow copy, the global address is the same as the local physical address (i.e., no translation is required).
  • [0016]
    Unfortunately, a local node typically cannot distinguish an access to a shadow page, which requires an address translation, and an access to a local address that does not require address translation. Accordingly, the local node typically performs an address translation on all local physical addresses during coherency operations. These address translations add unnecessary latency to local memory accesses and increase the bandwidth requirement of the address translation table.
  • [0017]
    A multiprocessor computer system that eliminates unnecessary address translations is thus desirable.
  • SUMMARY OF THE INVENTION
  • [0018]
    The problems outlined above are in large part solved by a multiprocessor computer system in which the local physical memory of a node includes two address spaces. Both a local address space and a coherent memory replication (CMR) address space are mapped to the local physical memory of a node. When a shadow copy is stored in a node, the data is assigned an address within the CMR space. Local data is assigned addresses within the local address space. When coherency operations occur, the address translation circuitry can determine whether the accessed data is local data or a shadow copy based upon the address. Accordingly, the address translation circuitry can perform a local physical address to global address translation for shadow copies. For addresses within the local address space, an address translation is not performed, which reduces the latency of the local data access and the bandwidth requirement of the address translation circuit.
  • [0019]
    Broadly speaking the present invention contemplates, a multiprocessor computer system comprising a first node, a second node and a global bus. The first node includes a first processor, a first cache coupled to the first processor, a first local bus coupled to the first cache, a first local memory coupled to the first local bus and a first system interface coupled to the first local bus. A first address space and a second address space are mapped to the first local memory and the first address space is configured to store data local to the first node. The first system interface includes a first directory configured to store coherency data for data to the first node. The second node includes a second processor, a second cache coupled to the second processor, a second local bus coupled to the second cache, a second local memory coupled to the second local bus and a second system interface coupled to the second local bus. The second system interface includes a second directory configured to store coherency data for data local to the second node. The global bus is coupled to the first system interface and the second system interface. The first address space is configured to store data local to the first node and the second address space is configured to store copies of data local to the second node. The data stored in the second address space is assigned a physical address local to the first node and the first system interface converts the physical address local to the first node to a global address prior to performing a request on the global bus.
  • [0020]
    The present invention further contemplates a method of performing selective address translation in a multiprocessing computer system comprising: mapping a first address space and a second address space to a local memory of a first node of the multiprocessing computer system; storing data local to the first node in the first address space; storing copies of data local to a second node of the multiprocessing computer system in the second address space, wherein the copies of data stored in the second address space are assigned local addresses of the first node; and converting the local addresses of the data stored in the second address space to global addresses prior to performing a global operation.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • [0021]
    Other objects and advantages of the invention will become apparent upon reading the following detailed description and upon reference to the accompanying drawings in which:
  • [0022]
    [0022]FIG. 1 is a block diagram of a multiprocessor computer system.
  • [0023]
    [0023]FIG. 1A is a conceptualized block diagram depicting a non-uniform memory architecture supported by one embodiment of the computer system shown in FIG. 1.
  • [0024]
    [0024]FIG. 1B is a conceptualized block diagram depicting a cache-only memory architecture supported by one embodiment of the computer system shown in FIG. 1.
  • [0025]
    [0025]FIG. 2 is a block diagram of one embodiment of a symmetric multiprocessing node depicted in FIG. 1.
  • [0026]
    [0026]FIG. 2A is an exemplary directory entry stored in one embodiment of a directory depicted in FIG. 2.
  • [0027]
    [0027]FIG. 3 is a block diagram of one embodiment of a system interface shown in FIG. FIG. 4 is a mapping of a physical address space and a logical address space of a four-node multiprocessing computer system according to one embodiment of the present invention.
  • [0028]
    [0028]FIG. 5 illustrates a local physical address according to one embodiment of the present invention.
  • [0029]
    [0029]FIG. 6 illustrates of a directory entry according to one embodiment of the present invention.
  • [0030]
    [0030]FIG. 7 is a block diagram illustrating a list of free memory and a list of CMR memory.
  • [0031]
    [0031]FIG. 8 is a block diagram illustrating an organization of a local memory and the mapping of pages within the local memory to entries in a local physical address to global address translation table.
  • [0032]
    [0032]FIG. 9 is a diagram illustrating the translation of a local physical address to a global address according to one embodiment of the present invention.
  • [0033]
    [0033]FIG. 10 illustrates an entry of a local physical address to global address translation table according to one embodiment of the present invention.
  • [0034]
    [0034]FIG. 11 is a block diagram illustrating an organization of a global address to local physical address translation table according to one embodiment of the present invention.
  • [0035]
    [0035]FIG. 12A is a block diagram illustrating an alternative organization of a global address to local physical address translation table according to one embodiment of the present invention.
  • [0036]
    [0036]FIG. 12B is a diagram illustrating an example of realigning table entries within a global address to local physical address translation table according to one embodiment of the present invention.
  • [0037]
    [0037]FIG. 12C is a diagram illustrating another example of realigning table entries within a global address to local physical address translation table according to one embodiment of the present invention.
  • [0038]
    [0038]FIG. 13 is a diagram illustrating the translation of a global address to a local physical address according to one embodiment of the present invention.
  • [0039]
    [0039]FIG. 14A is a flowchart illustrating the allocation of entries in a global address to local physical address table according to one embodiment of the present invention.
  • [0040]
    [0040]FIG. 14B is a flowchart illustrating the allocation of coherent replication memory according to one embodiment of the present invention.
  • [0041]
    [0041]FIG. 14C is a flowchart illustrating the realignment of entries in a global address to local physical address table according to one embodiment of the present invention.
  • [0042]
    While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the present invention as defined by the appended claims.
  • DETAILED DESCRIPTION OF THE INVENTION
  • [0043]
    Turning now to FIG. 1, a block diagram of one embodiment of a multiprocessing computer system 10 is shown. Computer system 10 includes multiple SMP nodes 12A-12D interconnected by a point-to-point network 14. Elements referred to herein with a particular reference number followed by a letter will be collectively referred to by the reference number alone. For example, SMP nodes 12A-12D will be collectively referred to as SMP nodes 12. In the embodiment shown, each SMP node 12 includes multiple processors, external caches, an SMP bus, a memory, and a system interface. For example, SMP node 12A is configured with multiple processors including processors 16A-16B. The processors 16 are connected to external caches 18, which are further coupled to an SMP bus 20. Additionally, a memory 22 and a system interface 24 are coupled to SMP bus 20. Still further, one or more input/output (I/O) interfaces 26 may be coupled to SMP bus 20. /O interfaces 26 are used to interface to peripheral devices such as serial and parallel ports, disk drives, modems, printers, etc. Other SMP nodes 12B-12D may be configured similarly.
  • [0044]
    A coherence operation from another node arriving at a node with a shadow address to a local physical address. Generally speaking, the memory, or physical address space, of a computer system is distributed among SMP nodes 12A-12D. The memory assigned to a node is referred to as the local memory of that node. Typically, accesses to a node's local memory are local transactions and accesses to other node's local memory are global transactions. In one embodiment, a node may store a shadow copy of data from another node's local memory (the node which stores the original data is referred to as the home node). Accordingly, accesses to the shadow copy of data may be performed locally rather than accessing the data from the home node. When a shadow copy of a data is stored to a local node, the data is assigned an address within the local physical address space of the local node. Although data accesses to a shadow copy may be local, coherency operations are typically still global. For example, if a local node attempts to write to a shadow copy without sufficient access rights, a global coherency operation, such as a write invalidation operation, is performed to obtain write access rights to the data. When a coherency operation is performed, the local physical address assigned to the shadow copy is translated to the global address of the data using the LPA2GA table.
  • [0045]
    In one embodiment, each node includes two logical address spaces. Both logical address spaces are mapped to the local memory. One address space, called coherent memory replication (CMR) space, stores shadow copies of data from other nodes. The remaining data is stored in a local address space. Portions of the local address space are allocated to CMR space as address space for shadow copies is required.
  • [0046]
    In one embodiment, an upper order address bit (called the CMR bit) differentiates addresses within the local address space from the addresses within the CMR address space. The memory management unit (MMU) of processor 16 handles addresses within the CMR space and the local address space in the same manner (i.e., the MMU ignores the CMR bit). System interface 24, however, uses the CMR bit to determine whether an address translation from a local physical address to a global address is required when a coherency operation is performed. The local physical address of data blocks in the local address space is the same as the global address. Therefore no address translation is required. Alternatively, addresses of data stored in the CMR address space are translations of the global addresses of that data and a local physical address to global address translation is required prior to performing a coherency operation.
  • [0047]
    As used herein, a memory operation is an operation causing the transfer of data from a source to a destination. The source and/or destination may be storage locations within the initiator, or may be storage locations within memory. When a source or destination is a storage location within memory, the source or destination is specified via an address conveyed with the memory operation. Memory operations may be read or write operations. A read operation causes transfer of data from a source outside of the initiator to a destination within the initiator. Conversely, a write operation causes transfer of data from a source within the initiator to a destination outside of the initiator. In the computer system shown in FIG. 1, a memory operation may include one or more transactions upon SMP bus 20 as well as one or more coherency operations upon network 14.
  • [0048]
    Each SMP node 12 is essentially an SMP system having memory 22 as the shared memory. Processors 16 are high performance processors. In one embodiment, each processor 16 is a SPARC processor compliant with version 9 of the SPARC processor architecture. It is noted, however, that any processor architecture may be employed by processors 16.
  • [0049]
    Typically, processors 16 include internal instruction and data caches. Therefore, external caches 18 are labeled as L2 caches (for level 2, wherein the internal caches are level 1 caches). If processors 16 are not configured with internal caches, then external caches 18 are level 1 caches. It is noted that the “level” nomenclature is used to identify proximity of a particular cache to the processing core within processor 16. Level 1 is nearest the processing core, level 2 is next nearest, etc. External caches 18 provide rapid access to memory addresses frequently accessed by the processor 16 coupled thereto. It is noted that external caches 18 may be configured in any of a variety of specific cache arrangements. For example, set-associative or direct-mapped configurations may be employed by external caches 18.
  • [0050]
    SMP bus 20 accommodates communication between processors 16 (through caches 18), memory 22, system interface 24, and I/O interface 26. In one embodiment, SMP bus 20 includes an address bus and related control signals, as well as a data bus and related control signals. A split-transaction bus protocol may be employed upon SMP bus 20. Generally speaking, a split-transaction bus protocol is a protocol in which a transaction occurring upon the address bus may differ from a concurrent transaction occurring upon the data bus. Transactions involving address and data include an address phase in which the address and related control information is conveyed upon the address bus, and a data phase in which the data is conveyed upon the data bus. Additional address phases and/or data phases for other transactions may be initiated prior to the data phase corresponding to a particular address phase. An address phase and the corresponding data phase may be correlated in a number of ways. For example, data transactions may occur in the same order that the address transactions occur. Alternatively, address and data phases of a transaction may be identified via a unique tag.
  • [0051]
    Memory 22 is configured to store data and instruction code for use by processors 16. Memory 22 preferably comprises dynamic random access memory (DRAM, although any type of memory may be used. Memory 22, in conjunction with similar illustrated memories in the other SMP nodes 12, forms a distributed shared memory system. Each address in the address space of the distributed shared memory is assigned to a particular node, referred to as the home node of the address. A processor within a different node than the home node may access the data at an address of the home node, potentially caching the data. Therefore, coherency is maintained between SMP nodes 12 as well as among processors 16 and caches 18 within a particular SMP node 12A-12D. System interface 24 provides internode coherency, while snooping upon SMP bus 20 provides intranode coherency.
  • [0052]
    In addition to maintaining internode coherency, system interface 24 detects addresses upon SMP bus 20 which require a data transfer to or from another SMP node 12. System interface 24 performs the transfer, and provides the corresponding data for the transaction upon SMP bus 20. In the embodiment shown, system interface 24 is coupled to a point-to-point network 14. However, it is noted that in alternative embodiments other networks may be used. In a point-to-point network, individual connections exist between each node upon the network. A particular node communicates directly with a second node via a dedicated link. To communicate with a third node, the particular node utilizes a different link than the one used to communicate with the second node.
  • [0053]
    It is noted that, although four SMP nodes 12 are shown in FIG. 1, embodiments of computer system 10 employing any number of nodes are contemplated.
  • [0054]
    [0054]FIGS. 1A and 1B are conceptualized illustrations of distributed memory architectures supported by one embodiment of computer system 10. Specifically, FIGS. 1A and 1B illustrate alternative ways in which each SMP node 12 of FIG. 1 may cache data and perform memory accesses. Details regarding the manner in which computer system 10 supports such accesses will be described in further detail below.
  • [0055]
    Turning now to FIG. 1A, a logical diagram depicting a first memory architecture 30 supported by one embodiment of computer system 10 is shown. Architecture 30 includes multiple processors 32A-32D, multiple caches 34A-34D, multiple memories 36A-36D, and an interconnect network 38. The multiple memories 36 form a distributed shared memory. Each address within the address space corresponds to a location within one of memories 36.
  • [0056]
    Architecture 30 is a non-uniform memory architecture (NUMA). In a NUMA architecture, the amount of time required to access a first memory address may be substantially different than the amount of time required to access a second memory address. The access time depends upon the origin of the access and the location of the memory 36A-36D which stores the accessed data. For example, if processor 32A accesses a first memory address stored in memory 36A, the access time may be significantly shorter than the access time for an access to a second memory address stored in one of memories 36B-36D. That is, an access by processor 32A to memory 36A may be completed locally (e.g. without transfers upon network 38), while a processor 32A access to memory 36B is performed via network 38. Typically, an access through network 38 is slower than an access completed within a local memory. For example, a local access might be completed in a few hundred nanoseconds while an access via the network might occupy a few microseconds.
  • [0057]
    Data corresponding to addresses stored in remote nodes may be cached in any of the caches 34. However, once a cache 34 discards the data corresponding to such a remote address, a subsequent access to the remote address is completed via a transfer upon network 38.
  • [0058]
    NUMA architectures may provide excellent performance characteristics for software applications which use addresses that correspond primarily to a particular local memory. Software applications which exhibit more random access patterns and which do not confine their memory accesses to addresses within a particular local memory, on the other hand, may experience a large amount of network traffic as a particular processor 32 performs repeated accesses to remote nodes.
  • [0059]
    Turning now to FIG. 1B, a logic diagram depicting a second memory architecture 40 supported by the computer system 10 of FIG. 1 is shown. Architecture 40 includes multiple processors 42A-42D, multiple caches 44A-44D, multiple memories 46A-46D, and network 48. However, memories 46 are logically coupled between caches 44 and network 48. Memories 46 serve as larger caches (e.g. a level 3 cache), storing addresses which are accessed by the corresponding processors 42. Memories 46 are said to “attract” the data being operated upon by a corresponding processor 42. As opposed to the NUMA architecture shown in FIG. 1A, architecture 40 reduces the number of accesses upon the network 48 by storing remote data in the local memory when the local processor accesses that data. The remote data stored in local memory is referred to herein as shadow pages of the remote data.
  • [0060]
    Architecture 40 is referred to as a cache-only memory architecture (COMA). Multiple locations within the distributed shared memory formed by the combination of memories 46 may store data corresponding to a particular address. No permanent mapping of a particular address to a particular storage location is assigned. Instead, the location storing data corresponding to the particular address changes dynamically based upon the processors 42 which access that particular address. Conversely, in the NUMA architecture a particular storage location within memories 46 is assigned to a particular address. Architecture 40 adjusts to the memory access patterns performed by applications executing thereon, and coherency is maintained between the memories 46.
  • [0061]
    In a preferred embodiment, computer system 10 supports both of the memory architectures shown in FIGS. 1A and 1B. In particular, a memory address may be accessed in a NUMA fashion from one SMP node 12A-12D while being accessed in a COMA manner from another SMP node 12A-12D. In one embodiment, a NUMA access is detected if the node ID bits of the address upon SMP bus 20 identify another SMP node 12 as the home node of the address presented. Otherwise, a COMA access is presumed. Additional details will be provided below. In one embodiment, a data accessed in a COMA manner is stored as a shadow page within the node accessing the data.
  • [0062]
    In one embodiment, the COMA architecture is implemented using a combination of hardware and software techniques. Hardware maintains coherency between the locally cached copies of pages, and software (e.g. the operating system employed in computer system 10) is responsible for deallocating and allocating cached pages.
  • [0063]
    [0063]FIG. 2 depicts details of one implementation of an SMP node 12A that generally conforms to the SMP node 12A shown in FIG. 1. Other nodes 12 may be configured similarly. It is noted that alternative specific implementations of each SMP node 12 of FIG. 1 are also possible. The implementation of SMP node 12A shown in FIG. 2 includes multiple subnodes such as subnodes 50A and 50B. Each subnode 50 includes two processors 16 and corresponding caches 18, a memory portion 56, an address controller 52, and a data controller 54. The memory portions 56 within subnodes 50 collectively form the memory 22 of the SMP node 12A of FIG. 1. Other subnodes (not shown) are further coupled to SMP bus 20 to form the I/O interfaces 26.
  • [0064]
    As shown in FIG. 2, SMP bus 20 includes an address bus 58 and a data bus 60. Address controller 52 is coupled to address bus 58, and data controller 54 is coupled to data bus 60. FIG. 2 also illustrates system interface 24, including a system interface logic block 62, a translation storage 64, a directory 66, and a memory tag (MTAG) 68. Logic block 62 is coupled to both address bus 58 and data bus 60, and asserts an ignore signal 70 upon address bus 58 under certain circumstances as will be explained further below. Additionally, logic block 62 is coupled to translation storage 64, directory 66, MTAG 68, and network 14.
  • [0065]
    For the embodiment of FIG. 2, each subnode 50 is configured upon a printed circuit board which may be inserted into a backplane upon which SMP bus 20 is situated. In this manner, the number of processors and/or I/O interfaces 26 included within an SMP node 12 may be varied by inserting or removing subnodes 50. For example, computer system 10 may initially be configured with a small number of subnodes 50. Additional subnodes 50 may be added from time to time as the computing power required by the users of computer system 10 grows.
  • [0066]
    Address controller 52 provides an interface between caches 18 and the address portion of SMP bus 20. In the embodiment shown, address controller 52 includes an out queue 72 and some number of in queues 74. Out queue 72 buffers transactions from the processors connected thereto until address controller 52 is granted access to address bus 58. Address controller 52 performs the transactions stored in out queue 72 in the order those transactions were placed into out queue 72 (i.e. out queue 72 is a FIFO queue). Transactions performed by address controller 52 as well as transactions received from address bus 58 which are to be snooped by caches 18 and caches internal to processors 16 are placed into in queue 74.
  • [0067]
    Similar to out queue 72, in queue 74 is a FIFO queue. All address transactions are stored in the in queue 74 of each subnode 50 (even within the in queue 74 of the subnode 50 which initiates the address transaction). Address transactions are thus presented to caches 18 and processors 16 for snooping in the order they occur upon address bus 58. The order that transactions occur upon address bus 58 is the order for SMP node 12A. However, the complete system is expected to have one global memory order. This ordering expectation creates a problem in both the NUMA and COMA architectures employed by computer system 10, since the global order may need to be established by the order of operations upon network 14. If two nodes perform a transaction to an address, the order that the corresponding coherency operations occur at the home node for the address defines the order of the two transactions as seen within each node. For example, if two write transactions are performed to the same address, then the second write operation to arrive at the address' home node should be the second write transaction to complete (i.e. a byte location which is updated by both write transactions stores a value provided by the second write transaction upon completion of both transactions). However, the node which performs the second transaction may actually have the second transaction occur first upon SMP bus 20. Ignore signal 70 allows the second transaction to be transferred to system interface 24 without any of the CPU's or I/O devices in the SMP node 12 reacting to the transaction.
  • [0068]
    Therefore, in order to operate effectively with the ordering constraints imposed by the out queue/in queue structure of address controller 52, system interface logic block 62 employs ignore signal 70. When a transaction is presented upon address bus 58 and system interface logic block 62 detects that a remote transaction is to be performed in response to the transaction, logic block 62 asserts the ignore signal 70. Assertion of the ignore signal 70 with respect to a transaction causes address controller 52 to inhibit storage of the transaction into in queues 74. Therefore, other transactions which may occur subsequent to the ignored transaction and which complete locally within SMP node 12A may complete out of order with respect to the ignored transaction without violating the ordering rules of in queue 74. In particular, transactions performed by system interface 24 in response to coherency activity upon network 14 may be performed and completed subsequent to the ignored transaction. When a response is received from the remote transaction, the ignored transaction may be reissued by system interface logic block 62 upon address bus 58. The transaction is thereby placed into in queue 74, and may complete in order with transactions occurring at the time of reissue.
  • [0069]
    It is noted that in one embodiment, once a transaction from a particular address controller 52 has been ignored, subsequent coherent transactions from that particular address controller 52 are also ignored. Transactions from a particular processor 16 may have an important ordering relationship with respect to each other, independent of the ordering requirements imposed by presentation upon address bus 58. For example, a transaction may be separated from another transaction by a memory synchronizing instruction such as the MEMBAR instruction included in the SPARC architecture. The processor 16 conveys the transactions in the order the transactions are to be performed with respect to each other.
  • [0070]
    Data controller 54 routes data to and from data bus 60, memory portion 56 and caches 18. Data controller 54 may include in and out queues similar to address controller 52. In one embodiment, data controller 54 employs multiple physical units in a byte-sliced bus configuration.
  • [0071]
    Processors 16 as shown in FIG. 2 include memory management units (MMUs) 76A-76B. MMUs 76 perform a virtual to physical address translation upon the data addresses generated by the instruction code executed upon processors 16, as well as the instruction addresses. The addresses generated in response to instruction execution are virtual addresses. In other words, the virtual addresses are the addresses created by the programmer of the instruction code. The virtual addresses are passed through an address translation mechanism (embodied in MMUs 76), from which corresponding physical addresses are created. The physical address identifies a storage location within memory 22.
  • [0072]
    Virtual to physical address translation is performed for many reasons. For example, the address translation mechanism may be used to grant or deny a particular computing task's access to certain memory addresses. In this manner, the data and instructions within one computing task are isolated from the data and instructions of another computing task. Additionally, portions of the data and instructions of a computing task may be “paged out” to a hard disk drive. When a portion is paged out, the translation is invalidated. Upon access to the portion by the computing task, an interrupt occurs due to the failed translation. The interrupt allows the operating system to retrieve the corresponding information from the hard disk drive. In this manner, more virtual memory may be available than actual memory in memory 22. Many other uses for virtual memory are well known.
  • [0073]
    Referring back to computer system 10 shown in FIG. 1 in conjunction with the SMP node 12A implementation illustrated in FIG. 2, the physical address computed by MMUs 76 may be a local physical address (LPA) defining a location within the memory 22 associated with the SMP node 12 in which the processor 16 is located. MTAG 68 stores a coherency state for each “coherency unit” in memory 22. When an address transaction is performed upon SMP bus 20, system interface logic block 62 examines the coherency state stored in MTAG 68 for the accessed coherency unit. If the coherency state indicates that the SMP node 12 has sufficient access rights to the coherency unit to perform the access, then the address transaction proceeds. If, however, the coherency state indicates that coherency operations should be performed prior to completion of the transaction, then system interface logic block 62 asserts the ignore signal 70. Logic block 62 performs coherency operations upon network 14 to acquire the appropriate coherency state. When the appropriate coherency state is acquired, logic block 62 reissues the ignored transaction upon SMP bus 20. Subsequently, the transaction completes.
  • [0074]
    Generally speaking, the coherency state maintained for a coherency unit at a particular storage location (e.g. a cache or a memory 22) indicates the access rights to the coherency unit at that SMP node 12. The access right indicates the validity of the coherency unit, as well as the read/write permission granted for the copy of the coherency unit within that SMP node 12. In one embodiment, the coherency states employed by computer system 10 are modified, owned, shared, and invalid. The modified state indicates that the SMP node 12 has updated the corresponding coherency unit. Therefore, other SMP nodes 12 do not have a copy of the coherency unit. Additionally, when the modified coherency unit is discarded by the SMP node 12, the coherency unit is stored back to the home node. The owned state indicates that the SMP node 12 is responsible for the coherency unit, but other SMP nodes 12 may have shared copies. Again, when the coherency unit is discarded by the SMP node 12, the coherency unit is stored back to the home node. The shared state indicates that the SMP node 12 may read the coherency unit but may not update the coherency unit without acquiring the owned state. Additionally, other SMP nodes 12 may have copies of the coherency unit as well. Finally, the invalid state indicates that the SMP node 12 does not have a copy of the coherency unit. In one embodiment, the modified state indicates write permission and any state but invalid indicates read permission to the corresponding coherency unit.
  • [0075]
    As used herein, a coherency unit is a number of contiguous bytes of memory which are treated as a unit for coherency purposes. For example, if one byte within the coherency unit is updated, the entire coherency unit is considered to be updated. In one specific embodiment, the coherency unit is a cache line, comprising 64 contiguous bytes. It is understood, however, that a coherency unit may comprise any number of bytes.
  • [0076]
    System interface 24 also includes a translation mechanism which utilizes translation storage 64 to store translations from a local physical address (LPA) to a global address (GA), and from a GA back to a LPA. Certain bits within a physical address identify the home node for the address, at which coherency information is stored for that global address. For example, an embodiment of computer system 10 may employ four SMP nodes 12 such as that of FIG. 1. In such an embodiment, two bits of the physical address identify the home node. Preferably, bits from the most significant portion of the physical address are used to identify the home node. The same bits are used in the physical address to identify NUMA accesses. If the bits of the physical address indicate that the local node is not the home node, then the physical address is a global address and the transaction is performed in NUMA mode. Therefore, the operating system places global addresses in MMUs 76 for any NUMA-type pages. Conversely, the operating system places LPAs in MMU 76 for any COMA-type pages. It is noted that an LPA may be the same as a GA (for NUMA accesses to remote address and accesses to addresses allocated to local memory). Alternatively, an LPA may be translated to a GA when the LPA identifies storage locations that store copies of data having a home in another SMP node 12, i.e. shadow pages.
  • [0077]
    The directory 66 of a particular home node identifies which SMP nodes 12 have copies of data corresponding to a given physical address assigned to the home node such that coherency between the copies may be maintained. Additionally, the directory 66 of the home node identifies the SMP node 12 which owns the coherency unit. Therefore, while local coherency between caches 18 and processors 16 is maintained via snooping, system-wide (or global) coherency is maintained using MTAG 68 and directory 66. Directory 66 stores the coherency information corresponding to the coherency units which are assigned to SMP node 12A (i.e. for which SMP node 12A is the home node).
  • [0078]
    It is noted that for the embodiment of FIG. 2, directory 66 and MTAG 68 store information for each coherency unit (i.e., on a coherency unit basis). Conversely, translation storage 64 stores local physical to global address translations defined for pages. A page includes multiple coherency units, and is typically several kilobytes or even megabytes in size.
  • [0079]
    Computer system 10 accordingly creates local physical address to global address translations on a page basis (thereby allocating a local memory page for storing a copy of a remotely stored global page). Therefore, blocks of memory 22 are allocated to a particular global address on a page basis as well. However, as stated above, coherency states and coherency activities are performed upon a coherency unit. Therefore, when a page is allocated in memory to a particular global address, the data corresponding to the page is not necessarily transferred to the allocated memory. Instead, as processors 16 access various coherency units within the page, those coherency units are transferred from the owner of the coherency unit. In this manner, the data actually accessed by SMP node 12A is transferred into the corresponding memory 22. Data not accessed by SMP node 12A may not be transferred, thereby reducing overall bandwidth usage upon network 14 in comparison to embodiments which transfer the page of data upon allocation of the page in memory 22.
  • [0080]
    It is noted that in one embodiment translation storage 64, directory 66, and/or MTAG 68 may be caches which store only a portion of the associated translation, directory, and MTAG information, respectively. The entirety of the translation, directory, and MTAG information may be stored in tables within memory 22 or a dedicated memory storage (not shown). If required information for an access is not found in the corresponding cache, the tables are accessed by system interface 24.
  • [0081]
    Turning now to FIG. 2A, an exemplary directory entry 71 is shown. Directory entry 71 may be employed by one embodiment of directory 66 shown in FIG. 2. Other embodiments of directory 66 may employ dissimilar directory entries. Directory entry 71 includes a valid bit 73, a write back bit 75, an owner field 77, and a sharers field 79. Directory entry 71 resides within the table of directory entries, and is located within the table via the global address identifying the corresponding coherency unit. More particularly, the directory entry 71 associated with a coherency unit is stored within the table of directory entries at an offset formed from the global address which identifies the coherency unit.
  • [0082]
    Valid bit 73 indicates, when set, that directory entry 71 is valid (i.e. that directory entry 71 is storing coherency information for a corresponding coherency unit). When clear, valid bit 73 indicates that directory entry 71 is invalid.
  • [0083]
    Owner field 77 identifies one of SMP nodes 12 as the owner of the coherency unit. The owning SMP node 12A-12D maintains the coherency unit in either the modified or owned states. Typically, the owning SMP node 12A-12D acquires the coherency unit in the modified state. Subsequently, the owning SMP node 12A-12D may then transition to the owned state upon providing a copy of the coherency unit to another SMP node 12A-12D. The other SMP node 12A-12D acquires the coherency unit in the shared state. In one embodiment, owner field 77 comprises two bits encoded to identify one of four SMP nodes 12A-12D as the owner of the coherency unit.
  • [0084]
    Sharers field 79 includes one bit assigned to each SMP node 12A-12D. If an SMP node 12A-12D is maintaining a shared copy of the coherency unit, the corresponding bit within sharers field 79 is set. Conversely, if the SMP node 12A-12D is not maintaining a shared copy of the coherency unit, the corresponding bit within sharers field 79 is clear. In this manner, sharers field 79 indicates all of the shared copies of the coherency unit which exist within the computer system 10 of FIG. 1.
  • [0085]
    Write back bit 75 indicates, when set, that the SMP node 12A-12D identified as the owner of the coherency unit via owner field 77 has written the updated copy of the coherency unit to the home SMP node 12. When clear, bit 75 indicates that the owning SMP node 12A-12D has not written the updated copy of the coherency unit to the home SMP node 12A-12D.
  • [0086]
    Turning now to FIG. 3, a block diagram of one embodiment of system interface 24 is shown. As shown in FIG. 3, system interface 24 includes directory 66, translation storage 64, and MTAG 68. Translation storage 64 is shown as a global address to local physical address (GA2LPA) translation unit 80 and a local physical address to global address (LPA2GA) translation unit 82.
  • [0087]
    System interface 24 also includes input and output queues for storing transactions to be performed upon SMP bus 20 or network 14. Specifically, for the embodiment shown, system interface 24 includes input header queue 84 and output header queue 86 for buffering header packets to and from network 14. Header packets identify an operation to be performed, and specify the number and format of any data packets which may follow. Output header queue 86 buffers header packets to be transmitted upon network 14, and input header queue 84 buffers header packets received from network 14 until system interface 24 processes the received header packets. Similarly, data packets are buffered in input data queue 88 and output data queue 90 until the data may be transferred upon SMP data bus 60 and network 14, respectively.
  • [0088]
    SMP out queue 92, SMP in queue 94, and SMP I/O in queue (PIQ) 96 are used to buffer address transactions to and from address bus 58. SMP out queue 92 buffers transactions to be presented by system interface 24 upon address bus 58. Reissue transactions queued in response to the completion of coherency activity with respect to an ignored transaction are buffered in SMP out queue 92. Additionally, transactions generated in response to coherency activity received from network 14 are buffered in SMP out queue 92. SMP in queue 94 stores coherency related transactions to be serviced by system interface 24. Conversely, SMP PIQ 96 stores I/O transactions to be conveyed to an I/O interface residing in another SMP node 12. I/O transactions generally are considered non-coherent and therefore do not generate coherency activities.
  • [0089]
    SMP in queue 94 and SMP PIQ 96 receive transactions to be queued from a transaction filter 98. Transaction filter 98 is coupled to MTAG 68 and SMP address bus 58. If transaction filter 98 detects an I/O transaction upon address bus 58 which identifies an I/O interface upon another SMP node 12, transaction filter 98 places the transaction into SMP PIQ 96. If a coherent transaction to an LPA address is detected by transaction filter 98, then the corresponding coherency state from MTAG 68 is examined. In accordance with the coherency state, transaction filter 98 may assert ignore signal 70 and may queue a coherency transaction in SMP in queue 94. Ignore signal 70 is asserted and a coherency transaction queued if MTAG 68 indicates that insufficient access rights to the coherency unit for performing the coherent transaction is maintained by SMP node 12A. Conversely, ignore signal 70 is deasserted and a coherency transaction is not generated if MTAG 68 indicates that a sufficient access right is maintained by SMP node 12A.
  • [0090]
    Transactions from SMP in queue 94 are processed by a request agent 100 within system interface 24. Prior to action by request agent 100, LPA2GA translation unit 82 translates the address of the transaction (if it is an LPA address) from the local physical address presented upon SMP address bus 58 into the corresponding global address. Request agent 100 then generates a header packet specifying a particular coherency request to be transmitted to the home node identified by the global address. The coherency request is placed into output header queue 86. Subsequently, a coherency reply is received into input header queue 84. Request agent 100 processes the coherency replies from input header queue 84, potentially generating reissue transactions for SMP out queue 92 (as described below).
  • [0091]
    Also included in system interface 24 is a home agent 102 and a slave agent 104. Home agent 102 processes coherency requests received from input header queue 84. From the coherency information stored in directory 66 with respect to a particular global address, home agent 102 determines if a coherency demand is to be transmitted to one or more slave agents in other SMP nodes 12. In one embodiment, home agent 102 blocks the coherency information corresponding to the affected coherency unit. In other words, subsequent requests involving the coherency unit are not performed until the coherency activity corresponding to the coherency request is completed. According to one embodiment, home agent 102 receives a coherency completion from the request agent which initiated the coherency request (via input header queue 84). The coherency completion indicates that the coherency activity has completed. Upon receipt of the coherency completion, home agent 102 removes the block upon the coherency information corresponding to the affected coherency unit. It is noted that, since the coherency information is blocked until completion of the coherency activity, home agent 102 may update the coherency information in accordance with the coherency activity performed immediately when the coherency request is received.
  • [0092]
    Slave agent 104 receives coherency demands from home agents of other SMP nodes 12 via input header queue 84. In response to a particular coherency demand, slave agent 104 may queue a coherency transaction in SMP out queue 92. In one embodiment, the coherency transaction may cause caches 18 and caches internal to processors 16 to invalidate the affected coherency unit. Alternatively, the coherency transaction may cause caches 18 and caches internal to processors 16 to change the coherency state of the coherency unit to shared. Prior to performing activities in response to a coherency demand, the global address received with the coherency demand is translated to a local physical address via GA2LPA translation unit 80. Once slave agent 104 has completed activity in response to a coherency demand, slave agent 104 transmits a coherency reply to the request agent which initiated the coherency request corresponding to the coherency demand.
  • [0093]
    According to one embodiment, the coherency protocol enforced by request agents 100, home agents 102, and slave agents 104 includes a write invalidate policy. In other words, when a processor 16 within an SMP node 12 updates a coherency unit, any copies of the coherency unit stored within other SMP nodes 12 are invalidated. However, other write policies may be used in other embodiments. For example, a write update policy may be employed. According to a write update policy, when an coherency unit is updated the updated data is transmitted to each of the copies of the coherency unit stored in each of the SMP nodes 12.
  • [0094]
    Referring back to FIG. 2, the verification and acquisition of coherency rights according to one embodiment of the present invention are discussed below. When processor 16 attempts to read or write to a memory location, the MMU within processor 16 converts the virtual address generated by the program to a physical address. The physical address includes a node ID field which indicates the home node to which the physical address is assigned. If the home node corresponds to the node which initiates the transaction (i.e. the requesting node), the address is referred to as a local physical address. Alternatively, if the node ID field identifies a node other than the requesting node, the address is referred to as a global address. Using the physical address, processor 16 determines whether the data that corresponds to the physical address is stored in cache 18. Cache 18 may store data corresponding to local physical addresses or data corresponding to global addresses (data accessed in a NUMA manner may be stored in cache with a global address). The data corresponding to local physical addresses may be one of two types. The local physical address may correspond to memory locations assigned to the local node or it may correspond to shadow copies of data from a remote node (i.e. data accessed in a COMA manner).
  • [0095]
    If the data is found in cache 18, processor 16 accesses the data from the cache. Alternatively, if the data is not located in cache 18, then a request for the data is output on SMP bus 20. If the physical address is a global address, system interface 24 will initiate a global transaction to acquire the desired data. Alternatively, if the physical address is a local physical address, system interface logic 62 will determine whether the node has sufficient access rights to perform the transaction by reading the entry of MTAG 68 that corresponds to address. If the node has sufficient access rights for the desired transaction, the transaction is performed on the data in memory 22. If the node does not have sufficient access rights, the node must acquire sufficient access rights before performing the transaction. The node obtains the access rights by initiating a coherency operation to obtain those rights.
  • [0096]
    In one embodiment, each node includes two logical address spaces. Both logical address spaces are mapped to the entire memory 22 and are synonyms for accessing the same memory location. A first address space, called CMR space, stores shadow copies of data from other nodes. The remaining data is stored in a second address space, called local address space.
  • [0097]
    In one embodiment, a local physical address includes an address bit, called a CMR bit, that indicates whether the local physical address corresponds to an address assigned to the requesting node (i.e., the requesting node is the home node for the data) or to a shadow page within the CMR address space (i.e., a shadow copy of data from a remote node). If the CMR bit is set, which indicates the data is a shadow page, system interface 24 translates the local physical address to a global address prior to performing a coherency operation. Alternatively, if the CMR bit is clear, which indicates the requesting node is the home node for the data, the local physical address is the same as the global address and no translation is necessary prior to performing a coherency operation. Addresses with the CMR bit set are mapped to CMR space. Addresses with the CMR bit cleared are mapped to local address space.
  • [0098]
    Without the CMR bit, system interface logic 24 cannot differentiate between a local physical address that corresponds to local data and a local physical address that corresponds to a shadow copy of remote data. Accordingly, system interface 24 will translate all local physical addresses prior to performing a coherency operation. Because the translation is unnecessary for local physical addresses that correspond to local data, the translation adds unnecessary latency to the transaction and increases the bandwidth that translation storage 64 must handle. A protocol for acquiring sufficient access rights is discussed in more detail in copending, commonly assigned patent application (A Multiprocessing Computer System Employing Local and Global Address Spaces And Multiple Access Modes), filed Jul. 1, 1996, Ser. No. 08/675,635, which is herein incorporated by reference in its entirety.
  • [0099]
    Turning now to FIG. 4, a diagram illustrating a physical address space and a logical address space of a four node multiprocessing computer system according to one embodiment of the present invention is shown. Physical address space 502 is divided among the four nodes of the multiprocessing computer system. Local physical address space 506 is a portion of physical address space 502 allocated to node 0. Local physical address space 508 is a portion of physical address space 502 allocated to node 1. Local physical address space 510 is a portion of physical address space 502 allocated to node 2. Local physical address space 512 is a portion of physical address space 502 allocated to node 3. Each node is the home node for the portion of physical address space 502 allocated to the node. In one embodiment, the local physical address space allocated to a node is physically located within the node. Accordingly, accesses by a home node to the local physical address space allocated to that node are local accesses. For example, if node 0 accesses data stored at an address within local physical address space 506, the data access is a local transaction. Other nodes, however, may have copies of the data, which may necessitate a global coherency operation prior to accessing the data.
  • [0100]
    In the illustrated embodiment, physical address space 502 is divided equally among the four nodes. In other embodiments, physical address space 502 may be divided unequally between the nodes. It is noted that the four node multiprocessor computer system of FIG. 4 is only an illustration. Multiprocessing computer systems with any number of nodes are contemplated.
  • [0101]
    The logical address space 504 of the computer system is also illustrated in FIG. 4. In the illustrated embodiment, two logical address spaces are mapped to each local physical address space within a node. For example, local address space 514 and CMR address space 516 are both mapped to local physical address space 506. In other words an access to an offset within local address space 514 accesses the same physical memory location as an access to CMR space 516 with the same offset.
  • [0102]
    Local address space 518 and CMR address space 520 are mapped to local physical address space 508 of node 1. In a similar manner, local address space 522 and CMR address space 524 are mapped to local physical address space 510 of node 2. Lastly, local address space 526 and CMR address space 528 are mapped to local physical address space 512 in node 3.
  • [0103]
    Turning now to FIG. 5, a format of an address 601 of a multiprocessing system according to one embodiment of the present invention is shown. In the illustrated embodiment, address 601 includes four fields: an offset field 602, a CMR bit 604, a node ID field 606 and a coherency field 608 computer. Offset field 602 identifies a page within an address space and an offset within the page. In the illustrated embodiment, offset field 602 is 37 bits. In one embodiment, the upper four bits of offset field 602 are reserved. As discussed above, CMR bit 604 identifies a logical address space within a node. In one embodiment, the CMR bit identifies either a local address space or a CMR space. In one particular embodiment, the logical address space and the CMR space are mapped to the same physical address space. Accordingly, a memory controller of a processor ignores the CMR bit. System interface 24, however, uses the CMR bit to determine whether an address translation is necessary prior to a global transaction, such as a coherency operation. Node ID field 606 identifies the home node of the address. In the illustrated embodiment, node ID field 606 is four bits. Accordingly, a system employing the illustrated address format can accommodate 16 nodes. If node ID field 606 identifies the requesting node, the address is a local physical address and accesses to the memory location are local. Alternatively, if node ID field 606 identifies a remote node, the address is a global address and accesses to the memory location are global. Coherency field 608 indicates whether the address is in a coherent memory address space or a non-coherent address space. The non-coherent memory address space stores data that is not cached, such as I/O data. In one embodiment, the non-coherent address space occupies half the address space of the multiprocessing computer system. In one particular embodiment, the non-coherent address space occupies the most significant half of the system address space.
  • [0104]
    Turning now to FIG. 6, an alternative format for a directory entry 702 according to one embodiment of the present invention is shown. Valid field 73, write back field 75, owner field 77 and sharer field 79 are similar to those discussed above in reference to FIG. 2A. Directory entry 702 may be employed in one embodiment of directory 66. Other embodiments of directory 66 may employ dissimilar directory entries. Directory entry 702 includes a COMA access (CA) bit 704. When set, the COMA access bit indicates that a COMA access has been made to the coherency unit that corresponds to the directory entry. Alternatively, when the COMA access bit is clear, it indicates that only NUMA accesses have been made to the coherency unit that corresponds to the directory entry.
  • [0105]
    If only NUMA accesses have been made to a particular coherency unit, a translation from a global address to a local physical address is not required when a reply is made to a coherency operation. Alternatively, if a COMA access to a coherency unit has been made, shadow copies of data may be stored, in one or more nodes, at a local physical address which is a translation of the global address. Accordingly, when a demand is made to a coherency operation that has been accessed in COMA mode, the slave node typically must translate the global address to a local physical address. In one embodiment, a bit within a demand to a coherency operation indicates whether the COMA access bit within the directory entry is asserted. Based upon the state of this bit, the node that receives the reply may determine whether a translation of the global address is required. In an alternative embodiment, a control signal can be asserted which indicates whether the COMA access bit of a directory entry is asserted.
  • [0106]
    For example, a node may request read-to-own (RTO) access rights to a coherency unit. In response to the RTO request, the home node may invalidate any copies of the data within the coherency unit in other nodes. In one embodiment, a bit within the invalidate demand indicates whether any nodes are storing the data within the coherency unit in COMA mode. If data is stored in COMA mode, the global address of the invalidate demand is translated to a local physical address (if data is stored in NUMA mode on that node, the translation may be a unity translation) and the data corresponding to the translated local physical address is invalidated. If data is only stored in NUMA mode in the system, a special invalidate command that indicates that no translation is required may be sent to the nodes. In this manner, the latency associated with the translation from the global address to local physical address may be eliminated.
  • [0107]
    In an alternative embodiment, directory 66 stores information indicative of which nodes are storing data in COMA mode and which nodes are storing data in NUMA mode. In this manner, translation invalidate commands may be sent to the nodes storing data in COMA mode, and invalidate commands specify no translation may be sent to nodes storing data in NUMA mode.
  • [0108]
    It is noted, that the COMA access bit of a directory entry may be asserted when no COMA data is stored in any of the nodes of the multiprocessing system. For example, a COMA access may be made to data within a particular coherency unit. The COMA access causes the COMA access bit of the directory entry corresponding to the coherency unit to be asserted. Subsequently, the COMA access data is discarded or invalidated by the node storing the COMA access data. In this embodiment, the COMA access bit may still be asserted and translations from a global address to a local physical address may be unnecessarily performed during a coherency operation. In an alternative embodiment, the COMA bit may be promptly reset when all COMA data within the multiprocessing computer system has been invalidated. Another example of unnecessary GA2LPA lookups is for a system where one node stores a coherence unit in COMA mode and the other nodes store it in NUMA mode. This scheme will cause a GA2LPA lookup in all nodes, even though only the node in COMA mode required the lookup.
  • [0109]
    Turning now to FIG. 7, a diagram illustrating a free memory list 802 and a CMR list 804 is shown. As discussed above, in one embodiment, two logical address spaces (local address space and CMR space) are mapped to the local physical address space of a node. In one particular embodiment, a list of free memory space 802 is maintained for each node. Free memory list 802 contains addresses of pages within the local memory that have not been allocated for data storage. When a processor needs data space in local memory, the processor stores the data to a page listed in the free memory list 802 and removes the address of the page, or pages, to which the data is stored from free memory list 802.
  • [0110]
    In one embodiment, a portion of the free memory of a node is allocated as free CMR space. CMR list 804 stores the addresses of pages of unallocated memory designated as CMR space. When the system needs to store data to CMR space, the system stores the data to a page within CMR list 804 and removes the address of the page from CMR list 804. The system allocates CMR space by moving addresses of pages from free list 802 to CMR list 804. As illustrated by reference numeral 806, an address of a page in free memory list 802 may be moved to CMR list 804 to allocate a page of local memory as CMR address space.
  • [0111]
    Turning now to FIG. 8, an organization of a local physical address space and a local physical address to global address (LPA2GA) translation table is shown. In some embodiments, the LPA2GA translation table of a node includes an entry for each page within the local physical address space of that node. As the size of the local physical address space increases, the size of the LPA2GA translation table also increases. As the size of the LPA2GA table increases, the access time of the table also increases. As the access time increases, it becomes impracticable to access the entire LPA2GA table. One alternative is to implement the LPA2GA translation table as a cache backed by memory. The most recently accessed translations may be stored in cache and the entire LPA2GA table stored in memory. This decreases the access time of the translation table if a translation is in the cache. However, cache misses are fairly costly in terms of latency. Additionally, the complexity of the LPA2GA translation table is substantially increased.
  • [0112]
    In an alternative embodiment illustrated in FIG. 8, several pages of local physical address space are mapped to one entry of LPA2GA translation table 104. In the illustrated embodiment, four pages of local physical address space are mapped to one entry of LPA2GA translation table 104. For example, in the illustrated embodiment, page 802, page 808, page 814 and page 820 of local physical address space 506 are mapped to entry 826 of LPA2GA translation table 104. Prior to allocating a page of local memory as a CMR page, the node verifies that the corresponding LPA2GA translation table entry that corresponds to that page is available. If the entry is not available, the page is not allocated as CMR space and a different page is selected from the free list. In a different embodiment, as discussed above in reference to FIG. 7, pages are allocated as CMR space by moving page addresses from free memory list 802 to CMR list 804. Prior to moving a page address from free list 802 to CMR list 804, it is verified that the LPA2GA translation table entry that corresponds to that page is available.
  • [0113]
    In the illustrated embodiment, only one of the four pages that map to an entry of LPA2GA translation table 104 may be allocated as CMR space. For example, assume that page 802 is allocated as CMR space. If the node then attempts to allocate page 808 as CMR space, the node will find that entry 826 currently stores a translation for page 802. Accordingly, the node will not allocate page 808 as CMR space and choose another page such as page 804 to allocate as CMR space. If entry 828 is available (i.e., pages 810, 816 and 822 are not allocated as CMR space), then page 804 will be allocated as CMR space and entry 828 is marked as unavailable. Subsequently, page 810, 816 and 822 cannot be allocated as CMR space. A translation may be stored to entry 828 at a later time (e.g., when a shadow page is stored to the page).
  • [0114]
    In other embodiments, more or less pages may be mapped to an entry of LPA2GA translation table 104. For example, eight pages of local physical address space may be mapped to one entry of LPA2GA 104.
  • [0115]
    In the above manner, the size of LPA2GA translation table 104 may be reduced. For example, if four pages are mapped to each entry, the size of LPA2GA translation table 104 is one quarter the size of a conventional LPA2GA translation table. By reducing the size of LPA2GA 104, the entire LPA2GA table may be maintained in a fast memory (e.g., an SRAM look-up table) without the need for a cache. The circuitry of the LPA2GA table is also reduced and the latency associated with a cache miss is eliminated.
  • [0116]
    In one embodiment, the allocation of memory space as CMR space is performed by software (e.g., the operating system of the node). In one particular embodiment, software verifies that the entry of LPA2GA translation table 104 that corresponds to a page is available prior to allocating that page of CMR space. A valid bit within the translation table entries of may be used to indicate that an entry is available or unavailable.
  • [0117]
    The above described system limits the amount of memory that may be allocated as CMR space. For example, if four pages are mapped to each LPA2GA translation table entry, a maximum of 25% of local memory may be allocated as CMR space. Further, 100% utilization of the maximum available CMR space is unlikely. It is reasonable to assume that at least 75% of the maximum available space may be utilized, which is typically sufficient for CMR space.
  • [0118]
    Turning now to FIG. 9, a translation of a local physical address to a global address according to one embodiment of the present invention is shown. In the illustrated embodiment, eight pages of local memory are mapped to each entry in LPA2GA translation table 104. As discussed above in reference to FIG. 8, mapping multiple pages of local memory to one LPA2GA table entry reduces the size of the LPA2GA table. In the illustrated embodiment, the LPA2GA table has 128 k entries for 1 M pages of local physical memory.
  • [0119]
    LPA address 901 is substantially the same as the address discussed above in reference to FIG. 5. LPA address 901 includes coherent field 608, node ID field 606, CMR bit 604 and offset field 602. In the illustrated embodiment, offset field 602 is divided into a page offset field 903 and an LPA page field 904. LPA page field 904 identifies a page of the local memory assigned to the node identified by node ID field 606. In one embodiment, the most significant four bits of LPA page field 904 are reserved. Accordingly, LPA page field 904, which is 24 bits including the reserved bits, may address up to 1M pages per node. Page offset field 903 identifies a byte, or word, within a page. In the illustrated embodiment, page offset field 903 is 13 bits and each page is accordingly 8 k bytes (or 8 k words).
  • [0120]
    LPA2GA table 104 is addressed by the 17 least significant bits of LPA page field 904. It is noted that in other embodiments, LPA2GA table 104 may be addressed by more or less bits. For example, if four pages of physical memory were mapped to each LPA2GA entry, the LPA2GA table may be addressed by 18 bits of LPA page field 904.
  • [0121]
    The format of LPA2GA table entry 915 according to one embodiment is illustrated in FIG. 10. LPA2GA table entry 915 includes a reserve field 916, a valid bit 918, a node ID field 920, a LPA page field 922 and a parity field 924. In other embodiments, a LPA2GA table entry may include additional fields or may omit fields included in table entry 915. Additionally, the fields may include more or less bits than the fields illustrated in FIG. 10. In the illustrated embodiment, reserve field 916 includes five reserve bits. Valid bit 918 indicates whether the corresponding table entry stores valid translation data. If the valid bit is clear, the table entry does not contain a valid translation and is available to store a translation. Node ID field 920 identifies the home node within the multiprocessing system that corresponds to the address. In the illustrated embodiment, node ID field 920 is four bits. Accordingly, sixteen nodes may be accommodated. LPA page field 922 identifies a page within the home node identified by node ID field 920. In the illustrated embodiment, LPA page field 922 is 24 bits. Accordingly, 4M pages may be accommodated. Parity field 924 stores two parity bits to verify the accuracy of the table entry. In one embodiment, the parity bits are checked by hardware each time hardware accesses a table entry, but are not checked by software accesses.
  • [0122]
    Referring back to FIG. 9, the least significant 17 bits of LPA page field 904 of local physical address 901 are used to address LPA2GA translation table 104. In the illustrated embodiment, no address tag is stored in table entry 915, even though multiple pages correspond to a table entry. As discussed above in reference to FIG. 8, only one of the pages that correspond to table entry 915 may be allocated as CMR space. Accordingly, only one translation is stored in each entry of LPA2GA table 104 and no comparisons of address tags are required.
  • [0123]
    Global address 902 includes fields substantially similar to LPA address 901. Global address 902 includes a coherent field 906, a node ID field 908, a reserve bit 910, an LPA page field 912, and a page offset field 914. Portions of global address 902 are taken directly from LPA address 901 and other portions are obtained from fields within the LPA2GA entry addressed by LPA address 901. In the illustrated example, page offset field 904 is taken directly from page offset field 903 of LPA address 901. Node ID field 920 and LPA page field 922 of the table entry addressed by LPA address 901 provide the data for node ID field 908 and LPA page field 912 of global address 902.
  • [0124]
    Turning now to FIG. 11, the organization of a global address to local physical address (GA2LPA) table is shown according to one embodiment of the present invention. Typically, the GA2LPA table of each node in a multiprocessing system must include one entry for each page in the multiprocessing system. For example, in a multiprocessing system with four nodes each including 1 M pages of local physical address space, the GA2LPA table must include 4M entries. The access time associated with a table of that size typically adds unacceptable latency to the transaction. In one embodiment, the access time of the GA2LPA table is reduced by providing a cache to store the most recently accessed GA2LPA translations. The cache is typically backed by memory which stores the entire GA2LPA table. Unfortunately, this solution adds complexity to the GA2LPA table, requires a significant amount of RAM to store the GA2LPA table, and adds significant latency in the case of a cache miss.
  • [0125]
    In an alternative embodiment, the size of GA2LPA table 112 may be reduced by recognizing that only shadow pages need address translations. Multiple global address pages are mapped to each entry in GA2LPA table. Prior to storing data as a shadow page (i.e. storing data in a COMA manner), GA2LPA table 112 is checked to see if the entry in the table that corresponds to the global address is available. If the entry is available, the global address is translated to a local physical address using a page address from CMR list 804 discussed above in reference to FIG. 7. Alternatively, if the corresponding entry in GA2LPA table 112 is unavailable (i.e., the entry is storing a translation), a shadow copy of the data is not stored and the data is stored in NUMA mode. Accordingly, there is some probability that a node may not be able to store data in COMA mode. This probability may be reduced by expanding the size of GA2LPA table 112 or making GA2LPA table 112 more associative.
  • [0126]
    In the embodiment illustrated in FIG. 11, a two-way set associative GA2LPA translation table 112 is shown. Accordingly, two pages associated with one entry of GPA2LPA table 1 12 may be stored as shadow pages. If one way of an entry is occupied, data may still be stored as to a shadow page and the conversion entered in the second way of the entry. If both entries contain valid translations, the page may not be stored as a shadow page and is stored in NUMA mode.
  • [0127]
    As discussed above, only pages which have a valid translation in the GA2LPA table are converted to shadow pages. If a global address received as part of a request does not have a corresponding translation in the GA2LPA table, then no shadow page exists that corresponds that global address. Accordingly, no GA2LPA translation is required. Any data that corresponds to the global address on that node is stored in NUMA mode and accordingly the global address may be used to access the data. In other words, the absence of a translation in GA2LPA table 112 provides information to the node that the page has only been stored in NUMA mode on that node.
  • [0128]
    As discussed in more detail below, a portion of the global address is used to address an entry of GA2LPA table 112. Because multiple pages are mapped to one entry, a portion of the global address (typically more significant bits than the bits used to address the entry) is compared to address tags stored with each entry. If the bits of the global address match either of the address tags, then GA2PLA table 112 stores a translation for the address and the translation data is used to form a local physical address. Alternatively, if the address tags do not match the bits of the global address, no translation exists for that global address and the global address is used to address any data on the node.
  • [0129]
    In other embodiments, other organizations for GA2LPA table 112 may be implemented. For example, GA2LPA table 112 may be organized as a four-way set associative table. Generally speaking, increasing the associatively of the table decreases the probability of not being able to store data in COMA mode. For example, if a four-way set associative GA2LPA table is used and the table is twice the size of the corresponding LPA2GA table, the probability of available space in the GA2LPA table is 98%, assuming that 75% of the available CMR memory is used. If only 50% of the available CMR space is used, the probability of finding available space in the GA2LPA table is 99.6%.
  • [0130]
    One possible organization of a four-way set associative cache is to put the address tags of all four ways in one word. This address tag word is accessed first. If none of the address tags match the bits of the global address, the address does not have a GA2LPA translation and no more accesses to GA2LPA table 112 are required. If one address tag matches the bits of the global address, the way that corresponds to the global address may be determined and the translation information corresponding to the way accessed. Alternatively, the four-ways may be sequentially accessed and the address tags compared to the bits of the global address. The same strategies may be used with other table organizations, such as a two-way set associative table.
  • [0131]
    Turning now to FIG. 12A, an alternative organization of a GA2LPA table is shown according to one embodiment of the present invention. GA2LPA table 122 is organized as a skewed-associative cache. Index function 124 and index function 126 convert the global address into two different look-up addresses for GA2LPA table 122. The address tags stored in the entries addressed by the look-up addresses are compared to some portion of the global address. The comparison must include enough bits such that the combination of the lookup location and the address tag uniquely identifies one GA address. If the address tag of an entry matches the global address, the translation data stored in that entry is used to form the local physical address. Alternatively, if neither address tag matches the global address, the global address is used to access the data. In other words, if neither entry stores a GPA2LPA translation for that global address, then the data that corresponds to the global address is stored in NUMA mode within the node and the data is accessed using the global address.
  • [0132]
    The look-up address generated by index function 126 for one address may be identical to the index function generated by index function 124 for a different address. In one embodiment, the look-up address generated by index function 124 is a subset of the address bits of the global address. In one embodiment, the look-up address generated by index function 126 may be the bit-wise exclusive OR of a plurality of bits within the global address.
  • [0133]
    Each entry in GA2LPA table 122 includes an address tag field 127, a mode bit 128 and a translation field 129. As discussed above, address tag field 127 stores the address tag of the global address that corresponds to an entry. Mode bit 128 is required to prevent false matches. The mode bit indicates whether the entry of the stored translation was derived using index function 124 or index function 126. The address tags and mode bit must match in order to select a table entry. Translation field 129 stores the data necessary to generate a local physical address from the global address. Translation field 129 is discussed in more detail below in reference to FIG. 13.
  • [0134]
    In one embodiment, when storing translations to GA2LPA table 122, the translation data is first attempted to be stored in the entry addressed by index function 124 (referred to herein as the primary entry). If the primary entry is used by another translation, the translation information is then attempted to be stored in the entry addressed by index function 126 (referred to herein as the secondary entry). If the secondary entry is also occupied, no translation is stored for that global address and no shadow page is allocated for that global address.
  • [0135]
    Turning now to FIG. 12B, a method for maximizing utilization of tables such as GA2LPA table 122 according to one embodiment of the present invention is shown. FIG. 12B illustrates a plurality of entries 132-142 in GA2LPA table 122. Columns 144, 146 and 148 illustrate the look-up addresses for a plurality of global addresses. Each global address has a primary entry and a secondary entry in GA2LPA table 122. In the illustrated embodiment, the primary entry is identified by “P” next to the entry and a secondary entry is identified “S” next to the entry. The entry in which the translation is stored is identified by a circle around the letter identifying the entry. In one embodiment, the primary entry corresponds to the look-up address generated by index function 124 and the secondary entry corresponds to the look-up address generated by index function 126. For example, the primary entry corresponding to global address 1 is entry 132 and the secondary entry corresponding to global address 1 is entry 138. In the illustrated embodiment, the translation for global address 1 is stored in the primary entry, which is entry 132. In a similar manner, the primary entry corresponding to global address 2 is entry 140 and the secondary entry is entry 136. In the illustrated embodiment, the translation for global address 2 is stored in entry 136. The primary entry corresponding to global address 3 is 136 and the secondary entry is entry 132.
  • [0136]
    In one embodiment, the translation for global address 3 cannot be stored in GA2LPA table 122 because both entries associated with global address are 3 occupied by other translations. The primary entry associated with global address 3 (entry 136) is occupied by global address 2 and the secondary entry (entry 132) is occupied by global address 1. To improve the availability of entries in GPA2LPA table 122, the translations for either global address 1 or global address 2 may be moved to the other entry associated with that address. For example, the translation for global address 2 is stored in the secondary entry associated with global address 2 (entry 136). If the translation is moved to the primary entry (entry 140), then entry 136 is available to store the translation for global address 3. Alternatively, the translation for global address 1 could be moved from entry 132 to entry 138, which makes entry 132 available to store the translation for global address 3. In this manner, the utilization of GA2LPA table 122 may be increased.
  • [0137]
    The utilization of the table approaches the utilization of a fully associative table while maintaining a relatively simple look-up function. Only two entries need to be accessed during look-up. In other words, from the look-up standpoint, the table is similar to a two-way skewed associative cache. The utilization of the table, however, approaches the utilization of a fully associative table. In one embodiment, software performs the realignment function of moving translations between entries to make space available for new entries.
  • [0138]
    Turning now to FIG. 12C, another illustration of a method for increasing the utilization of a translation table is shown. In the illustrated embodiment, the primary and secondary entries associated with five global addresses are shown in columns 152-160. The primary entry associated with global address 1 is entry 132 and the secondary entry is entry 138. The translation is stored in entry 132. The primary entry associated with global address 2 is entry 140 and the second entry is 136. The translation is stored in entry 136. The primary entry associated with global address 3 is entry 138 and the secondary entry is entry 142. The translation is stored in entry 138. The primary entry associated with global address 4 is entry 134 and the secondary entry is entry 140. The translation is stored in secondary entry 140. In a similar manner to that discussed above in reference to FIG. 12B, the translation for global address 5 cannot be stored in GA2LPA table 122 absent a method for improving the utilization of GA2LPA table 122.
  • [0139]
    The translation for global address 5 cannot be stored in the table because both the primary and secondary entries associated with global address 5, entries 136 and 132 respectively, are occupied by translations of other global addresses. The translation for global address 1 cannot be moved from entry 132 to entry 138 because entry 138 is currently occupied by the translation of global address 3. Likewise, the translation for global address 2 cannot be moved from entry 136 to entry 140 because entry 140 is occupied by the translation for global address 4. In order to make an entry available in GA2LPA table 122 for the translation of global address 5, either the entry storing the translation for global address 3 or global address 4 is moved. This allows the translation of either global address 1 or global address 2 to be moved, which allows the translation of global address 5 to be stored in GA2LPA table 122. For example, the translation for global address 3 may be moved from entry 138 to entry 142. The translation for global address 1 then may be moved from entry 132 to entry 138. The translation for global address 5 then may be stored in entry 132. Alternatively, the translation for global address 4 may be moved from entry 140 to entry 134. The translation of global address 2 may then be moved from entry 136 to entry 140 and the translation for global address 5 stored in entry 136.
  • [0140]
    The methodology illustrated in FIGS. 12B and 12C may be repeated for several iterations before an entry becomes available. Although the realignment of the GA2LPA table may be time consuming, the overhead is only incurred once for each new translation. Additionally, the realignment can occur off the critical path of the processor. In one embodiment, only one new translation can be added at one time. Although the method for increasing the utilization of a table is described above in reference to GPA2LPA table 122, it is noted that the methodology may be applied to any table that employs skewing or hashing functions.
  • [0141]
    Turning now to FIG. 13, a translation of a global address 902 to a local physical address 901 according to one embodiment of the present invention is shown. The fields of global address 902 and local physical address 901 are substantially similar to the fields discussed above in reference to FIG. 9. In the illustrated embodiment, page offset field 914 from global address 902 is copied to page offset field 903 of local physical address 901. Because address 901 is a local physical address, node ID field 606 identifies the home node of the local physical address. In one embodiment, CMR bit 604 is asserted in the local physical address because the local physical address identifies a shadow page of the page identified by global address 902. LPA page field 904 is obtained from an output of GA2LPA table 122.
  • [0142]
    In the illustrated embodiment, the least significant 17 bits of LPA page field 912 are provided to index function 124 and index function 126. The address tags 132 from the two entries that correspond to the look-up addresses output by index function 124 and index function 126 are compared to node ID field 908 and 7 bits of LPA page field 912 by comparator 134. If a match is found, the entry with the matching address tag outputs the translation information to LPA page field 904. In the illustrated embodiment, the two most significant bits of LPA page field 904 are always 0 to reduce the number of bits stored in each entry of GA2LPA table 122.
  • [0143]
    In the illustrated embodiment, each entry is GA2LPA table 122 includes an 11-bit address tag, a 22-bit LPA page translation, a mode bit, and two parity bits.
  • [0144]
    Turning now to FIG. 14A, a flowchart illustrating the storage of shadow pages and the allocation of entries within a GA2LPA table is shown. In step 202, portions of memory are allocated as CMR space. As discussed above, CMR space is used to store shadow copies of data from remote nodes. Step 202 is discussed in more detail below in reference to FIG. 14B. In step 204, a request to store a shadow copy of data (i.e., store data in COMA mode) is received. As discussed above, shadow pages are assigned a local physical address such that future accesses to the data are local rather than global. As discussed in more detail below, an entry within the GA2LPA table must be available before a node will allow a shadow page to be stored.
  • [0145]
    In decisional step 206, it is determined whether a GA2LPA table entry that corresponds with the global address of the data to be stored in the shadow page is available. In one embodiment, in order to reduce the number of entries in the GA2LPA table, multiple global addresses are assigned to each entry in the GA2LPA table. In one particular embodiment, the GA2LPA table is a set associative table such that translations of multiple global addresses that correspond to one entry may be stored. If the entry associated with the global address is available, then in step 208, the data is stored to a shadow page in the CMR space and the address of the page is removed from the CMR list. In step 210, the translation data for translating between the global address and local physical address is stored to the appropriate entries in the GA2LPA table and the LPA2GA table.
  • [0146]
    If in decisional step 206 no entry is available in the GA2LPA table, then in step 212, the GA2LPA table may be realigned. Realignment is discussed in more detail below in reference to FIG. 14C. In step 214, it is determined whether the realignment of step 212 was successful (i.e., a table entry corresponding the address is available). If the realignment was successful, then steps 208 and 210 described above are performed. Alternatively, if the realignment of step 212 was unsuccessful, then in step 216 the data is stored in NUMA mode.
  • [0147]
    Turning now to FIG. 14B, a flowchart illustrating the allocation of CMR space according to one embodiment of the present invention is shown. In step 218, the pages of the local memory of a node are mapped to entries in an LPA2GA table. In one embodiment, multiple pages of local memory are mapped to each entry in the LPA2GA table. In one particular embodiment, four pages of local memory are mapped to each entry in the LPA2GA table. Mapping multiple pages of local memory to each entry in the LPA2GA table effectively reduces the size of the LPA2GA table. However, as discussed in more detail below, pages may only be allocated in CMR space if a entry is available in the LPA2GPA table for storing the translation for that page.
  • [0148]
    In step 220, a page address from a free memory list is retrieved. In one embodiment, the free memory list is a list of addresses of pages which have not been allocated for storage. In the illustrated embodiment, CMR space is allocated by moving page addresses from the free memory list to a CMR list. Accordingly, the CMR list stores page addresses of available pages allocated as CMR space.
  • [0149]
    In decisional step 222, it is determined whether the LPA2GA table entry that corresponds to the retrieved page address is available. As discussed above, in one embodiment, multiple page addresses are mapped to one entry in the LPA2GA table. If an entry stores a translation for a page mapped to the same entry, the entry is not available. If the entry is not available, then in step 224, a new page address is retrieved from the free memory list. Steps 222 and 224 are repeated until a page address from the free memory list with an available entry in LPA2GA table is retrieved.
  • [0150]
    In step 226, the page address for the retrieved address with an available LPA2GA table entry is moved from the free memory list to the CMR list. In step 228, the LPA2GA table entry that corresponds to the retrieved page address is marked as unavailable. In another alternative embodiment, steps 226 and 228 may be performed in parallel. In another alternative embodiment, step 228 may be performed before step 226. In one embodiment, a table entry is marked as unavailable by asserting a valid bit within the table entry.
  • [0151]
    Turning now to FIG. 14C, a flowchart illustrating the realignment of entries in a GA2LPA table according to one embodiment of the present invention is shown. It is noted, that the realignment of a GA2LPA table is shown for a illustrative purposes only. The same methodology may be used for any table employing primary and secondary entries for an address, such as a table employing skewed associativity or hashing functions. The flowchart contemplates a table in which each address is mapped to a primary entry and a secondary entry. If both the primary entry and secondary entry of an address are occupied by other translations, the entries are realigned by moving the translation from the primary entry to the secondary entry or from the secondary entry to the primary entry. In this manner, an entry may be made available for storing a new translation. Several iterations of realignment may be required before an entry is made available.
  • [0152]
    [0152]FIG. 14C contemplates an instance in which both the primary and secondary entries of an address are occupied. In FIG. 14C, a flowchart for the realignment of a GA2LPA table to make the primary entry of a new translation available is shown. It is noted, that the same methodology may be used to realign the GA2LPA table such that the secondary entry of the new translations is available. It is contemplated, that the realignment to make the primary entry available and the realignment to make a secondary entry available are performed concurrently. The first entry made available is used for the translation and the realignment is suspended.
  • [0153]
    In step 230, the look-up address for the alternate entry of the translation stored in the primary entry is computed. For example, if the primary entry is a secondary entry of another translation, the look-up address of the primary entry of the translation is computed. In one embodiment, a mode bit indicates whether an entry corresponds to the primary or secondary entry of the address. The look-up address of the alternate entry may be determined by applying the inverse of the index function used to generate the entry address to obtain the original global address and then applying another index function. The mode bit may indicate that the translation stored in the primary entry is the secondary entry for the translation. The inverse of the index function used to generate the secondary entry look-up address is applied to the entry address, which outputs the global address of the translation. The primary index function is then applied to the global address to compute the look-up address of the primary entry of the translation.
  • [0154]
    In step 232, it is determined whether the alternate entry is available. If the alternate entry is available, then in step 234, the translation stored in the primary entry is moved to its alternate entry. The primary entry is now available to store a new translation.
  • [0155]
    Alternatively, if the alternate entry of the translation stored in the primary entry is unavailable, then in step 236, the alternate entry of the alternate entry of the translation stored in the primary entry is computed. It is determined whether this entry is available in decisional step 238. If the entry is available, then in step 240, the translation stored in the alternate entry of the translation stored in the primary entry is moved to its alternate entry. The alternate entry for the translation stored in the primary entry is now available. In step 242, the translation stored in the primary entry is moved to its alternate entry. The primary entry is now available to store the new translation.
  • [0156]
    If the translation in step 238 is not available, the look-up address for the alternate entry of the alternate entry of the alternate entry stored in the primary entry is computed. Steps similar to steps 238-244 are repeated until the table has been realigned to make space available for the new translation or until a predetermined number of iterations has been performed without successfully realigning the table.
  • [0157]
    Although SMP nodes 12 have been described in the above exemplary embodiments, generally speaking an embodiment of computer system 10 may include one or more processing nodes. As used herein, a processing node includes at least one processor and a corresponding memory. Additionally, circuitry for communicating with other processing nodes is included. When more than one processing node is included in an embodiment of computer system 10, the corresponding memories within the processing nodes form a distributed shared memory. A processing node may be referred to as remote or local. A processing node is a remote processing node with respect to a particular processor if the processing node does not include the particular processor. Conversely, the processing node which includes the particular processor is that particular processor's local processing node.
  • [0158]
    Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.

Claims (16)

    What is claimed is:
  1. 1. A multiprocessor computer system comprising:
    a first node including:
    a first processor;
    a first local memory coupled to said first processor, wherein a first address space and a second address space are mapped to said first local memory; and
    a first system interface coupled to said first processor;
    a second node including:
    a second processor;
    a second local memory coupled to said second processor; and
    a second system interface coupled to said second processor;
    a global bus coupled to said first system interface and said second system interface;
    wherein said first address space is configured to store data local to said first node and said second address space is configured to store copies of data local to said second node; and
    wherein said data stored in said second address space is assigned a physical address local to said first node; and
    wherein in response to detecting that a local address of a transaction initiated by said first processor corresponds to data which is not local to said first node, said first system interface is configured to convert said local address of said transaction to a global address prior to performing a request corresponding to said transaction on said global bus; and
    wherein in response to detecting that another local address of another transaction initiated by said first processor corresponds to data local to said first node, said first system interface is configured to perform a corresponding request on said global bus using said another local address without conversion.
  2. 2. The multiprocessing computer system of
    claim 1
    wherein said first node further comprises a memory tag register configured to store the access rights of the first node to data stored in said first memory and wherein said first node is configured to determine whether said first node has sufficient access rights to perform the memory access prior to performing a memory access, and to perform a coherency operation to acquire sufficient access rights if said first node has insufficient access rights to perform said memory access.
  3. 3. The multiprocessing computer system of
    claim 2
    wherein memory accesses include read accesses and write accesses.
  4. 4. The multiprocessing computer system of
    claim 3
    wherein said access rights include modify, own and share access rights.
  5. 5. The multiprocessing computer system of
    claim 4
    wherein sufficient access rights for a write access include modify access rights and sufficient access rights for a read access include modify, own or share access rights.
  6. 6. The multiprocessing computer system of
    claim 2
    wherein said request on said global bus is a coherency operation to obtain sufficient access rights for a memory access to data stored in said second address space.
  7. 7. The multiprocessing computer system of
    claim 6
    wherein said request on said global bus is a read-to-own request.
  8. 8. The multiprocessing computer system of
    claim 6
    wherein said request on said global bus is a read-to-share request.
  9. 9. The multiprocessing computer system of
    claim 1
    wherein translation data to convert said local address to said global address is stored in a local physical address to global address translation table.
  10. 10. The system of
    claim 1
    , wherein said detecting that said another local address corresponds to data local to said first node comprises detecting a bit of said another local address that indicates said data is local to said first node.
  11. 11. A method of performing selective address translation in a multiprocessing computer system comprising:
    mapping a first address space and a second address space to a local memory of a first node of said multiprocessing computer system;
    storing data local to said first node in said first address space;
    storing copies of data local to a second node of said multiprocessing computer system in said second address space, wherein said copies of data stored in said second address space are assigned local addresses of said first node;
    converting a local address of a transaction on said first node to a global address prior to performing a global operation corresponding to said transaction, in response to detecting said local address corresponds to data which is not local to said first node; and
    performing another global operation using another local address of another transaction without conversion, in response to detecting that said another local address corresponds to data local to said first node.
  12. 12. The method of
    claim 11
    further comprising:
    said first node receiving a request to access a first datum in local memory;
    determining said first node's access rights to said first datum;
    if said first node's access rights are insufficient to perform said access request, performing a coherency operation to obtain sufficient access rights.
  13. 13. The method of
    claim 12
    wherein said first node includes a first memory tag register configured to store access rights to data stored in local memory.
  14. 14. The method of
    claim 12
    wherein said global operation is a coherency operation to obtain access rights.
  15. 15. The method of
    claim 11
    further comprising storing a translation from said local address to said global address and storing a translation from said global address to said local address.
  16. 16. The method of
    claim 11
    , wherein said detecting that said another local address corresponds to data local to said first node comprises detecting a bit of said another local address that indicates said data is local to said first node.
US09875233 1997-09-05 2001-06-05 Selective address translation in coherent memory replication Expired - Lifetime US6446185B2 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US92438597 true 1997-09-05 1997-09-05
US09148738 US6351795B1 (en) 1997-09-05 1998-09-04 Selective address translation in coherent memory replication
US09875233 US6446185B2 (en) 1997-09-05 2001-06-05 Selective address translation in coherent memory replication

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US09875233 US6446185B2 (en) 1997-09-05 2001-06-05 Selective address translation in coherent memory replication
US10198905 US6618799B2 (en) 1997-09-05 2002-07-19 Selective address translation in coherent memory replication

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US09148738 Continuation US6351795B1 (en) 1997-09-05 1998-09-04 Selective address translation in coherent memory replication

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US10198905 Continuation US6618799B2 (en) 1997-09-05 2002-07-19 Selective address translation in coherent memory replication

Publications (2)

Publication Number Publication Date
US20010027512A1 true true US20010027512A1 (en) 2001-10-04
US6446185B2 US6446185B2 (en) 2002-09-03

Family

ID=25450157

Family Applications (11)

Application Number Title Priority Date Filing Date
US09148734 Expired - Lifetime US6401174B1 (en) 1997-09-05 1998-09-04 Multiprocessing computer system employing a cluster communication error reporting mechanism
US09148541 Expired - Lifetime US6370585B1 (en) 1997-09-05 1998-09-04 Multiprocessing computer system employing a cluster communication launching and addressing mechanism
US09148738 Expired - Lifetime US6351795B1 (en) 1997-09-05 1998-09-04 Selective address translation in coherent memory replication
US09148820 Expired - Fee Related US6308246B1 (en) 1997-09-05 1998-09-04 Skewed finite hashing function
US09148736 Expired - Lifetime US6332165B1 (en) 1997-09-05 1998-09-04 Multiprocessor computer system employing a mechanism for routing communication traffic through a cluster node having a slice of memory directed for pass through transactions
US09148218 Expired - Lifetime US6240501B1 (en) 1997-09-05 1998-09-04 Cache-less address translation
US09148820 Granted US20010042176A1 (en) 1997-09-05 1998-09-04 Skewed finite hashing function
US09148735 Expired - Lifetime US6449700B2 (en) 1997-09-05 1998-09-04 Multiprocessing computer system employing a cluster protection mechanism
US09875233 Expired - Lifetime US6446185B2 (en) 1997-09-05 2001-06-05 Selective address translation in coherent memory replication
US09940172 Expired - Lifetime US6654866B2 (en) 1997-09-05 2001-08-27 Skewed finite hashing function
US10198905 Expired - Lifetime US6618799B2 (en) 1997-09-05 2002-07-19 Selective address translation in coherent memory replication

Family Applications Before (8)

Application Number Title Priority Date Filing Date
US09148734 Expired - Lifetime US6401174B1 (en) 1997-09-05 1998-09-04 Multiprocessing computer system employing a cluster communication error reporting mechanism
US09148541 Expired - Lifetime US6370585B1 (en) 1997-09-05 1998-09-04 Multiprocessing computer system employing a cluster communication launching and addressing mechanism
US09148738 Expired - Lifetime US6351795B1 (en) 1997-09-05 1998-09-04 Selective address translation in coherent memory replication
US09148820 Expired - Fee Related US6308246B1 (en) 1997-09-05 1998-09-04 Skewed finite hashing function
US09148736 Expired - Lifetime US6332165B1 (en) 1997-09-05 1998-09-04 Multiprocessor computer system employing a mechanism for routing communication traffic through a cluster node having a slice of memory directed for pass through transactions
US09148218 Expired - Lifetime US6240501B1 (en) 1997-09-05 1998-09-04 Cache-less address translation
US09148820 Granted US20010042176A1 (en) 1997-09-05 1998-09-04 Skewed finite hashing function
US09148735 Expired - Lifetime US6449700B2 (en) 1997-09-05 1998-09-04 Multiprocessing computer system employing a cluster protection mechanism

Family Applications After (2)

Application Number Title Priority Date Filing Date
US09940172 Expired - Lifetime US6654866B2 (en) 1997-09-05 2001-08-27 Skewed finite hashing function
US10198905 Expired - Lifetime US6618799B2 (en) 1997-09-05 2002-07-19 Selective address translation in coherent memory replication

Country Status (5)

Country Link
US (11) US6401174B1 (en)
EP (2) EP1019840B1 (en)
JP (2) JP2001515244A (en)
DE (2) DE69817192D1 (en)
WO (2) WO1999012102A8 (en)

Cited By (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040135677A1 (en) * 2000-06-26 2004-07-15 Robert Asam Use of the data stored by a racing car positioning system for supporting computer-based simulation games
US20060129741A1 (en) * 2004-12-15 2006-06-15 International Business Machines Corporation Method and apparatus for accessing memory in a computer system architecture supporting heterogeneous configurations of memory structures
US7152026B1 (en) * 2001-12-07 2006-12-19 Ncr Corp. Versioned node configurations for parallel applications
US20090153897A1 (en) * 2007-12-18 2009-06-18 Blackmore Robert S Method, System and Program Product for Reserving a Global Address Space
US20090199195A1 (en) * 2008-02-01 2009-08-06 Arimilli Lakshminarayana B Generating and Issuing Global Shared Memory Operations Via a Send FIFO
US20090198971A1 (en) * 2008-02-01 2009-08-06 Arimilli Lakshminarayana B Heterogeneous Processing Elements
US20090199194A1 (en) * 2008-02-01 2009-08-06 Arimilli Lakshminarayana B Mechanism to Prevent Illegal Access to Task Address Space by Unauthorized Tasks
US20090199182A1 (en) * 2008-02-01 2009-08-06 Arimilli Lakshminarayana B Notification by Task of Completion of GSM Operations at Target Node
US20090199209A1 (en) * 2008-02-01 2009-08-06 Arimilli Lakshminarayana B Mechanism for Guaranteeing Delivery of Multi-Packet GSM Message
US20090199200A1 (en) * 2008-02-01 2009-08-06 Arimilli Lakshminarayana B Mechanisms to Order Global Shared Memory Operations
US20090198918A1 (en) * 2008-02-01 2009-08-06 Arimilli Lakshminarayana B Host Fabric Interface (HFI) to Perform Global Shared Memory (GSM) Operations
US20090199191A1 (en) * 2008-02-01 2009-08-06 Arimilli Lakshminarayana B Notification to Task of Completion of GSM Operations by Initiator Node
US20140029616A1 (en) * 2012-07-26 2014-01-30 Oracle International Corporation Dynamic node configuration in directory-based symmetric multiprocessing systems
US20150143066A1 (en) * 2013-10-18 2015-05-21 Hitachi Data Systems Engineering UK Limited Data configuration and migration in a cluster system
US20150149743A1 (en) * 2013-11-27 2015-05-28 Realtek Semiconductor Corp. Management method of virtual-to-physical address translation system using part of bits of virtual address as index
US20160019178A1 (en) * 1999-12-27 2016-01-21 Intel Corporation Memory mapping in a processor having multiple programmable units
US9342452B2 (en) 2011-10-07 2016-05-17 Hewlett Packard Enterprise Development Lp Mapping processor address ranges to persistent storage
WO2016200649A1 (en) * 2015-06-09 2016-12-15 Ultrata Llc Infinite memory fabric streams and apis
US20170153849A1 (en) * 2015-11-26 2017-06-01 Via Alliance Semiconductor Co., Ltd. Methods and apparatuses for generating machine code for driving an execution unit
US9886210B2 (en) 2015-06-09 2018-02-06 Ultrata, Llc Infinite memory fabric hardware implementation with router

Families Citing this family (170)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE69817192D1 (en) * 1997-09-05 2003-09-18 Sun Microsystems Inc Multiprocessor computer system using a group-protective mechanism
DE69806812D1 (en) * 1997-12-19 2002-08-29 Unilever Nv Olive oil-containing food composition
US6826651B2 (en) * 1998-05-29 2004-11-30 International Business Machines Corporation State-based allocation and replacement for improved hit ratio in directory caches
US6308284B1 (en) * 1998-08-28 2001-10-23 Emc Corporation Method and apparatus for maintaining data coherency
US6414955B1 (en) * 1999-03-23 2002-07-02 Innovative Technology Licensing, Llc Distributed topology learning method and apparatus for wireless networks
US6370595B1 (en) * 1999-03-29 2002-04-09 U.S. Philips Corporation Method of addressing a plurality of addressable units by a single address word
US6618698B1 (en) * 1999-08-12 2003-09-09 Quickturn Design Systems, Inc. Clustered processors in an emulation engine
US6457146B1 (en) * 1999-09-30 2002-09-24 Silicon Graphics, Inc. Method and apparatus for processing errors in a computer system
US7529799B2 (en) * 1999-11-08 2009-05-05 International Business Machines Corporation Method and apparatus for transaction tag assignment and maintenance in a distributed symmetric multiprocessor system
US20020067735A1 (en) * 1999-12-13 2002-06-06 Lindstrom Johan Erik Telecommunication system for control of multiple switches in a common address space
US6529968B1 (en) * 1999-12-21 2003-03-04 Intel Corporation DMA controller and coherency-tracking unit for efficient data transfers between coherent and non-coherent memory spaces
US6769046B2 (en) * 2000-02-14 2004-07-27 Palmchip Corporation System-resource router
US7168138B2 (en) * 2000-03-27 2007-01-30 Newfrey Llc Resilient clip fastener
US6667960B1 (en) * 2000-04-29 2003-12-23 Hewlett-Packard Development Company, L.P. Protocol for identifying components in a point-to-point computer system
US6785713B1 (en) 2000-05-08 2004-08-31 Citrix Systems, Inc. Method and apparatus for communicating among a network of servers utilizing a transport mechanism
US6789112B1 (en) 2000-05-08 2004-09-07 Citrix Systems, Inc. Method and apparatus for administering a server having a subsystem in communication with an event channel
US6785726B1 (en) * 2000-05-08 2004-08-31 Citrix Systems, Inc. Method and apparatus for delivering local and remote server events in a similar fashion
US7499872B1 (en) 2000-06-02 2009-03-03 Tuition Fund, Llc Methods and systems for applying rebates to higher education
US6598130B2 (en) * 2000-07-31 2003-07-22 Hewlett-Packard Development Company, L.P. Technique for referencing distributed shared memory locally rather than remotely
US20020124083A1 (en) 2000-09-06 2002-09-05 Sun Microsystems, Inc. Method and apparatus for increasing the efficiency of transactions and connection sharing in an enterprise environment
US6567817B1 (en) * 2000-09-08 2003-05-20 Hewlett-Packard Development Company, L.P. Cache management system using hashing
JP3377994B2 (en) * 2000-11-14 2003-02-17 三菱電機株式会社 Data distribution management device and a data distribution management method
US7016998B2 (en) * 2000-11-27 2006-03-21 Silicon Graphics, Inc. System and method for generating sequences and global interrupts in a cluster of nodes
US7072981B1 (en) 2000-12-21 2006-07-04 Cisco Technology, Inc. Preallocation of client network address translation addresses for client-server networks
US7089328B1 (en) * 2000-12-29 2006-08-08 Cisco Technology, Inc. Method allocation scheme for maintaining server load balancers services in a high throughput environment
US6813522B1 (en) * 2000-12-29 2004-11-02 Emc Corporation Method of sharing memory in a multi-processor system including a cloning of code and data
US7010726B2 (en) * 2001-03-01 2006-03-07 International Business Machines Corporation Method and apparatus for saving data used in error analysis
US6842857B2 (en) * 2001-04-12 2005-01-11 International Business Machines Corporation Method and apparatus to concurrently boot multiple processors in a non-uniform-memory-access machine
US7197536B2 (en) * 2001-04-30 2007-03-27 International Business Machines Corporation Primitive communication mechanism for adjacent nodes in a clustered computer system
US20070220000A1 (en) * 2001-05-09 2007-09-20 Walsh Aaron E Universal Cache
US6711652B2 (en) 2001-06-21 2004-03-23 International Business Machines Corporation Non-uniform memory access (NUMA) data processing system that provides precise notification of remote deallocation of modified data
US6654857B2 (en) 2001-06-21 2003-11-25 International Business Machines Corporation Non-uniform memory access (NUMA) computer system having distributed global coherency management
US6615322B2 (en) 2001-06-21 2003-09-02 International Business Machines Corporation Two-stage request protocol for accessing remote memory data in a NUMA data processing system
US6658538B2 (en) * 2001-06-21 2003-12-02 International Business Machines Corporation Non-uniform memory access (NUMA) data processing system having a page table including node-specific data storage and coherency control
US6760817B2 (en) 2001-06-21 2004-07-06 International Business Machines Corporation Method and system for prefetching utilizing memory initiated prefetch write operations
US6901485B2 (en) 2001-06-21 2005-05-31 International Business Machines Corporation Memory directory management in a multi-node computer system
US6760809B2 (en) 2001-06-21 2004-07-06 International Business Machines Corporation Non-uniform memory access (NUMA) data processing system having remote memory cache incorporated within system memory
US6754782B2 (en) 2001-06-21 2004-06-22 International Business Machines Corporation Decentralized global coherency management in a multi-node computer system
GB2378277B (en) * 2001-07-31 2003-06-25 Sun Microsystems Inc Multiple address translations
US6766386B2 (en) * 2001-08-28 2004-07-20 Broadcom Corporation Method and interface for improved efficiency in performing bus-to-bus read data transfers
US6578126B1 (en) * 2001-09-21 2003-06-10 Emc Corporation Memory system and method of using same
JP4721379B2 (en) * 2001-09-26 2011-07-13 株式会社日立製作所 Storage system, additional method of disk controller clusters and disk control cluster
US20030061462A1 (en) * 2001-09-26 2003-03-27 Fister James D.M. Memory expansion and enhanced system interaction using network-distributed memory mapping
US7330885B2 (en) * 2001-10-25 2008-02-12 Sun Microsystems, Inc. Method and apparatus for managing data time-outs
US20030093626A1 (en) * 2001-11-14 2003-05-15 Fister James D.M. Memory caching scheme in a distributed-memory network
US6839826B2 (en) * 2002-02-06 2005-01-04 Sandisk Corporation Memory device with pointer structure to map logical to physical addresses
US20030154221A1 (en) * 2002-02-13 2003-08-14 Sun Microsystems, Inc. System and method for accessing file system entities
US7085866B1 (en) * 2002-02-19 2006-08-01 Hobson Richard F Hierarchical bus structure and memory access protocol for multiprocessor systems
US6959372B1 (en) * 2002-02-19 2005-10-25 Cogent Chipware Inc. Processor cluster architecture and associated parallel processing methods
US7080156B2 (en) * 2002-03-21 2006-07-18 Sun Microsystems, Inc. Message routing in a torus interconnect
GB2389207B (en) * 2002-04-09 2004-05-12 Via Tech Inc Remote node accessing local memory by using distributed shared memory
CN1152330C (en) * 2002-04-09 2004-06-02 威盛电子股份有限公司 Maintain method for remote node to read local memory and its application device
GB2390702B (en) * 2002-04-22 2006-04-12 Micron Technology Inc Providing a register file memory with local addressing in a SIMD parallel processor
CA2384185A1 (en) * 2002-04-29 2003-10-29 Ibm Canada Limited-Ibm Canada Limitee Resizable cache sensitive hash table
US7451199B2 (en) * 2002-05-10 2008-11-11 International Business Machines Corporation Network attached storage SNMP single system image
US7269709B2 (en) * 2002-05-15 2007-09-11 Broadcom Corporation Memory controller configurable to allow bandwidth/latency tradeoff
US7171521B2 (en) * 2002-10-16 2007-01-30 Broadcom Corporation Coherent shared memory processing system
US7266587B2 (en) * 2002-05-15 2007-09-04 Broadcom Corporation System having interfaces, switch, and memory bridge for CC-NUMA operation
US7155525B2 (en) * 2002-05-28 2006-12-26 Newisys, Inc. Transaction management in systems having multiple multi-processor clusters
US7281055B2 (en) * 2002-05-28 2007-10-09 Newisys, Inc. Routing mechanisms in systems having multiple multi-processor clusters
US7251698B2 (en) * 2002-05-28 2007-07-31 Newisys, Inc. Address space management in systems having multiple multi-processor clusters
US7103636B2 (en) * 2002-05-28 2006-09-05 Newisys, Inc. Methods and apparatus for speculative probing of a remote cluster
US7017012B2 (en) * 2002-06-20 2006-03-21 Sun Microsystems, Inc. Distributed storage cache coherency system and method
US7254741B1 (en) * 2002-09-27 2007-08-07 Emc Corporation Attaining high availability through a power system branch architecture
US6918023B2 (en) * 2002-09-30 2005-07-12 International Business Machines Corporation Method, system, and computer program product for invalidating pretranslations for dynamic memory removal
US7051163B2 (en) * 2002-10-03 2006-05-23 Hewlett-Packard Development Company, L.P. Directory structure permitting efficient write-backs in a shared memory computer system
US20040078732A1 (en) * 2002-10-21 2004-04-22 International Business Machines Corporation SMP computer system having a distributed error reporting structure
US8185602B2 (en) * 2002-11-05 2012-05-22 Newisys, Inc. Transaction processing using multiple protocol engines in systems having multiple multi-processor clusters
US7577755B2 (en) * 2002-11-19 2009-08-18 Newisys, Inc. Methods and apparatus for distributing system management signals
US7028218B2 (en) * 2002-12-02 2006-04-11 Emc Corporation Redundant multi-processor and logical processor configuration for a file server
JP4014155B2 (en) * 2003-01-27 2007-11-28 インターナショナル・ビジネス・マシーンズ・コーポレーションInternational Business Maschines Corporation Information processing apparatus and method, a program, data structures, and computer-readable recording medium
US7418517B2 (en) * 2003-01-30 2008-08-26 Newisys, Inc. Methods and apparatus for distributing system management signals
US9110853B2 (en) * 2003-03-10 2015-08-18 Oracle America, Inc. Computer system with multiple classes of device IDs
US7571252B2 (en) * 2003-03-10 2009-08-04 Sun Microsystems, Inc. Computer system with multiple classes of transaction IDs
WO2004091136A3 (en) * 2003-04-04 2005-01-20 Robert E Cypher Multi-node system in which global address generated by processing subsystem includes global to local translation information
US20050027947A1 (en) * 2003-04-11 2005-02-03 Sun Microsystems, Inc. Multi-node computer system including a mechanism to encode node ID of a transaction-initiating node in invalidating proxy address packets
US20050010615A1 (en) * 2003-04-11 2005-01-13 Sun Microsystems, Inc. Multi-node computer system implementing memory-correctable speculative proxy transactions
US7386626B2 (en) * 2003-06-23 2008-06-10 Newisys, Inc. Bandwidth, framing and error detection in communications between multi-processor clusters of multi-cluster computer systems
US7577727B2 (en) * 2003-06-27 2009-08-18 Newisys, Inc. Dynamic multiple cluster system reconfiguration
GB0315504D0 (en) * 2003-07-02 2003-08-06 Advanced Risc Mach Ltd Coherent multi-processing system
US7117419B2 (en) * 2003-08-05 2006-10-03 Newisys, Inc. Reliable communication between multi-processor clusters of multi-cluster computer systems
US7159137B2 (en) * 2003-08-05 2007-01-02 Newisys, Inc. Synchronized communication between multi-processor clusters of multi-cluster computer systems
US7103823B2 (en) 2003-08-05 2006-09-05 Newisys, Inc. Communication between multi-processor clusters of multi-cluster computer systems
US7395347B2 (en) * 2003-08-05 2008-07-01 Newisys, Inc, Communication between and within multi-processor clusters of multi-cluster computer systems
US7231492B2 (en) * 2003-09-30 2007-06-12 Emc Corporation Data transfer method wherein a sequence of messages update tag structures during a read data transfer
US7325239B2 (en) * 2003-11-12 2008-01-29 International Business Machines Corporation Method and system of generically managing tables for network processors
US20050108300A1 (en) * 2003-11-17 2005-05-19 Terrascale Technologies Inc. Method for the management of local client cache buffers in a clustered computer environment
US7324995B2 (en) * 2003-11-17 2008-01-29 Rackable Systems Inc. Method for retrieving and modifying data elements on a shared medium
JP4023441B2 (en) * 2003-12-09 2007-12-19 日本電気株式会社 Computer systems and programs
US7124253B1 (en) * 2004-02-18 2006-10-17 Sun Microsystems, Inc. Supporting directory-based cache coherence in an object-addressed memory hierarchy
US7162647B2 (en) * 2004-03-11 2007-01-09 Hitachi, Ltd. Method and apparatus for cryptographic conversion in a data storage system
WO2005089235A3 (en) * 2004-03-13 2007-12-21 Cluster Resources Inc System and method providing object messages in a compute environment
US20050216695A1 (en) * 2004-03-26 2005-09-29 Jean-Pierre Bono Memory extension for a data processor to provide both common and separate physical memory areas for virtual memory spaces
US7346813B1 (en) 2004-04-05 2008-03-18 Sun Microsystems, Inc. Distributed event reporting hierarchy
EP1763766A4 (en) * 2004-05-04 2009-04-01 Robert M Price System and method for communicating with electronic devices
US7206915B2 (en) * 2004-06-03 2007-04-17 Emc Corp Virtual space manager for computer having a physical address extension feature
US20060015589A1 (en) * 2004-07-16 2006-01-19 Ang Boon S Generating a service configuration
US20060015772A1 (en) * 2004-07-16 2006-01-19 Ang Boon S Reconfigurable memory system
DE102004035843B4 (en) * 2004-07-23 2010-04-15 Infineon Technologies Ag Router Network Processor
KR101061850B1 (en) * 2004-09-08 2011-09-02 삼성전자주식회사 TFT array panel and a method of manufacturing the same
US7774562B2 (en) * 2004-09-17 2010-08-10 Hewlett-Packard Development Company, L.P. Timeout acceleration for globally shared memory transaction tracking table
US7383423B1 (en) * 2004-10-01 2008-06-03 Advanced Micro Devices, Inc. Shared resources in a chip multiprocessor
US20060168379A1 (en) * 2004-12-13 2006-07-27 Tim Frodsham Method, system, and apparatus for link latency management
US7360008B2 (en) * 2004-12-30 2008-04-15 Intel Corporation Enforcing global ordering through a caching bridge in a multicore multiprocessor system
US20060161718A1 (en) * 2005-01-20 2006-07-20 Berke Stuart A System and method for a non-uniform crossbar switch plane topology
US7281096B1 (en) * 2005-02-09 2007-10-09 Sun Microsystems, Inc. System and method for block write to memory
US8254411B2 (en) * 2005-02-10 2012-08-28 International Business Machines Corporation Data processing system, method and interconnect fabric having a flow governor
US7437617B2 (en) * 2005-02-11 2008-10-14 International Business Machines Corporation Method, apparatus, and computer program product in a processor for concurrently sharing a memory controller among a tracing process and non-tracing processes using a programmable variable number of shared memory write buffers
US7418629B2 (en) * 2005-02-11 2008-08-26 International Business Machines Corporation Synchronizing triggering of multiple hardware trace facilities using an existing system bus
US20060184837A1 (en) * 2005-02-11 2006-08-17 International Business Machines Corporation Method, apparatus, and computer program product in a processor for balancing hardware trace collection among different hardware trace facilities
US7437618B2 (en) * 2005-02-11 2008-10-14 International Business Machines Corporation Method in a processor for dynamically during runtime allocating memory for in-memory hardware tracing
US7376675B2 (en) * 2005-02-18 2008-05-20 International Business Machines Corporation Simulating multi-user activity while maintaining original linear request order for asynchronous transactional events
US8214353B2 (en) * 2005-02-18 2012-07-03 International Business Machines Corporation Support for schema evolution in a multi-node peer-to-peer replication environment
US9286346B2 (en) * 2005-02-18 2016-03-15 International Business Machines Corporation Replication-only triggers
US8037056B2 (en) 2005-02-18 2011-10-11 International Business Machines Corporation Online repair of a replicated table
US7353360B1 (en) * 2005-04-05 2008-04-01 Sun Microsystems, Inc. Method for maximizing page locality
US7813288B2 (en) * 2005-11-21 2010-10-12 Intel Corporation Transaction detection in link based computing system
CN2852260Y (en) * 2005-12-01 2006-12-27 华为技术有限公司 Server
US20070180128A1 (en) * 2006-01-18 2007-08-02 International Business Machines Corporation User account validity definition in clustered computer systems
US7426627B2 (en) * 2006-03-10 2008-09-16 Microsoft Corporation Selective address translation for a resource such as a hardware device
US7577820B1 (en) 2006-04-14 2009-08-18 Tilera Corporation Managing data in a parallel processing environment
US7774579B1 (en) * 2006-04-14 2010-08-10 Tilera Corporation Protection in a parallel processing environment using access information associated with each switch to prevent data from being forwarded outside a plurality of tiles
US20080059469A1 (en) * 2006-08-31 2008-03-06 International Business Machines Corporation Replication Token Based Synchronization
US20080114943A1 (en) * 2006-10-05 2008-05-15 Holt John M Adding one or more computers to a multiple computer system
US7774551B2 (en) * 2006-10-06 2010-08-10 Hewlett-Packard Development Company, L.P. Hierarchical cache coherence directory structure
US7571270B1 (en) 2006-11-29 2009-08-04 Consentry Networks, Inc. Monitoring of shared-resource locks in a multi-processor system with locked-resource bits packed into registers to detect starved threads
JP5079342B2 (en) * 2007-01-22 2012-11-21 ルネサスエレクトロニクス株式会社 Multi-processor system
US7941499B2 (en) * 2007-03-06 2011-05-10 Freescale Semiconductor, Inc. Interprocessor message transmission via coherency-based interconnect
US8037259B2 (en) * 2007-06-01 2011-10-11 General Dynamics Advanced Information Systems, Inc. System and method for managing addresses in a computing system
US7930459B2 (en) * 2007-09-28 2011-04-19 Intel Corporation Coherent input output device
WO2009052525A1 (en) * 2007-10-19 2009-04-23 Virident Systems, Inc. Managing memory systems containing components with asymmetric characteristics
US7844746B2 (en) * 2008-02-01 2010-11-30 International Business Machines Corporation Accessing an effective address and determining whether the effective address is associated with remotely coupled I/O adapters
US8380766B2 (en) * 2008-02-08 2013-02-19 Adaptive Intelligence Llc Systems and methods for handling addresses within a database application
US8397030B2 (en) 2008-06-24 2013-03-12 International Business Machines Corporation Efficient region coherence protocol for clustered shared-memory multiprocessor systems
US8145812B1 (en) 2008-07-25 2012-03-27 Gaurang Kavaiya Line driving and receiving system
US8776063B2 (en) * 2008-11-26 2014-07-08 Oracle America, Inc. Method and system for hardware feedback in transactional memory
US8799587B2 (en) * 2009-01-26 2014-08-05 International Business Machines Corporation Region coherence array for a mult-processor system having subregions and subregion prefetching
US8285942B2 (en) * 2009-01-27 2012-10-09 International Business Machines Corporation Region coherence array having hint bits for a clustered shared-memory multiprocessor system
US7908530B2 (en) * 2009-03-16 2011-03-15 Faraday Technology Corp. Memory module and on-line build-in self-test method thereof for enhancing memory system reliability
WO2010147590A1 (en) * 2009-06-18 2010-12-23 Hewlett-Packard Development Company, L.P. Processor topology switches
US8327228B2 (en) * 2009-09-30 2012-12-04 Intel Corporation Home agent data and memory management
US8799586B2 (en) * 2009-09-30 2014-08-05 Intel Corporation Memory mirroring and migration at home agent
US8370595B2 (en) * 2009-12-21 2013-02-05 International Business Machines Corporation Aggregate data processing system having multiple overlapping synthetic computers
US8364922B2 (en) * 2009-12-21 2013-01-29 International Business Machines Corporation Aggregate symmetric multiprocessor system
US8693133B1 (en) 2010-03-22 2014-04-08 Western Digital Technologies, Inc. Systems and methods for improving sequential data rate performance using sorted data zones for butterfly format
US8687306B1 (en) 2010-03-22 2014-04-01 Western Digital Technologies, Inc. Systems and methods for improving sequential data rate performance using sorted data zones
US9330715B1 (en) 2010-03-22 2016-05-03 Western Digital Technologies, Inc. Mapping of shingled magnetic recording media
US8103937B1 (en) * 2010-03-31 2012-01-24 Emc Corporation Cas command network replication
US9558119B2 (en) * 2010-06-23 2017-01-31 International Business Machines Corporation Main memory operations in a symmetric multiprocessing computer
KR101671494B1 (en) * 2010-10-08 2016-11-02 삼성전자주식회사 Multi Processor based on shared virtual memory and Method for generating address translation table
US9116845B2 (en) 2011-02-23 2015-08-25 Freescale Semiconductor, Inc. Remote permissions provisioning for storage in a cache and device therefor
US8949551B2 (en) 2011-02-23 2015-02-03 Freescale Semiconductor, Inc. Memory protection unit (MPU) having a shared portion and method of operation
US8639895B2 (en) 2011-07-14 2014-01-28 Freescale Semiconductor, Inc. Systems and methods for memory region descriptor attribute override
US8572345B2 (en) 2011-09-16 2013-10-29 Freescale Semiconductor, Inc. Memory management unit (MMU) having region descriptor globalization controls and method of operation
US20130097387A1 (en) * 2011-10-14 2013-04-18 The Board Of Trustees Of The Leland Stanford Junior University Memory-based apparatus and method
US8966187B2 (en) * 2011-12-01 2015-02-24 International Business Machines Corporation Flexible replication with skewed mapping in multi-core chips
US8856438B1 (en) * 2011-12-09 2014-10-07 Western Digital Technologies, Inc. Disk drive with reduced-size translation table
US9213493B1 (en) 2011-12-16 2015-12-15 Western Digital Technologies, Inc. Sorted serpentine mapping for storage drives
US9146705B2 (en) * 2012-04-09 2015-09-29 Microsoft Technology, LLC Split brain protection in computer clusters
US8699185B1 (en) 2012-12-10 2014-04-15 Western Digital Technologies, Inc. Disk drive defining guard bands to support zone sequentiality when butterfly writing shingled data tracks
US9176669B2 (en) * 2013-03-15 2015-11-03 Silicon Graphics International Corp. Address resource mapping in a shared memory computer system
US9864691B1 (en) * 2013-12-13 2018-01-09 EMC IP Holding Company LLC Deletion indication implementation based on internal model
JP6219514B2 (en) * 2013-12-19 2017-10-25 インテル・コーポレーション Computing device that provides a virtual multipath conditions accessible, the remote computing device for virtual multipath, a method of providing a virtual multipath conditions accessible method for virtual multipath, computing device, a plurality of methods to computing device program for execution, and machine-readable recording medium
US9411644B2 (en) 2014-03-07 2016-08-09 Cavium, Inc. Method and system for work scheduling in a multi-chip system
US9372800B2 (en) 2014-03-07 2016-06-21 Cavium, Inc. Inter-chip interconnect protocol for a multi-chip system
US9529532B2 (en) * 2014-03-07 2016-12-27 Cavium, Inc. Method and apparatus for memory allocation in a multi-node system
US9575881B2 (en) 2014-12-04 2017-02-21 Qualcomm Incorporated Systems and methods for providing improved latency in a non-uniform memory architecture
US9542333B2 (en) 2014-12-04 2017-01-10 Qualcomm Incorporated Systems and methods for providing improved latency in a non-uniform memory architecture
WO2016160200A8 (en) * 2015-03-27 2016-11-03 Intel Corporation Pooled memory address translation
US9424155B1 (en) 2016-01-27 2016-08-23 International Business Machines Corporation Use efficiency of platform memory resources through firmware managed I/O translation table paging

Family Cites Families (60)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4164787A (en) 1977-11-09 1979-08-14 Bell Telephone Laboratories, Incorporated Multiple microprocessor intercommunication arrangement
DE3066644D1 (en) 1979-12-03 1984-03-22 Honeywell Inf Systems Multiprocessor system
US4463420A (en) * 1982-02-23 1984-07-31 International Business Machines Corporation Multiprocessor cache replacement under task control
EP0113379B1 (en) 1982-12-13 1987-09-30 Fraunhofer-Gesellschaft Zur Förderung Der Angewandten Forschung E.V. Coupler for processors
DE3334797A1 (en) 1983-09-26 1985-01-03 Siemens Ag Multiprocessor computer, in particular multi-processor central control unit of an automatic telephone switching system-
US4814982A (en) 1984-12-24 1989-03-21 General Electric Company Reconfigurable, multiprocessor system with protected, multiple, memories
JPH0542702B2 (en) 1984-12-24 1993-06-29 Sony Corp
US5021945A (en) * 1985-10-31 1991-06-04 Mcc Development, Ltd. Parallel processor system for processing natural concurrencies and method therefor
US4933846A (en) * 1987-04-24 1990-06-12 Network Systems Corporation Network communications adapter with dual interleaved memory banks servicing multiple processors
US5341483A (en) 1987-12-22 1994-08-23 Kendall Square Research Corporation Dynamic hierarchial associative memory
US5117350A (en) * 1988-12-15 1992-05-26 Flashpoint Computer Corporation Memory address mechanism in a distributed memory architecture
US5339398A (en) 1989-07-31 1994-08-16 North American Philips Corporation Memory architecture and method of data organization optimized for hashing
US5247673A (en) 1989-12-04 1993-09-21 Bull Hn Information Systems Italia S.P.A. Multiprocessor system having distributed shared resources and dynamic global data replication
US5161156A (en) * 1990-02-02 1992-11-03 International Business Machines Corporation Multiprocessing packet switching connection system having provision for error correction and recovery
DE4034333A1 (en) 1990-05-29 1991-12-05 Grossenhainer Textilmaschinen Contactless monitor for yarn, etc. - has ultrasonic transmitter and receiver flanking material at gap from it for monitoring without being affected by local environment
DE4034444C2 (en) 1990-10-30 1992-08-20 Ant Nachrichtentechnik Gmbh, 7150 Backnang, De
US5590345A (en) 1990-11-13 1996-12-31 International Business Machines Corporation Advanced parallel array processor(APAP)
US5794059A (en) 1990-11-13 1998-08-11 International Business Machines Corporation N-dimensional modified hypercube
US5765011A (en) 1990-11-13 1998-06-09 International Business Machines Corporation Parallel processing system having a synchronous SIMD processing with processing elements emulating SIMD operation using individual instruction streams
US5237673A (en) * 1991-03-20 1993-08-17 Digital Equipment Corporation Memory management method for coupled memory multiprocessor systems
US5410654A (en) * 1991-07-22 1995-04-25 International Business Machines Corporation Interface with address decoder for selectively generating first and second address and control signals respectively in response to received address and control signals
US5485594A (en) * 1992-07-17 1996-01-16 International Business Machines Corporation Apparatus and method using an atomic fetch and add for establishing temporary ownership of a common system resource in a multiprocessor data processing system
WO1994003856A1 (en) * 1992-08-07 1994-02-17 Massachusetts Institute Of Technology Column-associative cache
US5647056A (en) * 1992-11-18 1997-07-08 Canon Information Systems, Inc. Method and apparatus for managing access to a networked peripheral
JPH084273B2 (en) * 1992-11-30 1996-01-17 日本電気株式会社 Complex communication network
US5590308A (en) * 1993-09-01 1996-12-31 International Business Machines Corporation Method and apparatus for reducing false invalidations in distributed systems
US5701482A (en) * 1993-09-03 1997-12-23 Hughes Aircraft Company Modular array processor architecture having a plurality of interconnected load-balanced parallel processing nodes
EP0720797B1 (en) * 1993-09-24 2001-12-05 Qualcomm Incorporated Multirate serial viterbi decoder for code division multiple access system applications
US5499359A (en) * 1994-01-18 1996-03-12 Borland International, Inc. Methods for improved referential integrity in a relational database management system
US5392280A (en) 1994-04-07 1995-02-21 Mitsubishi Electric Research Laboratories, Inc. Data transmission system and scheduling protocol for connection-oriented packet or cell switching networks
US5539892A (en) * 1994-08-02 1996-07-23 Motorola, Inc. Address translation lookaside buffer replacement apparatus and method with user override
EP0745941B1 (en) 1995-06-02 2003-08-06 Sun Microsystems, Inc. A system and method for providing a flexible memory hierarchy
US5613071A (en) 1995-07-14 1997-03-18 Intel Corporation Method and apparatus for providing remote memory access in a distributed memory multiprocessor system
US5924125A (en) 1995-08-01 1999-07-13 Arya; Siamak Method and apparatus for parallel access to consecutive TLB entries
US5710907A (en) 1995-12-22 1998-01-20 Sun Microsystems, Inc. Hybrid NUMA COMA caching system and methods for selecting between the caching modes
US5694335A (en) * 1996-03-12 1997-12-02 Hollenberg; Dennis D. Secure personal applications network
US5761716A (en) * 1996-05-01 1998-06-02 International Business Machines Corporation Rate based memory replacement mechanism for replacing cache entries when the cache is full
US5940870A (en) * 1996-05-21 1999-08-17 Industrial Technology Research Institute Address translation for shared-memory multiprocessor clustering
US5802568A (en) * 1996-06-06 1998-09-01 Sun Microsystems, Inc. Simplified least-recently-used entry replacement in associative cache memories and translation lookaside buffers
US5860146A (en) * 1996-06-25 1999-01-12 Sun Microsystems, Inc. Auxiliary translation lookaside buffer for assisting in accessing data in remote address spaces
US5860109A (en) * 1996-07-01 1999-01-12 Sun Microsystems, Inc. Methods and apparatus for a coherence transformer for connecting computer system coherence domains
US5887138A (en) 1996-07-01 1999-03-23 Sun Microsystems, Inc. Multiprocessing computer system employing local and global address spaces and COMA and NUMA access modes
US5897664A (en) * 1996-07-01 1999-04-27 Sun Microsystems, Inc. Multiprocessor system having mapping table in each node to map global physical addresses to local physical addresses of page copies
US5734922A (en) * 1996-07-01 1998-03-31 Sun Microsystems, Inc. Multiprocessing system configured to detect and efficiently provide for migratory data access patterns
US5862316A (en) * 1996-07-01 1999-01-19 Sun Microsystems, Inc. Multiprocessing system having coherency-related error logging capabilities
US5862357A (en) 1996-07-02 1999-01-19 Sun Microsystems, Inc. Hierarchical SMP computer system
US5845071A (en) * 1996-09-27 1998-12-01 Hewlett-Packard Co. Error containment cluster of nodes
US5920900A (en) * 1996-12-30 1999-07-06 Cabletron Systems, Inc. Hash-based translation method and apparatus with multiple level collision resolution
US5802602A (en) 1997-01-17 1998-09-01 Intel Corporation Method and apparatus for performing reads of related data from a set-associative cache memory
US5884018A (en) * 1997-01-28 1999-03-16 Tandem Computers Incorporated Method and apparatus for distributed agreement on processor membership in a multi-processor system
US6151688A (en) * 1997-02-21 2000-11-21 Novell, Inc. Resource management in a clustered computer system
US5992290A (en) 1997-04-09 1999-11-30 Cubic Defense Systems, Inc. Aircraft interface device and crossover cable kit
US5968176A (en) * 1997-05-29 1999-10-19 3Com Corporation Multilayer firewall system
US5887134A (en) * 1997-06-30 1999-03-23 Sun Microsystems System and method for preserving message order while employing both programmed I/O and DMA operations
US6014710A (en) * 1997-06-30 2000-01-11 Sun Microsystems, Inc. System and method for message transmission between network nodes using remote wires
DE69817192D1 (en) * 1997-09-05 2003-09-18 Sun Microsystems Inc Multiprocessor computer system using a group-protective mechanism
US6112263A (en) * 1997-12-15 2000-08-29 Intel Corporation Method for multiple independent processes controlling access to I/O devices in a computer system
US6038651A (en) * 1998-03-23 2000-03-14 International Business Machines Corporation SMP clusters with remote resource managers for distributing work to other clusters while reducing bus traffic to a minimum
US6163829A (en) * 1998-04-17 2000-12-19 Intelect Systems Corporation DSP interrupt control for handling multiple interrupts
US6185631B1 (en) * 1998-10-14 2001-02-06 International Business Machines Corporation Program for transferring execution of certain channel functions to a control unit and having means for combining certain commands and data packets in one sequence

Cited By (40)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160071558A1 (en) * 1999-12-27 2016-03-10 Intel Corporation Memory mapping in a processor having multiple programmable units
US20160019178A1 (en) * 1999-12-27 2016-01-21 Intel Corporation Memory mapping in a processor having multiple programmable units
US20160070664A1 (en) * 1999-12-27 2016-03-10 Intel Corporation Memory mapping in a processor having multiple programmable units
US9824037B2 (en) * 1999-12-27 2017-11-21 Intel Corporation Memory mapping in a processor having multiple programmable units
US9824038B2 (en) * 1999-12-27 2017-11-21 Intel Corporation Memory mapping in a processor having multiple programmable units
US9830284B2 (en) * 1999-12-27 2017-11-28 Intel Corporation Memory mapping in a processor having multiple programmable units
US9830285B2 (en) * 1999-12-27 2017-11-28 Intel Corporation Memory mapping in a processor having multiple programmable units
US20160034420A1 (en) * 1999-12-27 2016-02-04 Intel Corporation Memory mapping in a processor having multiple programmable units
US20040135677A1 (en) * 2000-06-26 2004-07-15 Robert Asam Use of the data stored by a racing car positioning system for supporting computer-based simulation games
US7152026B1 (en) * 2001-12-07 2006-12-19 Ncr Corp. Versioned node configurations for parallel applications
US20060129741A1 (en) * 2004-12-15 2006-06-15 International Business Machines Corporation Method and apparatus for accessing memory in a computer system architecture supporting heterogeneous configurations of memory structures
US7970980B2 (en) * 2004-12-15 2011-06-28 International Business Machines Corporation Method and apparatus for accessing memory in a computer system architecture supporting heterogeneous configurations of memory structures
US20090153897A1 (en) * 2007-12-18 2009-06-18 Blackmore Robert S Method, System and Program Product for Reserving a Global Address Space
US7921261B2 (en) 2007-12-18 2011-04-05 International Business Machines Corporation Reserving a global address space
US20090199209A1 (en) * 2008-02-01 2009-08-06 Arimilli Lakshminarayana B Mechanism for Guaranteeing Delivery of Multi-Packet GSM Message
US8200910B2 (en) 2008-02-01 2012-06-12 International Business Machines Corporation Generating and issuing global shared memory operations via a send FIFO
US8214604B2 (en) 2008-02-01 2012-07-03 International Business Machines Corporation Mechanisms to order global shared memory operations
US8239879B2 (en) 2008-02-01 2012-08-07 International Business Machines Corporation Notification by task of completion of GSM operations at target node
US8255913B2 (en) 2008-02-01 2012-08-28 International Business Machines Corporation Notification to task of completion of GSM operations by initiator node
US8275947B2 (en) 2008-02-01 2012-09-25 International Business Machines Corporation Mechanism to prevent illegal access to task address space by unauthorized tasks
US20090199182A1 (en) * 2008-02-01 2009-08-06 Arimilli Lakshminarayana B Notification by Task of Completion of GSM Operations at Target Node
US20090199194A1 (en) * 2008-02-01 2009-08-06 Arimilli Lakshminarayana B Mechanism to Prevent Illegal Access to Task Address Space by Unauthorized Tasks
US20090198971A1 (en) * 2008-02-01 2009-08-06 Arimilli Lakshminarayana B Heterogeneous Processing Elements
US8893126B2 (en) 2008-02-01 2014-11-18 International Business Machines Corporation Binding a process to a special purpose processing element having characteristics of a processor
US20090199195A1 (en) * 2008-02-01 2009-08-06 Arimilli Lakshminarayana B Generating and Issuing Global Shared Memory Operations Via a Send FIFO
US20090199191A1 (en) * 2008-02-01 2009-08-06 Arimilli Lakshminarayana B Notification to Task of Completion of GSM Operations by Initiator Node
US20090198918A1 (en) * 2008-02-01 2009-08-06 Arimilli Lakshminarayana B Host Fabric Interface (HFI) to Perform Global Shared Memory (GSM) Operations
US20090199200A1 (en) * 2008-02-01 2009-08-06 Arimilli Lakshminarayana B Mechanisms to Order Global Shared Memory Operations
US8146094B2 (en) 2008-02-01 2012-03-27 International Business Machines Corporation Guaranteeing delivery of multi-packet GSM messages
US8484307B2 (en) 2008-02-01 2013-07-09 International Business Machines Corporation Host fabric interface (HFI) to perform global shared memory (GSM) operations
US9342452B2 (en) 2011-10-07 2016-05-17 Hewlett Packard Enterprise Development Lp Mapping processor address ranges to persistent storage
US20140029616A1 (en) * 2012-07-26 2014-01-30 Oracle International Corporation Dynamic node configuration in directory-based symmetric multiprocessing systems
US8848576B2 (en) * 2012-07-26 2014-09-30 Oracle International Corporation Dynamic node configuration in directory-based symmetric multiprocessing systems
US9235581B2 (en) * 2013-10-18 2016-01-12 Hitachi Data Systems Engineering UK Limited Data configuration and migration in a cluster system
US20150143066A1 (en) * 2013-10-18 2015-05-21 Hitachi Data Systems Engineering UK Limited Data configuration and migration in a cluster system
US20150149743A1 (en) * 2013-11-27 2015-05-28 Realtek Semiconductor Corp. Management method of virtual-to-physical address translation system using part of bits of virtual address as index
US9824023B2 (en) * 2013-11-27 2017-11-21 Realtek Semiconductor Corp. Management method of virtual-to-physical address translation system using part of bits of virtual address as index
US9886210B2 (en) 2015-06-09 2018-02-06 Ultrata, Llc Infinite memory fabric hardware implementation with router
WO2016200649A1 (en) * 2015-06-09 2016-12-15 Ultrata Llc Infinite memory fabric streams and apis
US20170153849A1 (en) * 2015-11-26 2017-06-01 Via Alliance Semiconductor Co., Ltd. Methods and apparatuses for generating machine code for driving an execution unit

Also Published As

Publication number Publication date Type
US6449700B2 (en) 2002-09-10 grant
US6308246B1 (en) 2001-10-23 grant
US20030097539A1 (en) 2003-05-22 application
DE69817192D1 (en) 2003-09-18 grant
US6654866B2 (en) 2003-11-25 grant
WO1999012103A3 (en) 1999-06-03 application
DE69819927D1 (en) 2003-12-24 grant
EP1010090A1 (en) 2000-06-21 application
EP1010090B1 (en) 2003-08-13 grant
US6446185B2 (en) 2002-09-03 grant
JP2001515243A (en) 2001-09-18 application
JP2001515244A (en) 2001-09-18 application
US6401174B1 (en) 2002-06-04 grant
US6618799B2 (en) 2003-09-09 grant
WO1999012102A8 (en) 1999-05-06 application
US6240501B1 (en) 2001-05-29 grant
WO1999012103A2 (en) 1999-03-11 application
US20020019921A1 (en) 2002-02-14 application
US6351795B1 (en) 2002-02-26 grant
WO1999012102A1 (en) 1999-03-11 application
US6332165B1 (en) 2001-12-18 grant
US6370585B1 (en) 2002-04-09 grant
EP1019840B1 (en) 2003-11-19 grant
US20020004886A1 (en) 2002-01-10 application
EP1019840A2 (en) 2000-07-19 application
US20010042176A1 (en) 2001-11-15 application

Similar Documents

Publication Publication Date Title
US5247648A (en) Maintaining data coherency between a central cache, an I/O cache and a memory
US6438653B1 (en) Cache memory control circuit including summarized cache tag memory summarizing cache tag information in parallel processor system
US5446863A (en) Cache snoop latency prevention apparatus
US5671391A (en) Coherent copyback protocol for multi-level cache memory systems
US5398325A (en) Methods and apparatus for improving cache consistency using a single copy of a cache tag memory in multiple processor computer systems
US6754782B2 (en) Decentralized global coherency management in a multi-node computer system
US6370622B1 (en) Method and apparatus for curious and column caching
US6912612B2 (en) Shared bypass bus structure
US5893149A (en) Flushing of cache memory in a computer system
US6615322B2 (en) Two-stage request protocol for accessing remote memory data in a NUMA data processing system
US5524235A (en) System for arbitrating access to memory with dynamic priority assignment
US4928225A (en) Coherent cache structures and methods
US6636906B1 (en) Apparatus and method for ensuring forward progress in coherent I/O systems
US6343344B1 (en) System bus directory snooping mechanism for read/castout (RCO) address transaction
US6658522B1 (en) Method to reduce overhead associated with system I/O in a multiprocessor computer system
US6295598B1 (en) Split directory-based cache coherency technique for a multi-processor computer system
US6651143B2 (en) Cache management using a buffer for invalidation requests
US6338119B1 (en) Method and apparatus with page buffer and I/O page kill definition for improved DMA and L1/L2 cache performance
US6421775B1 (en) Interconnected processing nodes configurable as at least one non-uniform memory access (NUMA) data processing system
US5029070A (en) Coherent cache structures and methods
US5710907A (en) Hybrid NUMA COMA caching system and methods for selecting between the caching modes
US6912628B2 (en) N-way set-associative external cache with standard DDR memory devices
US6631448B2 (en) Cache coherence unit for interconnecting multiprocessor nodes having pipelined snoopy protocol
US6353877B1 (en) Performance optimization and system bus duty cycle reduction by I/O bridge partial cache line write
US5893144A (en) Hybrid NUMA COMA caching system and methods for selecting between the caching modes

Legal Events

Date Code Title Description
FPAY Fee payment

Year of fee payment: 4

FPAY Fee payment

Year of fee payment: 8

FPAY Fee payment

Year of fee payment: 12