US20220137816A1 - Native memory semantic remote memory access system - Google Patents
Native memory semantic remote memory access system Download PDFInfo
- Publication number
- US20220137816A1 US20220137816A1 US17/085,133 US202017085133A US2022137816A1 US 20220137816 A1 US20220137816 A1 US 20220137816A1 US 202017085133 A US202017085133 A US 202017085133A US 2022137816 A1 US2022137816 A1 US 2022137816A1
- Authority
- US
- United States
- Prior art keywords
- memory
- subsystem
- computing system
- memory access
- clustered
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
- G06F3/0605—Improving or facilitating administration, e.g. storage management by facilitating the interaction with a user or administrator
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0635—Configuration or reconfiguration of storage systems by changing the path, e.g. traffic rerouting, path reconfiguration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
Definitions
- the present disclosure relates generally to information handling systems, and more particularly to providing clustered memory in information handling systems.
- Information handling systems such as, for example, server devices and/or other computing systems known in the art, may be provided in “clusters” such that applications in any particular server device have “local” access to memory subsystems included in that server device, as well as “remote” access to memory subsystems that are remote to that server device (e.g., in another server device and/or otherwise coupled to that server device via a network).
- the local memory subsystems in any server device is accessible by applications in that server device via the use of conventional memory pointer access techniques/native memory semantics utilized by the processing system (e.g., an x86 processor) in that server device, while the remote memory subsystems require the utilization of remote memory access software (e.g., middleware, libraries, sockets, etc.) by that processing system in order to generate remote memory access communications, which raises some issues.
- processing system e.g., an x86 processor
- remote memory access software e.g., middleware, libraries, sockets, etc.
- remote memory access software on processing systems may perform remote memory access operations over a network (e.g., by generating Remote Direct Memory Access (RDMA) communications via RDMA protocols and/or Ethernet communications via Ethernet protocols that may utilize an Application Programming Interface (API) mechanism with the middleware/libraries/sockets discussed above to provide for the accessing of remote memory subsystems (e.g., by allowing the processing system to call function(s) to read a remote memory address, and the performing of those function(s) to program a network controller in a networking device (e.g., a Network Interface Controller (NIC) device) to perform RDMA and/or Ethernet communications that result in remote memory operations at that remote memory address.).
- a network controller e.g., a Network Interface Controller (NIC) device
- the remote memory subsystem access operations discussed above are not “true” memory reference operations (i.e., memory pointer access techniques/native memory semantics utilized by a processing system) like the local memory subsystem access operations that utilize pointers that point to an address in a local memory subsystem that will be read from or written to.
- the networking devices that couple their server devices to the network are connected to components in that server device via a Peripheral Component Interconnect express (PCIe) bus and “behind” an Input/Output Memory Management Unit (I/O MMU), thus preventing “pure” MMU memory access, and requiring the remote memory access software discussed above.
- PCIe Peripheral Component Interconnect express
- I/O MMU Input/Output Memory Management Unit
- an Information Handling System may include a secondary processing subsystem; and a secondary memory subsystem that is coupled to the secondary processing subsystem and that includes instructions that, when executed by the secondary processing subsystem, cause the secondary processing subsystem to provide a clustered memory management engine that is configured to: receive, from a central processing subsystem, a first processor memory-centric access request associated with a first memory access operation; determine, using memory management hardware included in the secondary processing subsystem, that the first processor memory-centric access request is directed to a first memory subsystem in a first computing system that is coupled to the secondary processing subsystem via a network; generate, using remote memory access hardware included in the secondary processing subsystem, first memory access information for performing the memory access operation at the first memory subsystem; and instruct a networking device coupled to the secondary processing subsystem to utilize the first memory access information to perform the first memory access operation at the first memory subsystem to cause the networking device to transmit at least one first memory access communication that provides for the performance of the first memory access operation at the first memory subsystem.
- a clustered memory management engine that
- FIG. 1 is a schematic view illustrating an embodiment of an Information Handling System (IHS).
- IHS Information Handling System
- FIG. 2 is a schematic view illustrating an embodiment of a networked system.
- FIG. 4 is a schematic view illustrating an embodiment of a System Control Processor (SCP) subsystem that may be included in the computing system of FIG. 3 and that may provide the clustered memory functionality of the present disclosure.
- SCP System Control Processor
- FIG. 5 is a flow chart illustrating an embodiment of a method for providing clustered memory.
- FIG. 6 is a schematic view illustrating an embodiment of a global clustered memory space.
- FIG. 7A is a schematic view illustrating an embodiment of the computing system of FIG. 3 operating during the method of FIG. 5 .
- FIG. 7B is a schematic view illustrating an embodiment of the computing system of FIG. 3 operating during the method of FIG. 5 .
- FIG. 7D is a schematic view illustrating an embodiment of the SCP subsystem of FIG. 4 operating during the method of FIG. 5 .
- FIG. 7E is a schematic view illustrating an embodiment of the SCP subsystem of FIG. 4 operating during the method of FIG. 5 .
- FIG. 7F is a schematic view illustrating an embodiment of the SCP subsystem of FIG. 4 operating during the method of FIG. 5 .
- FIG. 7G is a schematic view illustrating an embodiment of the SCP subsystem of FIG. 4 operating during the method of FIG. 5 .
- FIG. 7H is a schematic view illustrating an embodiment of the SCP subsystem of FIG. 4 operating during the method of FIG. 5 .
- FIG. 7I is a schematic view illustrating an embodiment of the SCP subsystem of FIG. 4 operating during the method of FIG. 5 .
- FIG. 7K is a schematic view illustrating an embodiment of the SCP subsystem of FIG. 4 operating during the method of FIG. 5 .
- FIG. 7L is a schematic view illustrating an embodiment of the SCP subsystem of FIG. 4 operating during the method of FIG. 5 .
- FIG. 7M is a schematic view illustrating an embodiment of the SCP subsystem of FIG. 4 operating during the method of FIG. 5 .
- FIG. 7N is a schematic view illustrating an embodiment of the computing system of FIG. 3 operating during the method of FIG. 5 .
- FIG. 7O is a schematic view illustrating an embodiment of the computing system of FIG. 3 operating during the method of FIG. 5 .
- an information handling system may include any instrumentality or aggregate of instrumentalities operable to compute, calculate, determine, classify, process, transmit, receive, retrieve, originate, switch, store, display, communicate, manifest, detect, record, reproduce, handle, or utilize any form of information, intelligence, or data for business, scientific, control, or other purposes.
- an information handling system may be a personal computer (e.g., desktop or laptop), tablet computer, mobile device (e.g., personal digital assistant (PDA) or smart phone), server (e.g., blade server or rack server), a network storage device, or any other suitable device and may vary in size, shape, performance, functionality, and price.
- the information handling system may include random access memory (RAM), one or more processing resources such as a central processing unit (CPU) or hardware or software control logic, ROM, and/or other types of nonvolatile memory. Additional components of the information handling system may include one or more disk drives, one or more network ports for communicating with external devices as well as various input and output (I/O) devices, such as a keyboard, a mouse, touchscreen and/or a video display. The information handling system may also include one or more buses operable to transmit communications between the various hardware components.
- RAM random access memory
- processing resources such as a central processing unit (CPU) or hardware or software control logic
- ROM read-only memory
- Additional components of the information handling system may include one or more disk drives, one or more network ports for communicating with external devices as well as various input and output (I/O) devices, such as a keyboard, a mouse, touchscreen and/or a video display.
- I/O input and output
- the information handling system may also include one or more buses operable to transmit communications between the various
- IHS 100 includes a processor 102 , which is connected to a bus 104 .
- Bus 104 serves as a connection between processor 102 and other components of IHS 100 .
- An input device 106 is coupled to processor 102 to provide input to processor 102 .
- Examples of input devices may include keyboards, touchscreens, pointing devices such as mouses, trackballs, and trackpads, and/or a variety of other input devices known in the art.
- Programs and data are stored on a mass storage device 108 , which is coupled to processor 102 . Examples of mass storage devices may include hard discs, optical disks, magneto-optical discs, solid-state storage devices, and/or a variety of other mass storage devices known in the art.
- IHS 100 further includes a display 110 , which is coupled to processor 102 by a video controller 112 .
- a system memory 114 is coupled to processor 102 to provide the processor with fast storage to facilitate execution of computer programs by processor 102 .
- Examples of system memory may include random access memory (RAM) devices such as dynamic RAM (DRAM), synchronous DRAM (SDRAM), solid state memory devices, and/or a variety of other memory devices known in the art.
- RAM random access memory
- DRAM dynamic RAM
- SDRAM synchronous DRAM
- solid state memory devices solid state memory devices
- a chassis 116 houses some or all of the components of IHS 100 . It should be understood that other buses and intermediate circuits can be deployed between the components described above and processor 102 to facilitate interconnection between the components and the processor 102 .
- the networked system 200 includes a plurality of computing systems 202 a, 202 b , and up to 202 c.
- the computing system 202 a - 202 c may be provided by the IHS 100 discussed above with reference to FIG. 1 , and/or may include some or all of the components of the IHS 100 , and in specific examples may be provided by server devices.
- computing systems provided in the networked system 200 may include any computing systems that may be configured to operate similarly as the computing systems 202 a - 202 c discussed below.
- each of the computing systems may be coupled to a network 204 that may be provided by a Local Area Network (LAN), the Internet, combinations thereof, and/or any other networks that would be apparent to one of skill in the art in possession of the present disclosure.
- LAN Local Area Network
- a management system 206 is also coupled to the network 204 .
- the management system 206 may be provided by the IHS 100 discussed above with reference to FIG. 1 , and/or may include some or all of the components of the IHS 100 , and in specific examples may be provided by one or more management server devices that may be configured to perform management functionality for the computing systems 202 a - 202 c and/or the SCP subsystems in those computing systems (e.g., the SCP manager for the SCP subsystems discussed in further detail below).
- one or more network-attached devices 208 are also coupled to the network 204 .
- clustered memory system of the present disclosure may be utilized with a variety of components and component configurations, and/or may be provided in a variety of computing system/network configurations, while remaining within the scope of the present disclosure as well.
- a computing system 300 may provide any or all of the computing systems 202 a - 202 c discussed above with reference to FIG. 2 .
- the computing system 300 may be provided by the IHS 100 discussed above with reference to FIG. 1 and/or may include some or all of the components of the IHS 100 , and in specific examples may be provided by a server device.
- a server device While illustrated and discussed as being provided by a server device, one of skill in the art in possession of the present disclosure will recognize that the functionality of the computing system 300 discussed below may be provided by other computing systems that are configured to operate similarly as the computing system 300 discussed below.
- the computing system 300 includes a chassis 302 that houses the components of the computing system 300 , only some of which are illustrated and discussed below.
- the chassis 302 may house a clustered memory management subsystem that, in the embodiments illustrated and discussed below, may be provided by a System Control Processor (SCP) subsystem 304 that is provided according to the teachings of the present disclosure to perform the clustered memory management functionality that is discussed in further detail below.
- SCP System Control Processor
- the SCP subsystem 304 may be conceptualized as an “enhanced” SmartNIC device that may be configured to perform functionality that is not available in conventional SmartNIC devices such as, for example, the platform root-of-trust functionality described by the inventors of the present disclosure in U.S. patent application Ser. No. 17/027,835, attorney docket no. 16356.2212US01, filed on Sep.
- the SCP subsystem 304 may be provided by the IHS 100 discussed above with reference to FIG. 1 and/or may include some or all of the components of the IHS 100 .
- the SCP subsystem 304 may be provided as an SCP card that is configured to connect to a slot on a motherboard in the chassis 302 .
- the SCP subsystem 304 may be integrated into a motherboard in the chassis 302 .
- the SCP subsystem 304 may be a separate/co-motherboard circuit board that is connected to a motherboard in the chassis 302 (e.g., a two-part motherboard having a first portion that enables conventional motherboard functionality, and a second portion that enables the SCP functionality discussed below).
- the SCP subsystem 304 may be provided in the computing system 300 in a variety of manners that will fall within the scope of the preset disclosure.
- the chassis 302 may also house the Baseboard Management Controller (BMC) subsystem 306 that is coupled to the SCP subsystem 304 , and which one of skill in the art in possession of the present disclosure will recognize as being configured to manage an interface between system management software in the computing system 300 and hardware in the computing system 300 , as well as perform other BMC operations that would be apparent to one of skill in the art in possession of the present disclosure.
- BMC Baseboard Management Controller
- the BMC subsystem 310 may be configured to utilize a dedicated management network connection (e.g., illustrated by the dashed line in FIG.
- NCSI Network Communications Services Interface
- the chassis 302 may also house a central processing system 308 that is coupled to the SCP subsystem 304 and the BMC subsystem 306 , and which may include one or more Central Processing Units (CPUs) 308 a (e.g., an x86 host processor provided by the processor 102 discussed above with reference to FIG. 1 ) and/or other processing components that would be apparent to one of skill in the art in possession of the present disclosure, a Memory Management Unit (MMU) 308 b, dedicated memory subsystems, and/or other central processing subsystem components that would be apparent to one of skill in the art in possession of the present disclosure.
- CPUs Central Processing Units
- MMU Memory Management Unit
- the dedicated memory subsystems in the central processing subsystem 308 may include instructions that, when executed by the CPU(s) 308 a in the central processing subsystem 308 , cause the CPU(s) 308 a to provide one or more applications 309 (e.g., an operating system, applications provided via an operating system, and/or other applications known in the art) that are configured to utilize the clustered memory system as discussed below.
- applications 309 e.g., an operating system, applications provided via an operating system, and/or other applications known in the art
- the chassis 302 may also house a plurality of memory subsystems 310 a, 310 b, and up to 310 c, each of which is coupled to the central processing subsystem 308 and any of which may provide one or more of the dedicated processor memory subsystems discussed above.
- any of the memory subsystems 310 a - 310 c may be provided by Storage Class Memory (SCM) devices (e.g., “host” SCM devices utilized by the central processing subsystem 308 provided by a x86 host processing system), Dual Data Rate (DDR) memory devices (e.g., “host” DDR memory devices utilized by the central processing subsystem 308 provided by a x86 host processing system), and/or any other memory subsystems/devices/technology that would be apparent to one of skill in the art in possession of the present disclosure.
- SCM Storage Class Memory
- DDR Dual Data Rate
- computing systems may include a variety of components and/or component configurations for providing conventional computing system functionality, as well as the functionality discussed below, while remaining within the scope of the present disclosure as well.
- the BMC subsystem 306 described above with reference to FIG. 3 may be omitted, and the SCP subsystem 304 may be configured to provide a BMC subsystem that performs the functionality of the BMC subsystem 306 in FIG. 3 .
- an embodiment of an SCP subsystem 400 may provide the SCP subsystem 304 discussed above with reference to FIG. 3 .
- the SCP subsystem 400 may be provided by the IHS 100 discussed above with reference to FIG. 1 and/or may include some or all of the components of the IHS 100 , and in specific examples may be provided as an SCP card, may be integrated into a motherboard, and/or may be provided as a separate/co-motherboard circuit board.
- the functionality of the SCP subsystem 400 discussed below may be provided by other devices that are configured to operate similarly as the SCP subsystem 400 discussed below.
- the SCP subsystem 400 includes a chassis 402 (e.g., a circuit board) that supports the components of the SCP subsystem 400 , only some of which are illustrated below.
- the chassis 402 may support a secondary processing subsystem (which may be distinguished from the central processing subsystem 308 in the computing system 300 discussed herein) such as a networking/management processing subsystem (e.g., an SCP processing subsystem) including one or more networking/management processors (not illustrated, but which may include the processor 102 discussed above with reference to FIG. 1 ), and a secondary memory subsystem such as a networking/management memory subsystem (e.g., an SCP memory subsystem, not illustrated, but which may include the memory 114 discussed above with reference to FIG.
- a networking/management processing subsystem e.g., an SCP processing subsystem
- a secondary memory subsystem such as a networking/management memory subsystem (e.g., an SCP memory subsystem, not illustrated, but which may include the memory 114 discussed above with reference to FIG.
- clustered memory management engine 404 that is configured to perform the functionality of the clustered memory management engines and/or SCP subsystems discussed below.
- the clustered memory management engine 404 may include a remote memory access sub-engine 404 a that may be provided by a hardware subsystem (e.g., a hardware accelerator device such as a high-speed state machine) that may be part of (or coupled to) the secondary processing/memory subsystems, and that is configured (e.g., a dedicated hardware accelerator device) or configurable (e.g., a Field Programmable Gate Array (FPGA) device) to perform the remote memory access operations described in further detail below.
- a hardware subsystem e.g., a hardware accelerator device such as a high-speed state machine
- FPGA Field Programmable Gate Array
- the clustered memory management engine 404 may also include an MMU sub-engine 404 b that may be provided by a hardware subsystem (e.g., a hardware accelerator device) that may be part of (or coupled to) the secondary processing/memory subsystems, and that is configured (e.g., a dedicated hardware accelerator device) or configurable (e.g., a Field Programmable Gate Array (FPGA) device) to perform the MMU operations described in further detail below.
- a hardware subsystem e.g., a hardware accelerator device
- FPGA Field Programmable Gate Array
- the hardware subsystems that provide the remote memory access sub-engine 404 a and the MMU sub-engine 404 b may provide for relative high speed, dedicated remote memory access operations and MMU operations, respectively, relative to conventional clustered memory systems in which such functionality is enabled via software executed by the x86 host processor.
- the chassis 402 may also support a cache device 405 that is coupled to the clustered memory management engine 404 and that is configured to store cached data associated with previous memory access operations, discussed in further detail below.
- the chassis 402 may also support a central processing subsystem coupling 408 that may be connected to the central processing subsystem 308 in the computing system 300 , and that may include the one or more Compute Express Link (CxL) root .mem/.cache Base Address Register (BAR) windows discussed below, and/or other central processing subsystem coupling components that would be apparent to one of skill in the art in possession of the present disclosure.
- CxL Compute Express Link
- BAR Base Address Register
- the chassis 402 may also support a storage system (not illustrated, but which may include the storage 108 discussed above with reference to FIG. 1 , a Static Random Access Memory (SRAM), etc.) that is coupled to the clustered memory management engine 404 (e.g., via a coupling between the storage system and the secondary processing subsystem) and that may include a clustered memory management database 406 that is configured to store any of the information utilized by the clustered memory management engine 404 discussed below.
- the chassis 402 may also support a BMC coupling 410 such as, for example, an Out-Of-Band (OOB) management coupling that is configured to connect to the BMC subsystem 306 discussed above with reference to FIG. 3 .
- a BMC coupling 410 such as, for example, an Out-Of-Band (OOB) management coupling that is configured to connect to the BMC subsystem 306 discussed above with reference to FIG. 3 .
- the networking device 412 may also include wireless communication systems (e.g., BLUETOOTH®, Near Field Communication (NFC) components, WiFi components, etc.), component connections to computing system components (e.g., the central processing subsystem 308 ) in the computing system 300 , and/or any other communication components that would be apparent to one of skill in the art in possession of the present disclosure.
- wireless communication systems e.g., BLUETOOTH®, Near Field Communication (NFC) components, WiFi components, etc.
- component connections to computing system components e.g., the central processing subsystem 308
- computing system components e.g., the central processing subsystem 308
- the chassis 402 may also support one or more memory subsystems 416 that are coupled to the clustered memory management engine 404 via one or more memory port(s) 414 (e.g., CxL “host” ports).
- memory subsystems 416 may be provided by SCM devices (e.g., “SCP” SCM devices), DDR memory devices (e.g., “SCP” DDR memory devices), and/or any other memory subsystems/devices/technology that would be apparent to one of skill in the art in possession of the present disclosure.
- the systems and methods of the present disclosure provide a clustered memory management subsystem that provides a clustered memory hardware abstraction layer and utilizes hardware subsystems to identify processor memory-centric access requests that are directed to remote memory subsystems, and generate remote memory access information for use by a networking device to perform remote memory access operations.
- the clustered memory system of the present disclosure may include a first computing system that is coupled to a second computing system via a network, and that includes a clustered memory management subsystem coupled to a central processing subsystem and a networking device.
- the clustered memory management subsystem receives a processor memory-centric access request associated with a memory access operation from the central processing subsystem, and uses memory management hardware to determine that the processor memory-centric access request is directed to a second memory subsystem in the second computing system.
- the clustered memory management subsystem uses remote memory access hardware to generate memory access information for performing the memory access operation at the second memory subsystem, and instructs the networking device to utilize the memory access information to transmit at least one memory access communication that performs the memory access operation at the second memory subsystem.
- a global memory address space may be presented to the operating system and/or other applications in a computing system, while allowing memory access operations requested for those applications by a central processing subsystem via processor memory-centric access requests to be converted to remote memory access operations via high-speed, dedicated hardware and without the use of remote memory access software (e.g., middleware/libraries/sockets) executed by that central processing subsystem.
- remote memory access software e.g., middleware/libraries/sockets
- the SCP subsystems 304 in each computing system 202 a - 202 c / 300 may establish secure communication channels with each other.
- each of the SCP subsystems 304 in the computing systems 202 a - 202 c / 300 may be configured to perform the secure communication functionality described by the inventors of the present disclosure in U.S. patent application Ser. No. 17/079,737, attorney docket no. 16356.2217US01, filed on Oct. 26, 2020, the disclosure of which is incorporated herein by reference in its entirety.
- the SCP subsystem 304 in the computing system 202 b / 300 may identify the SCP subsystem 304 in the computing system 202 a / 300 , sign a second SCP authentication communication with a second private key, and transmit the second signed SCP authentication communication to the SCP subsystem 304 in the computing system 202 a / 300 , while the SCP subsystem 304 in the computing system 202 a / 300 signs a first SCP authentication communication with a first private key, and transmits the first signed SCP authentication communication to the SCP subsystem 304 in the computing system 202 b / 300 .
- the SCP subsystem 304 in the computing system 202 b / 300 may then authenticate the first SCP authentication communication using a first public key, the SCP subsystem 304 in the computing system 202 a / 300 may authenticate the second SCP authentication communication using a second public key and, in response, the SCP subsystems 304 in the computing systems 202 a / 300 and 202 b / 300 will establish a secure communication channel.
- the SCP subsystem 304 in the computing system 202 b / 300 may then identify the SCP subsystem 304 in the computing system 202 c / 300 , sign a second SCP authentication communication with a second private key, and transmit the second signed SCP authentication communication to the SCP subsystem 304 in the computing system 202 c / 300 , while the SCP subsystem 304 in the computing system 202 c / 300 signs a third SCP authentication communication with a third private key, and transmits the third signed SCP authentication communication to the SCP subsystem 304 in the computing system 202 b / 300 .
- the SCP subsystem 304 in the computing system 202 b / 300 may then authenticate the third SCP authentication communication using a third public key, the SCP subsystem 304 in the computing system 202 c / 300 may authenticate the second SCP authentication communication using a second public key and, in response, the SCP subsystems 304 in the computing systems 202 b / 300 and 202 c / 300 will establish a secure communication channel.
- the SCP subsystem 304 in the computing system 202 b / 300 may then attest to the authentication of the SCP subsystem 304 in the computing system 202 c / 300 to the SCP subsystem 304 in the computing system 202 a / 300 , and attest to the authentication of the SCP subsystem 304 in the computing system 202 a / 300 to the SCP subsystem 304 in the computing system 202 c / 300 , which allows the SCP subsystems 304 in the computing systems 202 a / 300 and 202 c / 300 to establish a secure communication channel without transmitting signed SCP authentication communications.
- any SCP subsystem authenticated by an SCP subsystem may have its authentication attested to other SCP subsystems to quickly and securely grow the secure communication fabric between the SCP subsystems.
- enabling keys may provide for use of the secure communication channels by each of the SCP subsystems 304 in the computing systems 202 a / 300 , 202 b / 300 , and 202 c / 300 to securely exchange communications, and continued performance of the platform root-of-trust functionality referenced above by those SCP subsystems will ensure that the secure communication channels are only maintained with trusted SCP subsystems and/or computing systems.
- secure communication channels may be established between the SCP subsystems 304 in the computing systems 202 a - 202 c / 300 that each operate as trusted systems to perform any of the inter-SCP subsystem communications discussed below in a secure manner.
- the method 500 begins at block 502 where a global memory address map is generated.
- the SCP subsystems 304 / 400 in the computing systems 202 a - 202 c / 300 may power on, reset, reboot, and/or otherwise initialize and, in response, the clustered memory management engine 404 in each of those SCP subsystems 304 / 400 may retrieve its “local” memory subsystem information, clustered memory map information, and/or other clustered memory information known in the art, and broadcast that clustered memory information to the other SCP subsystems 304 / 400 .
- the clustered memory management engine 404 in each of the SCP subsystems 304 / 400 in the computing systems 202 a - 202 c / 300 may retrieve the clustered memory information broadcast by each of the other SCP subsystems 304 / 400 , which allows the clustered memory management engine 404 in each of the SCP subsystems 304 / 400 to generate a global memory address map, configure memory subsystem offsets, configure Base Address Register (BAR) windowing registers, and/or perform any other global memory information that one of skill in the art in possession of the present disclosure will appreciate would provide for the clustered memory functionality discussed below.
- the global memory address map generation may involve the management system 206 (e.g., an SCP manager) while remaining within the scope of the present disclosure as well.
- global memory address map 600 is illustrated that may be generated by a simplified clustered memory system provided according to the teachings of the present disclosure.
- global memory address map 600 is provided for two computing systems, with a first computing system (e.g., the computing system 202 a in the examples below) including the central processing subsystem 600 and the SCP subsystem 602 coupled together and associated with a local address space 604 that is provided by an address space 604 a that is directly connected to the central processing subsystem 600 (e.g., an address space provided by the memory subsystems 310 a - 310 c in the computing system 300 discussed above), and address spaces 604 b and 604 c that are directly connected to the SCP subsystem 602 (e.g., address spaces provided by the memory subsystems 416 in the SCP system 400 discussed above), and a second computing system (e.g., the computing system 202 b in the examples below) including the central processing subsystem 606 and the SCP sub
- FIG. 6 illustrates how both the central processing subsystem 600 and SCP subsystem 602 in the first computing system, and both the central processing subsystem 606 and SCP subsystem 608 in the second computing system, are associated with a global address space 614 that includes the address space 604 a that is directly connected to the central processing subsystem 600 , the address spaces 604 b and 604 c that are directly connected to the SCP subsystem 602 , the address space 610 a that is directly connected to the central processing subsystem 606 , and the address spaces 610 b and 610 c that are directly connected to the SCP subsystem 608 .
- the central processing subsystem 600 and SCP subsystem 602 in the first computing system may view the address spaces 604 a - 604 c as “local” address space (i.e., local to their first computing system) and the address spaces 610 a - 610 c as “remote” address space (i.e., remote from their first computing system), while the central processing subsystem 606 and SCP subsystem 608 in the second computing system may view the address spaces 604 a - 604 c as “remote” address space (i.e., remote from their second computing system) and the address spaces 610 a - 610 c as “local” address space (i.e., local to their second computing system).
- memory subsystems offsets may be configured for the memory subsystem(s) in the first computing system and the second computing system in order to address the use of different/inconsistent addressing information for the memory subsystems in the first and second computing systems (e.g., a remote memory address in a memory subsystem included in the second computing system may be defined as “address 1000” in the first computing system while being defined as “address 2000” in the second computing system, with the memory subsystem offsets utilized to convert a memory access request for “address 1000” by the central processing subsystem 600 in the first computing system to a memory access request for “address 2000” in the second computing system).
- a remote memory address in a memory subsystem included in the second computing system may be defined as “address 1000” in the first computing system while being defined as “address 2000” in the second computing system
- the memory subsystem offsets utilized to convert a memory access request for “address 1000” by the central processing subsystem 600 in the first computing system to a memory access request for
- a BAR may provide the beginning of the address space assigned to a “local” SCP subsystem by its corresponding “local” central processing subsystem (e.g., an x86 host CPU), and all of the memory subsystem for local SCP subsystem, as well as remote memory subsystems, may be accessed by that central processing subsystem generating an access identified by “BAR+an offset” that identifies where the access (e.g., a read or write) is targeted.
- a BAR may provide the beginning of the address space assigned to a “local” SCP subsystem by its corresponding “local” central processing subsystem (e.g., an x86 host CPU), and all of the memory subsystem for local SCP subsystem, as well as remote memory subsystems, may be accessed by that central processing subsystem generating an access identified by “BAR+an offset” that identifies where the access (e.g., a read or write) is targeted.
- configuration operations may be performed at block 502 to provide BAR windowing register functionality discussed above, as well as any other BAR windowing functionality known in the art, while remaining within the scope of the present disclosure.
- BAR windowing register functionality discussed above, as well as any other BAR windowing functionality known in the art, while remaining within the scope of the present disclosure.
- global memory address maps utilized with the clustered memory system of the present disclosure may be more complicated and/or may include different features than described above while remaining within the scope of the present disclosure as well.
- a clustered memory management subsystem receives a processor memory-centric access request from a central processing subsystem.
- the application 309 provided in the computing system 202 a / 300 may perform application memory access request operations 700 that may include generating and transmitting an application request to perform a memory access to the CPU 308 a in the computing system 202 a / 300 , with that application request identifying a memory read operation, a memory write operation, and/or other memory access operations known in the art, along with any other memory access operation information (e.g., data to write as part of the memory write operation, etc.) that would be apparent to one of skill in the art in possession of the present disclosure.
- any other memory access operation information e.g., data to write as part of the memory write operation, etc.
- the SCP subsystem 304 / 400 in the computing system 202 a / 300 may present the global memory address space to the applications 309 (e.g., an operating system and/or other applications) in the computing system 202 a / 300 , which allows the application 309 to identify a memory address from that global memory address space in its application request to perform a memory access.
- the applications 309 e.g., an operating system and/or other applications
- the CPU 308 a in the computing system 202 a / 300 may perform CPU memory access request operations 702 that include generating and transmitting a CPU request to perform a memory access to the MMU 308 b in the computing system 202 a / 300 .
- the CPU request to perform the memory access is a processor memory-centric access request that is generated by the CPU 308 a and that includes a memory subsystem address pointer (e.g., *ptr) that points to a memory address (e.g., from the global memory address space) at which the memory access operation requested by the application (e.g., the memory read operation, memory write operation, and/or other memory access operations) should be performed, and that may include any other memory access operation information (e.g., data to write as part of the memory write operation, etc.) that would be apparent to one of skill in the art in possession of the present disclosure.
- a memory subsystem address pointer e.g., *ptr
- processor memory-centric access requests described herein utilize the native protocol of the central processing subsystem (e.g., a CPU) to perform CPU load operations, and CPU store operations.
- the central processing subsystem e.g., a CPU
- the generation of an Ethernet packet may require a CPU to perform thousands of load/store operations in order to program an Ethernet controller and create an Ethernet descriptor that allows the Ethernet controller to send and/or receive data packet.
- CPUs utilize processor memory-centric access requests (e.g., native load/store instructions) to communicate with Dynamic Random Access Memory (DRAM), and load/store operations an conventionally mapped to native CPU instructions, while Ethernet send/receive operations are not conventionally mapped to native CPU instructions.
- processor memory-centric access requests e.g., native load/store instructions
- DRAM Dynamic Random Access Memory
- load/store operations an conventionally mapped to native CPU instructions
- Ethernet send/receive operations are not conventionally mapped to native CPU instructions.
- the CPU 308 a may utilize other memory pointer access techniques/native memory semantics to provide the CPU request to perform the memory access while remaining within the scope of the present disclosure as well.
- the MMU 308 b in the computing system 202 a / 300 may reference a global memory address map (e.g., the global memory address map 600 discussed above) to determine whether that the CPU request to perform the memory access is directed to a computing system “local” memory subsystem (e.g., using the example of the global memory address map 600 above, the processor memory-centric access request points to a memory address included in the address space 604 a provided by any of the memory subsystems 310 a - 310 c in the computing system 202 a / 300 ), or a computing system “remote” memory subsystem (e.g., using the example of the global memory address map 600 above, the processor memory-centric access request points to a memory address included in the address spaces 604 b or 604 c provided by any of the memory subsystems 416 included in the SCP subsystem 304 in
- the memory access operation 704 specified in the CPU request to perform the memory access may include a memory write operation, and the memory access operations 704 may include writing data included in that CPU request to the memory subsystem 310 a.
- the memory access operation 704 specified in the CPU request to perform the memory access may include a memory read operation, and the memory access operations 704 may include reading data from the memory subsystem 310 a and returning it to the application 309 .
- the memory access operation 704 may include any of a variety of memory access operations while remaining within the scope of the present disclosure as well.
- the MMU 308 b in the computing system 202 a / 300 may perform CPU memory request forwarding operations 706 to forward the CPU request to perform the memory access to the SCP subsystem 304 in the computing system 202 a / 300 .
- the MMU sub-engine 404 b in the clustered memory management engine 404 may perform memory access request receiving operations 708 that include receiving the CPU request to perform the memory access (e.g., the processor memory-centric access request) via the central processing subsystem coupling 408 and from the MMU 308 a in the central processing subsystem 308 in the computing system 202 a / 300 .
- memory access request receiving operations 708 include receiving the CPU request to perform the memory access (e.g., the processor memory-centric access request) via the central processing subsystem coupling 408 and from the MMU 308 a in the central processing subsystem 308 in the computing system 202 a / 300 .
- the CPU request to perform the memory access may be transmitted by the MMU 308 a in the central processing subsystem 308 in the computing system 202 a / 300 via a CxL port on that central processing subsystem 308 , and received by the MMU sub-engine 404 b in clustered memory management engine 404 of the SCP subsystem 304 / 400 in the computing system 202 a / 300 via CxL root .mem/.cache BAR windows provided by the central processing subsystem coupling 408 (e.g., in-bound memory windows associated with global memory address space).
- the central processing subsystem coupling 408 e.g., in-bound memory windows associated with global memory address space.
- the method 500 proceeds to block 508 where the clustered memory management subsystem performs local memory access operations.
- the MMU sub-engine 404 b in the clustered memory management engine 404 that is provided by the hardware subsystem and included in the SCP subsystem 304 / 400 in the computing system 202 a / 300 may determine that the CPU request to perform the memory access is directed to an SCP “local” memory subsystem (e.g., the processor memory-centric access request points to a memory address included in the address spaces 604 c or 604 c provided by the memory subsystem(s) 416 in the SCP subsystem 304 / 400 in the computing system 202 a / 300 ).
- the clustered memory management engine 404 in the SCP subsystem 304 / 400 in the computing system 202 a / 300 may perform the memory access operation 710 specified in that CPU request via the memory port(s) 414 (e.g., CxL host ports) and on the memory subsystem(s) 416 in the SCP subsystem 304 / 400 in the computing system 202 a / 300 .
- the memory port(s) 414 e.g., CxL host ports
- the memory access operation 710 specified in the CPU request to perform the memory access may include a memory write operation, and the memory access operations 710 may include writing data included in that CPU request to the memory subsystem(s) 416 .
- the memory access operation 710 specified in the CPU request to perform the memory access may include a memory read operation, and the memory access operation 710 may include reading data from the memory subsystem(s) 416 and returning it to the application 309 (e.g., via the central processing subsystem coupling 408 in that SCP subsystem 304 / 400 and the central processing subsystem 308 in the computing system 202 a / 300 ).
- the memory access operation 710 may include any of a variety of memory access operations while remaining within the scope of the present disclosure as well.
- the method 500 proceeds to block 510 where the clustered memory management subsystem generates memory access information for performing remote memory access operation.
- the MMU sub-engine 404 b in the clustered memory management engine 404 that is provided by the hardware subsystem and included in the SCP subsystem 304 / 400 in the computing system 202 a / 300 may determine that the CPU request to perform the memory access is directed to an SCP “remote” memory subsystem (e.g., the processor memory-centric access request points to a memory address included in the address spaces 612 a - 612 c provided by the memory subsystems included in the computing systems 202 b or 202 c ).
- the clustered memory management engine 404 in the SCP subsystem 304 / 400 in the computing system 202 a / 300 may determine whether data associated with the memory access operation identified in that CPU request (e.g., a memory read operation) in stored in the cache device 405 and, if so, may perform the memory access operation (e.g., reading the data) using the cache device 405 and without the need to proceed to block 510 .
- data associated with the memory access operation identified in that CPU request e.g., a memory read operation
- the cache device 405 may perform the memory access operation (e.g., reading the data) using the cache device 405 and without the need to proceed to block 510 .
- some remote memory access operations may be accomplished via the “local” cache device and, in the case of memory read operations, may include the clustered memory management engine 404 in the SCP subsystem 304 / 400 in the computing system 202 a / 300 determining whether the latest read data satisfying the remote memory access operations is located in the cache device 405 and, if so, transmitting that read data via the central processing subsystem coupling 408 in the SCP subsystem 304 / 400 in the computing system 202 a / 300 and to the application 309 via the central processing subsystem 308 in the computing system 202 a / 300 .
- SCP subsystem 304 / 400 in the computing system 202 a / 300 may operate to manage cache coherency in some embodiments, while in other embodiments cache coherency management may be performed using application-level software coherency mapping techniques that would be apparent to one of skill in the art in possession of the present disclosure.
- the MMU sub-engine 404 b in the clustered memory management engine 404 that is provided by the hardware subsystem and included in the SCP subsystem 304 / 400 in the computing system 202 a / 300 may perform memory request transmission operations 712 to transmit the CPU request to perform the memory access (e.g., the processor memory-centric access request) to the remote memory access sub-engine 404 a in the clustered memory management engine 404 that is provided by the hardware subsystem (e.g., a high speed state machine) included in the SCP subsystem 304 / 400 in the computing system 202 a / 300 .
- the hardware subsystem e.g., a high speed state machine
- the remote memory access sub-engine 404 a in the clustered memory management engine 404 that is provided by the hardware subsystem and included in the SCP subsystem 304 / 400 in the computing system 202 a / 300 may perform remote memory access generation operations 714 to generate memory access information for performing remote memory access operation, and store that memory access information in the clustered memory management database 406 (e.g., provided by SRAM device(s)).
- the clustered memory management database 406 e.g., provided by SRAM device(s)
- the remote memory access generation operations 704 may include the use of any information included in the CPU request to perform the memory access (e.g., the processor memory-centric access request) in order to generate an Ethernet descriptor for the controller 412 a (e.g., an Ethernet controller) in the networking device 412 that identifies memory address information, memory access operations, and/or any other remote memory access information that one of skill in the art in possession of the present disclosure would recognize as allowing for the remote memory access operations discussed below.
- any information included in the CPU request to perform the memory access e.g., the processor memory-centric access request
- an Ethernet descriptor for the controller 412 a e.g., an Ethernet controller
- the remote memory access generation operations 704 may include the utilization of the global memory address map and the CPU request in order to generate RDMA descriptors and/or any of a variety of RDMA information and/or instructions that allow for RDMA communications that enable the remote memory access operations discussed below.
- memory subsystems offsets may be utilized in the generation of the memory access information in order to address the use of different/inconsistent addressing information for the memory subsystems in the computing systems 202 b or 202 c by the computing system 202 a.
- the generation of RDMA information for the performance of RDMA communications in the present disclosure may be performed by the remote memory access sub-engine 404 a provided by the hardware subsystem, and without the conventional middleware, libraries, sockets, and/or other software required by the x86 host processor in conventional clustered memory systems.
- the remote memory access operations discussed below may be enabled using Ethernet communications according to Ethernet protocols and/or other protocols while remaining within the scope of the present disclosure as well.
- the method 500 then proceeds to block 512 where the clustered memory management subsystem instructs a networking device to utilize the memory access information to perform the remote memory access operation.
- the remote memory access sub-engine 404 a in the clustered memory management engine 404 that is provided by the hardware subsystem and included in the SCP subsystem 304 / 400 in the computing system 202 a / 300 may perform memory access information utilization instruction operations 716 that include transmitting an instruction to the controller 412 a in the networking device 412 to utilize the memory access information in the clustered memory management database 406 to perform a remote memory access operation.
- the method 500 then proceeds to block 514 where the networking device retrieves the memory access information and performs the remote memory access operation.
- the controller 412 a e.g., an Ethernet controller
- the networking device 412 in the SCP subsystem 304 / 400 in the computing system 202 a / 300 may perform remote memory access information access operations 718 to access the remote memory access information in the clustered memory management database 406 and use that remote memory access information to perform remote memory access operations.
- the controller 412 a in the networking device 412 in the SCP subsystem 304 / 400 in the computing system 202 a / 300 may perform remote memory access operations 720 that include transmitting remote memory access communications (e.g., the RDMA or Ethernet communications discussed above) via the network 204 to the computing system 202 b / 300 .
- remote memory access communications e.g., the RDMA or Ethernet communications discussed above
- the networking device 412 in the SCP subsystem 304 / 400 in the computing system 202 b / 300 may receive those remote memory access communications via the network 204 and, in response, perform remote memory access communication transmission operations 722 that include transmitting the remote memory access communications to the clustered memory management engine 404 in the SCP subsystem 304 / 400 in the computing system 202 b / 300 .
- the clustered memory management engine 404 in the SCP subsystem 304 / 400 in the computing system 202 b / 300 may receive the remote memory access communications and, in response, determine whether that remote memory access communication is directed to an SCP “local” memory subsystem (e.g., using the example of the global memory address map 600 discussed above, the remote memory access communication identifies a memory address included in the address spaces 604 b or 604 c provided by the memory subsystem(s) 416 in the SCP subsystem 304 / 400 in the computing system 202 b / 300 ), or a computing system “local ” memory subsystem (e.g., using the example of the global memory address map 600 discussed above, the remote memory access communication identifies a memory address included in the address space 604 a provided by the memory subsystems 310 a - 310 c included in the computing system 202 / 300 ).
- SCP “local” memory subsystem e.g., using the example of the global memory address map 600 discussed above, the remote
- the clustered memory management engine 404 in the SCP subsystem 304 / 400 in the computing system 202 b / 300 may perform memory access operations 724 specified in those remote memory access communications via the memory port(s) 414 (e.g., CxL host ports) and on the memory subsystem(s) 416 in the SCP subsystem 304 / 400 in the computing system 202 b / 300 .
- the memory port(s) 414 e.g., CxL host ports
- the memory access operation 724 specified in the remote memory access communication may include a memory write operation, and the memory access operations 724 may include writing data included in that remote memory access communication to the memory subsystem(s) 416 in the computing system 202 b / 300 .
- the memory access operation 724 specified in the remote memory access communication may include a memory read operation, and the memory access operation 724 may include reading data from the memory subsystem(s) 416 in the SCP subsystem 304 / 400 in the computing system 202 b / 300 and returning it to the application 309 in the computing system 202 a / 300 (e.g., via the networking device 412 in the SCP subsystem 304 / 400 in the computing system 202 b / 300 and though the network 204 to the SCP subsystem 304 / 400 in the computing system 202 a / 300 , via the networking device 412 , the clustered memory management engine 404 , and the central processing subsystem coupling 408 in the SCP subsystem 304 / 400 in the computing system 202 a / 300 , and via the central processing subsystem 308 in the computing system 202 a / 300 ).
- the memory access operation 724 may include a memory read operation, and the memory access operation 724 may include reading data from the memory subsystem
- the clustered memory management engine 404 in the SCP subsystem 304 / 400 in the computing system 202 b / 300 may perform memory access operations 726 specified in those remote memory access communications via the central processing subsystem coupling 408 in the SCP subsystem 304 / 400 in the computing system 202 b / 300 and the central processing subsystem 308 in the computing system 202 b / 300 , and on the memory subsystem 310 b in the computing system 202 b / 300 .
- the memory access operation 726 specified in the remote memory access communication may include a memory write operation, and the memory access operations 726 may include writing data included in that remote memory access communication to the memory subsystem 310 b in the computing system 202 b / 300 .
- the memory access operation 726 specified in the remote memory access communication may include a memory read operation, and the memory access operation 726 may include reading data from the memory subsystem 310 b in the computing system 202 b / 300 and returning it to the application 309 in the computing system 202 a / 300 (e.g., the SCP subsystem 304 / 400 in the computing system 202 b / 300 and though the network 204 to the SCP subsystem 304 / 400 in the computing system 202 a / 300 , via the networking device 412 , the clustered memory management engine 404 , and the central processing subsystem coupling 408 in the SCP subsystem 304 / 400 in the computing system 202 a / 300 , and via the central processing subsystem 308 in the computing system 202 a / 300 ).
- the memory access operation 726 may include any of a variety of memory access operations while remaining within
- the method 500 then proceeds to block 516 where the clustered memory management subsystem determines a memory access operation completion status and reports it to the central processing subsystem.
- the clustered memory management engine 404 in the SCP subsystem 304 / 400 in the computing system 202 b / 300 may report a completion status (e.g., a successful/completed status, an unsuccessful/incomplete status, etc.) of its memory access operation to the clustered memory management engine 404 in the SCP subsystem 304 / 400 in the computing system 202 a / 300 , and the clustered memory management engine 404 in the SCP subsystem 304 / 400 in the computing system 202 a / 300 may report that completion status to the central processing subsystem 308 in the computing system 202 a / 300 .
- a completion status e.g., a successful/completed status, an unsuccessful/incomplete status, etc.
- completion status reporting operations may include the SCP subsystem 304 in the computing system 202 a / 300 receive the completion status for the remote memory access operations from the SCP subsystem 304 in the computing system 202 b / 300 and transmitting that completion status to the central processing subsystem 308 in the computing system 202 a / 300 , which may then transmit that completion status to the application 309 provided in the computing system 202 a / 300 .
- the application 308 may be informed of the completion status (e.g., a successful/completed status, an unsuccessful/incomplete status, etc.) of its memory access request.
- the clustered memory system of the present disclosure may include a first server device that is coupled to a second server device via a network, and that includes a SCP subsystem coupled to a central processing subsystem and a networking device.
- the SCP subsystem receives a processor memory-centric access request associated with a memory access operation from the central processing subsystem, and uses memory management hardware to determine that the processor memory-centric access request is directed to a second memory subsystem in the second server device.
- the SCP subsystem uses remote memory access hardware to generate memory access information for performing the memory access operation at the second memory subsystem, and instructs the networking device to utilize the memory access information to transmit at least one memory access communication that performs the memory access operation at the second memory subsystem.
- a global memory address space may be presented to the operating system and/or other applications in a server device, while allowing memory access operations requested for those applications by a central processing subsystem via processor memory-centric access requests to be converted to remote memory access operations via high speed, dedicated hardware and without the use of remote memory access software (e.g., middleware/libraries/sockets) executed by that central processing subsystem.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Storage Device Security (AREA)
Abstract
Description
- The present disclosure relates generally to information handling systems, and more particularly to providing clustered memory in information handling systems.
- As the value and use of information continues to increase, individuals and businesses seek additional ways to process and store information. One option available to users is information handling systems. An information handling system generally processes, compiles, stores, and/or communicates information or data for business, personal, or other purposes thereby allowing users to take advantage of the value of the information. Because technology and information handling needs and requirements vary between different users or applications, information handling systems may also vary regarding what information is handled, how the information is handled, how much information is processed, stored, or communicated, and how quickly and efficiently the information may be processed, stored, or communicated. The variations in information handling systems allow for information handling systems to be general or configured for a specific user or specific use such as financial transaction processing, airline reservations, enterprise data storage, or global communications. In addition, information handling systems may include a variety of hardware and software components that may be configured to process, store, and communicate information and may include one or more computer systems, data storage systems, and networking systems.
- Information handling systems such as, for example, server devices and/or other computing systems known in the art, may be provided in “clusters” such that applications in any particular server device have “local” access to memory subsystems included in that server device, as well as “remote” access to memory subsystems that are remote to that server device (e.g., in another server device and/or otherwise coupled to that server device via a network). In conventional clustered memory systems, the local memory subsystems in any server device is accessible by applications in that server device via the use of conventional memory pointer access techniques/native memory semantics utilized by the processing system (e.g., an x86 processor) in that server device, while the remote memory subsystems require the utilization of remote memory access software (e.g., middleware, libraries, sockets, etc.) by that processing system in order to generate remote memory access communications, which raises some issues. For example, remote memory access software on processing systems may perform remote memory access operations over a network (e.g., by generating Remote Direct Memory Access (RDMA) communications via RDMA protocols and/or Ethernet communications via Ethernet protocols that may utilize an Application Programming Interface (API) mechanism with the middleware/libraries/sockets discussed above to provide for the accessing of remote memory subsystems (e.g., by allowing the processing system to call function(s) to read a remote memory address, and the performing of those function(s) to program a network controller in a networking device (e.g., a Network Interface Controller (NIC) device) to perform RDMA and/or Ethernet communications that result in remote memory operations at that remote memory address.).
- As will be appreciated by one of skill in the art, the remote memory subsystem access operations discussed above are not “true” memory reference operations (i.e., memory pointer access techniques/native memory semantics utilized by a processing system) like the local memory subsystem access operations that utilize pointers that point to an address in a local memory subsystem that will be read from or written to. Furthermore, the networking devices that couple their server devices to the network are connected to components in that server device via a Peripheral Component Interconnect express (PCIe) bus and “behind” an Input/Output Memory Management Unit (I/O MMU), thus preventing “pure” MMU memory access, and requiring the remote memory access software discussed above. As will be appreciated by one of skill in the art in possession of the present disclosure, it would be relatively easier for an application programmer to configure an application to utilize a shared memory pool of local and remote memory subsystems using memory pointer access techniques/native memory semantics, rather than having to program or integrate with the remote memory subsystem software discussed above to generate remote memory access communications from function calls.
- Accordingly, it would be desirable to provide a clustered memory system that addresses the issues discussed above.
- According to one embodiment, an Information Handling System (IHS) may include a secondary processing subsystem; and a secondary memory subsystem that is coupled to the secondary processing subsystem and that includes instructions that, when executed by the secondary processing subsystem, cause the secondary processing subsystem to provide a clustered memory management engine that is configured to: receive, from a central processing subsystem, a first processor memory-centric access request associated with a first memory access operation; determine, using memory management hardware included in the secondary processing subsystem, that the first processor memory-centric access request is directed to a first memory subsystem in a first computing system that is coupled to the secondary processing subsystem via a network; generate, using remote memory access hardware included in the secondary processing subsystem, first memory access information for performing the memory access operation at the first memory subsystem; and instruct a networking device coupled to the secondary processing subsystem to utilize the first memory access information to perform the first memory access operation at the first memory subsystem to cause the networking device to transmit at least one first memory access communication that provides for the performance of the first memory access operation at the first memory subsystem.
-
FIG. 1 is a schematic view illustrating an embodiment of an Information Handling System (IHS). -
FIG. 2 is a schematic view illustrating an embodiment of a networked system. -
FIG. 3 is a schematic view illustrating an embodiment of a computing system that may be included in the networked system ofFIG. 2 and that may utilize the clustered memory functionality of the present disclosure. -
FIG. 4 is a schematic view illustrating an embodiment of a System Control Processor (SCP) subsystem that may be included in the computing system ofFIG. 3 and that may provide the clustered memory functionality of the present disclosure. -
FIG. 5 is a flow chart illustrating an embodiment of a method for providing clustered memory. -
FIG. 6 is a schematic view illustrating an embodiment of a global clustered memory space. -
FIG. 7A is a schematic view illustrating an embodiment of the computing system ofFIG. 3 operating during the method ofFIG. 5 . -
FIG. 7B is a schematic view illustrating an embodiment of the computing system ofFIG. 3 operating during the method ofFIG. 5 . -
FIG. 7C is a schematic view illustrating an embodiment of the computing system ofFIG. 3 operating during the method ofFIG. 5 . -
FIG. 7D is a schematic view illustrating an embodiment of the SCP subsystem ofFIG. 4 operating during the method ofFIG. 5 . -
FIG. 7E is a schematic view illustrating an embodiment of the SCP subsystem ofFIG. 4 operating during the method ofFIG. 5 . -
FIG. 7F is a schematic view illustrating an embodiment of the SCP subsystem ofFIG. 4 operating during the method ofFIG. 5 . -
FIG. 7G is a schematic view illustrating an embodiment of the SCP subsystem ofFIG. 4 operating during the method ofFIG. 5 . -
FIG. 7H is a schematic view illustrating an embodiment of the SCP subsystem ofFIG. 4 operating during the method ofFIG. 5 . -
FIG. 7I is a schematic view illustrating an embodiment of the SCP subsystem ofFIG. 4 operating during the method ofFIG. 5 . -
FIG. 7J is a schematic view illustrating an embodiment of the networked system ofFIG. 2 operating during the method ofFIG. 5 . -
FIG. 7K is a schematic view illustrating an embodiment of the SCP subsystem ofFIG. 4 operating during the method ofFIG. 5 . -
FIG. 7L is a schematic view illustrating an embodiment of the SCP subsystem ofFIG. 4 operating during the method ofFIG. 5 . -
FIG. 7M is a schematic view illustrating an embodiment of the SCP subsystem ofFIG. 4 operating during the method ofFIG. 5 . -
FIG. 7N is a schematic view illustrating an embodiment of the computing system ofFIG. 3 operating during the method ofFIG. 5 . -
FIG. 7O is a schematic view illustrating an embodiment of the computing system ofFIG. 3 operating during the method ofFIG. 5 . - For purposes of this disclosure, an information handling system may include any instrumentality or aggregate of instrumentalities operable to compute, calculate, determine, classify, process, transmit, receive, retrieve, originate, switch, store, display, communicate, manifest, detect, record, reproduce, handle, or utilize any form of information, intelligence, or data for business, scientific, control, or other purposes. For example, an information handling system may be a personal computer (e.g., desktop or laptop), tablet computer, mobile device (e.g., personal digital assistant (PDA) or smart phone), server (e.g., blade server or rack server), a network storage device, or any other suitable device and may vary in size, shape, performance, functionality, and price. The information handling system may include random access memory (RAM), one or more processing resources such as a central processing unit (CPU) or hardware or software control logic, ROM, and/or other types of nonvolatile memory. Additional components of the information handling system may include one or more disk drives, one or more network ports for communicating with external devices as well as various input and output (I/O) devices, such as a keyboard, a mouse, touchscreen and/or a video display. The information handling system may also include one or more buses operable to transmit communications between the various hardware components.
- In one embodiment, IHS 100,
FIG. 1 , includes aprocessor 102, which is connected to abus 104.Bus 104 serves as a connection betweenprocessor 102 and other components of IHS 100. Aninput device 106 is coupled toprocessor 102 to provide input toprocessor 102. Examples of input devices may include keyboards, touchscreens, pointing devices such as mouses, trackballs, and trackpads, and/or a variety of other input devices known in the art. Programs and data are stored on amass storage device 108, which is coupled toprocessor 102. Examples of mass storage devices may include hard discs, optical disks, magneto-optical discs, solid-state storage devices, and/or a variety of other mass storage devices known in the art.IHS 100 further includes adisplay 110, which is coupled toprocessor 102 by avideo controller 112. Asystem memory 114 is coupled toprocessor 102 to provide the processor with fast storage to facilitate execution of computer programs byprocessor 102. Examples of system memory may include random access memory (RAM) devices such as dynamic RAM (DRAM), synchronous DRAM (SDRAM), solid state memory devices, and/or a variety of other memory devices known in the art. In an embodiment, achassis 116 houses some or all of the components ofIHS 100. It should be understood that other buses and intermediate circuits can be deployed between the components described above andprocessor 102 to facilitate interconnection between the components and theprocessor 102. - Referring now to
FIG. 2 , an embodiment of anetworked system 200 is illustrated in which the clustered memory system of the present disclosure may be utilized. In the illustrated embodiment, thenetworked system 200 includes a plurality ofcomputing systems IHS 100 discussed above with reference toFIG. 1 , and/or may include some or all of the components of theIHS 100, and in specific examples may be provided by server devices. However, while discussed as being provided by server devices, one of skill in the art in possession of the present disclosure will recognize that computing systems provided in thenetworked system 200 may include any computing systems that may be configured to operate similarly as the computing systems 202 a-202 c discussed below. In the illustrated embodiment, each of the computing systems may be coupled to anetwork 204 that may be provided by a Local Area Network (LAN), the Internet, combinations thereof, and/or any other networks that would be apparent to one of skill in the art in possession of the present disclosure. - In the illustrated embodiment, a
management system 206 is also coupled to thenetwork 204. In an embodiment, themanagement system 206 may be provided by theIHS 100 discussed above with reference toFIG. 1 , and/or may include some or all of the components of theIHS 100, and in specific examples may be provided by one or more management server devices that may be configured to perform management functionality for the computing systems 202 a-202 c and/or the SCP subsystems in those computing systems (e.g., the SCP manager for the SCP subsystems discussed in further detail below). In the illustrated embodiment, one or more network-attacheddevices 208 are also coupled to thenetwork 204. In an embodiment, the network-attached device(s) 208 may be provided by a variety of different network-attached devices that are accessible to the computing systems 202 a-202 c via thenetwork 204, and in specific examples may be provided by one or more Non-Volatile Memory express (NVMe) storage devices that may be configured to provide a network-attached storage system for any or all of the computing systems 202 a-202 c. However, while a specificnetworked system 200 has been illustrated and described, one of skill in the art in possession of the present disclosure will recognize that the clustered memory system of the present disclosure may be utilized with a variety of components and component configurations, and/or may be provided in a variety of computing system/network configurations, while remaining within the scope of the present disclosure as well. - Referring now to
FIG. 3 , an embodiment of acomputing system 300 is illustrated that may provide any or all of the computing systems 202 a-202 c discussed above with reference toFIG. 2 . As such, thecomputing system 300 may be provided by theIHS 100 discussed above with reference toFIG. 1 and/or may include some or all of the components of theIHS 100, and in specific examples may be provided by a server device. However, while illustrated and discussed as being provided by a server device, one of skill in the art in possession of the present disclosure will recognize that the functionality of thecomputing system 300 discussed below may be provided by other computing systems that are configured to operate similarly as thecomputing system 300 discussed below. In the illustrated embodiment, thecomputing system 300 includes achassis 302 that houses the components of thecomputing system 300, only some of which are illustrated and discussed below. - For example, the
chassis 302 may house a clustered memory management subsystem that, in the embodiments illustrated and discussed below, may be provided by a System Control Processor (SCP)subsystem 304 that is provided according to the teachings of the present disclosure to perform the clustered memory management functionality that is discussed in further detail below. In some examples, theSCP subsystem 304 may be conceptualized as an “enhanced” SmartNIC device that may be configured to perform functionality that is not available in conventional SmartNIC devices such as, for example, the platform root-of-trust functionality described by the inventors of the present disclosure in U.S. patent application Ser. No. 17/027,835, attorney docket no. 16356.2212US01, filed on Sep. 22, 2020, and the secure communication functionality described by the inventors of the present disclosure in U.S. patent application Ser. No. 17/079,737, attorney docket no. 16356.2217US01, filed on Oct. 26, 2020, the disclosures of which is incorporated herein by reference in their entirety. However, while the clustered memory management subsystem that enables the clustered memory management functionality according to the teachings of the present disclosure is illustrated and described as an enhanced SmartNIC device provided by an SCP subsystem, one of skill in the art in possession of the present disclosure will appreciated that the clustered memory management functionality described herein may be enabled on other devices while remaining within the scope of the present disclosure as well. - In an embodiment, the
SCP subsystem 304 may be provided by theIHS 100 discussed above with reference toFIG. 1 and/or may include some or all of the components of theIHS 100. In specific examples, theSCP subsystem 304 may be provided as an SCP card that is configured to connect to a slot on a motherboard in thechassis 302. In other examples, theSCP subsystem 304 may be integrated into a motherboard in thechassis 302. In yet other examples theSCP subsystem 304 may be a separate/co-motherboard circuit board that is connected to a motherboard in the chassis 302 (e.g., a two-part motherboard having a first portion that enables conventional motherboard functionality, and a second portion that enables the SCP functionality discussed below). However, while a few specific examples are provided, one of skill in the art in possession of the present disclosure will appreciate that theSCP subsystem 304 may be provided in thecomputing system 300 in a variety of manners that will fall within the scope of the preset disclosure. - In an embodiment, the
chassis 302 may also house the Baseboard Management Controller (BMC)subsystem 306 that is coupled to theSCP subsystem 304, and which one of skill in the art in possession of the present disclosure will recognize as being configured to manage an interface between system management software in thecomputing system 300 and hardware in thecomputing system 300, as well as perform other BMC operations that would be apparent to one of skill in the art in possession of the present disclosure. As such, the BMC subsystem 310 may be configured to utilize a dedicated management network connection (e.g., illustrated by the dashed line inFIG. 3 ), or may be configured to utilize a network connection included in the SCP subsystem 304 (e.g., via a Network Communications Services Interface (NCSI) that allows the use of a NIC port on a NIC device in the SCP subsystem 304). - The
chassis 302 may also house acentral processing system 308 that is coupled to theSCP subsystem 304 and theBMC subsystem 306, and which may include one or more Central Processing Units (CPUs) 308 a (e.g., an x86 host processor provided by theprocessor 102 discussed above with reference toFIG. 1 ) and/or other processing components that would be apparent to one of skill in the art in possession of the present disclosure, a Memory Management Unit (MMU) 308 b, dedicated memory subsystems, and/or other central processing subsystem components that would be apparent to one of skill in the art in possession of the present disclosure. As discussed below, the dedicated memory subsystems in thecentral processing subsystem 308 may include instructions that, when executed by the CPU(s) 308 a in thecentral processing subsystem 308, cause the CPU(s) 308 a to provide one or more applications 309 (e.g., an operating system, applications provided via an operating system, and/or other applications known in the art) that are configured to utilize the clustered memory system as discussed below. - The
chassis 302 may also house a plurality ofmemory subsystems central processing subsystem 308 and any of which may provide one or more of the dedicated processor memory subsystems discussed above. For example, any of the memory subsystems 310 a-310 c may be provided by Storage Class Memory (SCM) devices (e.g., “host” SCM devices utilized by thecentral processing subsystem 308 provided by a x86 host processing system), Dual Data Rate (DDR) memory devices (e.g., “host” DDR memory devices utilized by thecentral processing subsystem 308 provided by a x86 host processing system), and/or any other memory subsystems/devices/technology that would be apparent to one of skill in the art in possession of the present disclosure. However, while aspecific computing system 300 has been illustrated, one of skill in the art in possession of the present disclosure will recognize that computing systems (or other devices operating according to the teachings of the present disclosure in a manner similar to that described below for the computing system 300) may include a variety of components and/or component configurations for providing conventional computing system functionality, as well as the functionality discussed below, while remaining within the scope of the present disclosure as well. For example, in some embodiments, theBMC subsystem 306 described above with reference toFIG. 3 may be omitted, and theSCP subsystem 304 may be configured to provide a BMC subsystem that performs the functionality of theBMC subsystem 306 inFIG. 3 . - With reference to
FIG. 4 , an embodiment of anSCP subsystem 400 is illustrated that may provide theSCP subsystem 304 discussed above with reference toFIG. 3 . As such, theSCP subsystem 400 may be provided by theIHS 100 discussed above with reference toFIG. 1 and/or may include some or all of the components of theIHS 100, and in specific examples may be provided as an SCP card, may be integrated into a motherboard, and/or may be provided as a separate/co-motherboard circuit board. However, while illustrated and discussed as being provided in different manners in acomputing system 300, one of skill in the art in possession of the present disclosure will recognize that the functionality of theSCP subsystem 400 discussed below may be provided by other devices that are configured to operate similarly as theSCP subsystem 400 discussed below. - In the illustrated embodiment, the
SCP subsystem 400 includes a chassis 402 (e.g., a circuit board) that supports the components of theSCP subsystem 400, only some of which are illustrated below. For example, thechassis 402 may support a secondary processing subsystem (which may be distinguished from thecentral processing subsystem 308 in thecomputing system 300 discussed herein) such as a networking/management processing subsystem (e.g., an SCP processing subsystem) including one or more networking/management processors (not illustrated, but which may include theprocessor 102 discussed above with reference toFIG. 1 ), and a secondary memory subsystem such as a networking/management memory subsystem (e.g., an SCP memory subsystem, not illustrated, but which may include thememory 114 discussed above with reference toFIG. 1 ) that is coupled to the secondary processing system and that includes instructions that, when executed by the secondary processing system, cause the secondary processing system to provide a clusteredmemory management engine 404 that is configured to perform the functionality of the clustered memory management engines and/or SCP subsystems discussed below. - In the illustrated embodiments, the clustered
memory management engine 404 may include a remote memory access sub-engine 404 a that may be provided by a hardware subsystem (e.g., a hardware accelerator device such as a high-speed state machine) that may be part of (or coupled to) the secondary processing/memory subsystems, and that is configured (e.g., a dedicated hardware accelerator device) or configurable (e.g., a Field Programmable Gate Array (FPGA) device) to perform the remote memory access operations described in further detail below. Furthermore, the clusteredmemory management engine 404 may also include an MMU sub-engine 404 b that may be provided by a hardware subsystem (e.g., a hardware accelerator device) that may be part of (or coupled to) the secondary processing/memory subsystems, and that is configured (e.g., a dedicated hardware accelerator device) or configurable (e.g., a Field Programmable Gate Array (FPGA) device) to perform the MMU operations described in further detail below. As will be appreciated by one of skill in the art in possession of the present disclosure, the hardware subsystems that provide the remote memory access sub-engine 404 a and the MMU sub-engine 404 b may provide for relative high speed, dedicated remote memory access operations and MMU operations, respectively, relative to conventional clustered memory systems in which such functionality is enabled via software executed by the x86 host processor. - In the illustrated embodiment, the
chassis 402 may also support acache device 405 that is coupled to the clusteredmemory management engine 404 and that is configured to store cached data associated with previous memory access operations, discussed in further detail below. Thechassis 402 may also support a central processing subsystem coupling 408 that may be connected to thecentral processing subsystem 308 in thecomputing system 300, and that may include the one or more Compute Express Link (CxL) root .mem/.cache Base Address Register (BAR) windows discussed below, and/or other central processing subsystem coupling components that would be apparent to one of skill in the art in possession of the present disclosure. - The
chassis 402 may also support a storage system (not illustrated, but which may include thestorage 108 discussed above with reference toFIG. 1 , a Static Random Access Memory (SRAM), etc.) that is coupled to the clustered memory management engine 404 (e.g., via a coupling between the storage system and the secondary processing subsystem) and that may include a clusteredmemory management database 406 that is configured to store any of the information utilized by the clusteredmemory management engine 404 discussed below. Thechassis 402 may also support aBMC coupling 410 such as, for example, an Out-Of-Band (OOB) management coupling that is configured to connect to theBMC subsystem 306 discussed above with reference toFIG. 3 . Thechassis 402 may also support a networking device 412 (e.g., a Network Interface Controller (NIC) device) that is coupled to the clustered memory management engine 404 (e.g., via a coupling between thecommunication system 412 and the secondary processing subsystem), that includes acontroller 412 a (e.g., an Ethernet controller), and that may connect theSCP subsystem 304/400 to thenetwork 204 discussed above with reference toFIG. 2 . As will be appreciated by one of skill in the art in possession of the present disclosure, thenetworking device 412 may also include wireless communication systems (e.g., BLUETOOTH®, Near Field Communication (NFC) components, WiFi components, etc.), component connections to computing system components (e.g., the central processing subsystem 308) in thecomputing system 300, and/or any other communication components that would be apparent to one of skill in the art in possession of the present disclosure. - The
chassis 402 may also support one ormore memory subsystems 416 that are coupled to the clusteredmemory management engine 404 via one or more memory port(s) 414 (e.g., CxL “host” ports). For example, any of thememory subsystems 416 may be provided by SCM devices (e.g., “SCP” SCM devices), DDR memory devices (e.g., “SCP” DDR memory devices), and/or any other memory subsystems/devices/technology that would be apparent to one of skill in the art in possession of the present disclosure. However, while aspecific SCP subsystem 400 has been illustrated and described, one of skill in the art in possession of the present disclosure will recognize that SCP subsystems (or other devices operating according to the teachings of the present disclosure in a manner similar to that described below for the SCP subsystem 400) may include a variety of components and/or component configurations for providing the functionality discussed below while remaining within the scope of the present disclosure as well. - Referring now to
FIG. 5 , an embodiment of amethod 500 for providing clustered memory is illustrated. As discussed below, the systems and methods of the present disclosure provide a clustered memory management subsystem that provides a clustered memory hardware abstraction layer and utilizes hardware subsystems to identify processor memory-centric access requests that are directed to remote memory subsystems, and generate remote memory access information for use by a networking device to perform remote memory access operations. For example, the clustered memory system of the present disclosure may include a first computing system that is coupled to a second computing system via a network, and that includes a clustered memory management subsystem coupled to a central processing subsystem and a networking device. The clustered memory management subsystem receives a processor memory-centric access request associated with a memory access operation from the central processing subsystem, and uses memory management hardware to determine that the processor memory-centric access request is directed to a second memory subsystem in the second computing system. The clustered memory management subsystem then uses remote memory access hardware to generate memory access information for performing the memory access operation at the second memory subsystem, and instructs the networking device to utilize the memory access information to transmit at least one memory access communication that performs the memory access operation at the second memory subsystem. Thus, a global memory address space may be presented to the operating system and/or other applications in a computing system, while allowing memory access operations requested for those applications by a central processing subsystem via processor memory-centric access requests to be converted to remote memory access operations via high-speed, dedicated hardware and without the use of remote memory access software (e.g., middleware/libraries/sockets) executed by that central processing subsystem. - In some embodiments, during or prior to the
method 500, theSCP subsystems 304 in each computing system 202 a-202 c/300 may establish secure communication channels with each other. For example, each of theSCP subsystems 304 in the computing systems 202 a-202 c/300 may be configured to perform the secure communication functionality described by the inventors of the present disclosure in U.S. patent application Ser. No. 17/079,737, attorney docket no. 16356.2217US01, filed on Oct. 26, 2020, the disclosure of which is incorporated herein by reference in its entirety. Thus, as described in that application, theSCP subsystem 304 in thecomputing system 202 b/300 may identify theSCP subsystem 304 in thecomputing system 202 a/300, sign a second SCP authentication communication with a second private key, and transmit the second signed SCP authentication communication to theSCP subsystem 304 in thecomputing system 202 a/300, while theSCP subsystem 304 in thecomputing system 202 a/300 signs a first SCP authentication communication with a first private key, and transmits the first signed SCP authentication communication to theSCP subsystem 304 in thecomputing system 202 b/300. TheSCP subsystem 304 in thecomputing system 202 b/300 may then authenticate the first SCP authentication communication using a first public key, theSCP subsystem 304 in thecomputing system 202 a/300 may authenticate the second SCP authentication communication using a second public key and, in response, theSCP subsystems 304 in thecomputing systems 202 a/300 and 202 b/300 will establish a secure communication channel. - As also described in that application, the
SCP subsystem 304 in thecomputing system 202 b/300 may then identify theSCP subsystem 304 in thecomputing system 202 c/300, sign a second SCP authentication communication with a second private key, and transmit the second signed SCP authentication communication to theSCP subsystem 304 in thecomputing system 202 c/300, while theSCP subsystem 304 in thecomputing system 202 c/300 signs a third SCP authentication communication with a third private key, and transmits the third signed SCP authentication communication to theSCP subsystem 304 in thecomputing system 202 b/300. TheSCP subsystem 304 in thecomputing system 202 b/300 may then authenticate the third SCP authentication communication using a third public key, theSCP subsystem 304 in thecomputing system 202 c/300 may authenticate the second SCP authentication communication using a second public key and, in response, theSCP subsystems 304 in thecomputing systems 202 b/300 and 202 c/300 will establish a secure communication channel. - As also described in that application, the
SCP subsystem 304 in thecomputing system 202 b/300 may then attest to the authentication of theSCP subsystem 304 in thecomputing system 202 c/300 to theSCP subsystem 304 in thecomputing system 202 a/300, and attest to the authentication of theSCP subsystem 304 in thecomputing system 202 a/300 to theSCP subsystem 304 in thecomputing system 202 c/300, which allows theSCP subsystems 304 in thecomputing systems 202 a/300 and 202 c/300 to establish a secure communication channel without transmitting signed SCP authentication communications. Furthermore, any SCP subsystem authenticated by an SCP subsystem may have its authentication attested to other SCP subsystems to quickly and securely grow the secure communication fabric between the SCP subsystems. As also described in that application, enabling keys may provide for use of the secure communication channels by each of theSCP subsystems 304 in thecomputing systems 202 a/300, 202 b/300, and 202 c/300 to securely exchange communications, and continued performance of the platform root-of-trust functionality referenced above by those SCP subsystems will ensure that the secure communication channels are only maintained with trusted SCP subsystems and/or computing systems. As such, secure communication channels may be established between theSCP subsystems 304 in the computing systems 202 a-202 c/300 that each operate as trusted systems to perform any of the inter-SCP subsystem communications discussed below in a secure manner. - The
method 500 begins atblock 502 where a global memory address map is generated. In an embodiment, atblock 502, theSCP subsystems 304/400 in the computing systems 202 a-202 c/300 may power on, reset, reboot, and/or otherwise initialize and, in response, the clusteredmemory management engine 404 in each of thoseSCP subsystems 304/400 may retrieve its “local” memory subsystem information, clustered memory map information, and/or other clustered memory information known in the art, and broadcast that clustered memory information to theother SCP subsystems 304/400. As such, atblock 502, the clusteredmemory management engine 404 in each of theSCP subsystems 304/400 in the computing systems 202 a-202 c/300 may retrieve the clustered memory information broadcast by each of theother SCP subsystems 304/400, which allows the clusteredmemory management engine 404 in each of theSCP subsystems 304/400 to generate a global memory address map, configure memory subsystem offsets, configure Base Address Register (BAR) windowing registers, and/or perform any other global memory information that one of skill in the art in possession of the present disclosure will appreciate would provide for the clustered memory functionality discussed below. As will be appreciated by one of skill in the art in possession of the present disclosure, while not explicitly described above, the global memory address map generation may involve the management system 206 (e.g., an SCP manager) while remaining within the scope of the present disclosure as well. - With reference to
FIG. 6 , an embodiment of a globalmemory address map 600 is illustrated that may be generated by a simplified clustered memory system provided according to the teachings of the present disclosure. In the illustrated example, global memory address map 600 is provided for two computing systems, with a first computing system (e.g., the computing system 202 a in the examples below) including the central processing subsystem 600 and the SCP subsystem 602 coupled together and associated with a local address space 604 that is provided by an address space 604 a that is directly connected to the central processing subsystem 600 (e.g., an address space provided by the memory subsystems 310 a-310 c in the computing system 300 discussed above), and address spaces 604 b and 604 c that are directly connected to the SCP subsystem 602 (e.g., address spaces provided by the memory subsystems 416 in the SCP system 400 discussed above), and a second computing system (e.g., the computing system 202 b in the examples below) including the central processing subsystem 606 and the SCP subsystem 608 coupled together and associated with a local address space 610 that is provided by an address space 610 a that is directly connected to the central processing subsystem 606 (e.g., an address space provided by the memory subsystems 310 a-310 c in the computing system 300 discussed above), and address spaces 610 b and 610 c that are directly connected to the SCP subsystem 608 (e.g., address spaces provided by the memory subsystems 416 in the SCP system 400 discussed above). Furthermore, a network 612 (e.g., thenetwork 204 discussed above with reference toFIG. 2 ) couples together the first computing system and the second computing system via theirrespective SCP subsystems - As such,
FIG. 6 illustrates how both thecentral processing subsystem 600 andSCP subsystem 602 in the first computing system, and both thecentral processing subsystem 606 andSCP subsystem 608 in the second computing system, are associated with aglobal address space 614 that includes theaddress space 604 a that is directly connected to thecentral processing subsystem 600, theaddress spaces SCP subsystem 602, theaddress space 610 a that is directly connected to thecentral processing subsystem 606, and theaddress spaces SCP subsystem 608. One of skill in the art in possession of the present disclosure will appreciate how thecentral processing subsystem 600 andSCP subsystem 602 in the first computing system may view theaddress spaces 604 a-604 c as “local” address space (i.e., local to their first computing system) and theaddress spaces 610 a-610 c as “remote” address space (i.e., remote from their first computing system), while thecentral processing subsystem 606 andSCP subsystem 608 in the second computing system may view theaddress spaces 604 a-604 c as “remote” address space (i.e., remote from their second computing system) and theaddress spaces 610 a-610 c as “local” address space (i.e., local to their second computing system). - Furthermore, while not described in detail herein, one of skill in the art in possession of the present disclosure will appreciate how memory subsystems offsets may be configured for the memory subsystem(s) in the first computing system and the second computing system in order to address the use of different/inconsistent addressing information for the memory subsystems in the first and second computing systems (e.g., a remote memory address in a memory subsystem included in the second computing system may be defined as “address 1000” in the first computing system while being defined as “address 2000” in the second computing system, with the memory subsystem offsets utilized to convert a memory access request for “address 1000” by the
central processing subsystem 600 in the first computing system to a memory access request for “address 2000” in the second computing system). - Furthermore, while not described in detail herein, one of skill in the art in possession of the present disclosure will also appreciate how BAR windowing registers in the central processing subsystem coupling 408 may be configured at
block 502 as well. As will be recognized by one of skill in the art in possession of the present disclosure, a BAR may provide the beginning of the address space assigned to a “local” SCP subsystem by its corresponding “local” central processing subsystem (e.g., an x86 host CPU), and all of the memory subsystem for local SCP subsystem, as well as remote memory subsystems, may be accessed by that central processing subsystem generating an access identified by “BAR+an offset” that identifies where the access (e.g., a read or write) is targeted. As such, configuration operations may be performed atblock 502 to provide BAR windowing register functionality discussed above, as well as any other BAR windowing functionality known in the art, while remaining within the scope of the present disclosure. However, while a specific, simplified global memory address map is illustrated and described, one of skill in the art in possession of the present disclosure will appreciate that global memory address maps utilized with the clustered memory system of the present disclosure may be more complicated and/or may include different features than described above while remaining within the scope of the present disclosure as well. - The
method 500 then proceeds to block 504 where a clustered memory management subsystem receives a processor memory-centric access request from a central processing subsystem. As illustrated inFIG. 7A , in some embodiments, theapplication 309 provided in thecomputing system 202 a/300 may perform application memoryaccess request operations 700 that may include generating and transmitting an application request to perform a memory access to theCPU 308 a in thecomputing system 202 a/300, with that application request identifying a memory read operation, a memory write operation, and/or other memory access operations known in the art, along with any other memory access operation information (e.g., data to write as part of the memory write operation, etc.) that would be apparent to one of skill in the art in possession of the present disclosure. As will be appreciated by one of skill in the art in possession of the present disclosure, theSCP subsystem 304/400 in thecomputing system 202 a/300 may present the global memory address space to the applications 309 (e.g., an operating system and/or other applications) in thecomputing system 202 a/300, which allows theapplication 309 to identify a memory address from that global memory address space in its application request to perform a memory access. - In response to receiving the application request to perform the memory access from the
application 309, theCPU 308 a in thecomputing system 202 a/300 may perform CPU memoryaccess request operations 702 that include generating and transmitting a CPU request to perform a memory access to theMMU 308 b in thecomputing system 202 a/300. In several embodiments, the CPU request to perform the memory access is a processor memory-centric access request that is generated by theCPU 308 a and that includes a memory subsystem address pointer (e.g., *ptr) that points to a memory address (e.g., from the global memory address space) at which the memory access operation requested by the application (e.g., the memory read operation, memory write operation, and/or other memory access operations) should be performed, and that may include any other memory access operation information (e.g., data to write as part of the memory write operation, etc.) that would be apparent to one of skill in the art in possession of the present disclosure. - As such, one of skill in the art in possession of the present disclosure will appreciate that the processor memory-centric access requests described herein utilize the native protocol of the central processing subsystem (e.g., a CPU) to perform CPU load operations, and CPU store operations. For example, one of skill in the art in possession of the present disclosure will recognize that the generation of an Ethernet packet may require a CPU to perform thousands of load/store operations in order to program an Ethernet controller and create an Ethernet descriptor that allows the Ethernet controller to send and/or receive data packet. Furthermore, one of skill in the art in possession of the present disclosure will also recognize that CPUs utilize processor memory-centric access requests (e.g., native load/store instructions) to communicate with Dynamic Random Access Memory (DRAM), and load/store operations an conventionally mapped to native CPU instructions, while Ethernet send/receive operations are not conventionally mapped to native CPU instructions. However, while a specific CPU-memory-access request is described, one of skill in the art in possession of the present disclosure will appreciate that the
CPU 308 a may utilize other memory pointer access techniques/native memory semantics to provide the CPU request to perform the memory access while remaining within the scope of the present disclosure as well. - As will be appreciated by one of skill in the art in possession of the present disclosure, in response to receiving the CPU request to perform the memory access from the
CPU 308 a, theMMU 308 b in thecomputing system 202 a/300 may reference a global memory address map (e.g., the globalmemory address map 600 discussed above) to determine whether that the CPU request to perform the memory access is directed to a computing system “local” memory subsystem (e.g., using the example of the globalmemory address map 600 above, the processor memory-centric access request points to a memory address included in theaddress space 604 a provided by any of the memory subsystems 310 a-310 c in thecomputing system 202 a/300), or a computing system “remote” memory subsystem (e.g., using the example of the globalmemory address map 600 above, the processor memory-centric access request points to a memory address included in theaddress spaces memory subsystems 416 included in theSCP subsystem 304 in thecomputing system 202 a/300, or to a memory address included in theaddress spaces 612 a-612 c provided by the memory subsystems included inother computing systems - With reference to
FIG. 7B , in response to theMMU 308 a in thecomputing system 202 a/300 determining that the CPU request to perform the memory access is directed to a computing system “local” memory subsystem (e.g., the processor memory-centric access request points to a memory address included in theaddress space 604 a provided by thememory subsystem 310 a in thecomputing system 202 a/300 in this example), thecentral processing subsystem 308 in thecomputing system 202 a/300 may perform thememory access operation 704 specified in that CPU request on thememory subsystem 310 a in thecomputing system 202 a/300. As discussed above, in some examples thememory access operation 704 specified in the CPU request to perform the memory access may include a memory write operation, and thememory access operations 704 may include writing data included in that CPU request to thememory subsystem 310 a. As also discussed above, in some examples thememory access operation 704 specified in the CPU request to perform the memory access may include a memory read operation, and thememory access operations 704 may include reading data from thememory subsystem 310 a and returning it to theapplication 309. However, while two specific examples are provided, one of skill in the art in possession of the present disclosure will appreciate that thememory access operation 704 may include any of a variety of memory access operations while remaining within the scope of the present disclosure as well. - With reference to
FIG. 7C , in response to theMMU 308 a in thecomputing system 202 a/300 determining that the CPU request to perform the memory access is directed to a computing system “remote” memory subsystem (e.g., using the example of the globalmemory address map 600 above, the processor memory-centric access request points to a memory address included in theaddress spaces memory subsystems 416 included in theSCP subsystem 304 or a memory address included in theaddress spaces 612 a-612 c provided by the memory subsystems included in thecomputing system 202 a/300), theMMU 308 b in thecomputing system 202 a/300 may perform CPU memoryrequest forwarding operations 706 to forward the CPU request to perform the memory access to theSCP subsystem 304 in thecomputing system 202 a/300. - As such, with reference to
FIG. 7D and in an embodiment of block 504, the MMU sub-engine 404 b in the clusteredmemory management engine 404 that is provided by the hardware subsystem and included in theSCP subsystem 304/400 in thecomputing system 202 a/300 may perform memory accessrequest receiving operations 708 that include receiving the CPU request to perform the memory access (e.g., the processor memory-centric access request) via the central processing subsystem coupling 408 and from theMMU 308 a in thecentral processing subsystem 308 in thecomputing system 202 a/300. In a specific example, the CPU request to perform the memory access (e.g., the processor memory-centric access request) may be transmitted by theMMU 308 a in thecentral processing subsystem 308 in thecomputing system 202 a/300 via a CxL port on thatcentral processing subsystem 308, and received by the MMU sub-engine 404 b in clusteredmemory management engine 404 of theSCP subsystem 304/400 in thecomputing system 202 a/300 via CxL root .mem/.cache BAR windows provided by the central processing subsystem coupling 408 (e.g., in-bound memory windows associated with global memory address space). However, while specific subsystems for receiving the processor memory-centric access request at the clustered memory management subsystem are described, one of skill in the art in possession of the present disclosure will appreciate that other subsystems/techniques will fall within the scope of the present disclosure as well. - The
method 500 then proceeds to decision block 506 where it is determined whether the processor memory-centric access request provides a local memory access or a remote memory access. In an embodiment, atdecision block 506 in response to receiving the CPU request to perform the memory access from theCPU 308 a (e.g., the processor memory-centric access request), the MMU sub-engine 404 b in the clusteredmemory management engine 404 that is provided by the hardware subsystem and included in theSCP subsystem 304/400 in thecomputing system 202 a/300 may reference a global memory address map (e.g., the globalmemory address map 600 discussed above) to determine whether that the CPU request to perform the memory access is directed to an SCP “local” memory subsystem (e.g., using the example of the globalmemory address map 600 above, the processor memory-centric access request points to a memory address included in theaddress spaces SCP subsystem 304/400 in thecomputing system 202 a/300), or an SCP “remote” memory subsystem (e.g., using the example of the globalmemory address map 600 above, the processor memory-centric access request points to a memory address included in theaddress spaces 612 a-612 c provided by the memory subsystems included in thecomputing systems - If, at
decision block 506, it is determined that the processor memory-centric access request provides a local memory access, themethod 500 proceeds to block 508 where the clustered memory management subsystem performs local memory access operations. In an embodiment, atdecision block 506 the MMU sub-engine 404 b in the clusteredmemory management engine 404 that is provided by the hardware subsystem and included in theSCP subsystem 304/400 in thecomputing system 202 a/300 may determine that the CPU request to perform the memory access is directed to an SCP “local” memory subsystem (e.g., the processor memory-centric access request points to a memory address included in theaddress spaces SCP subsystem 304/400 in thecomputing system 202 a/300). With reference toFIG. 7E , in response to determining that the CPU request to perform the memory access is directed to an SCP “local” memory subsystem, the clusteredmemory management engine 404 in theSCP subsystem 304/400 in thecomputing system 202 a/300 may perform thememory access operation 710 specified in that CPU request via the memory port(s) 414 (e.g., CxL host ports) and on the memory subsystem(s) 416 in theSCP subsystem 304/400 in thecomputing system 202 a/300. - As discussed above, in some examples the
memory access operation 710 specified in the CPU request to perform the memory access may include a memory write operation, and thememory access operations 710 may include writing data included in that CPU request to the memory subsystem(s) 416. As also discussed above, in some examples thememory access operation 710 specified in the CPU request to perform the memory access may include a memory read operation, and thememory access operation 710 may include reading data from the memory subsystem(s) 416 and returning it to the application 309 (e.g., via the central processing subsystem coupling 408 in thatSCP subsystem 304/400 and thecentral processing subsystem 308 in thecomputing system 202 a/300). However, while two specific examples are provided, one of skill in the art in possession of the present disclosure will appreciate that thememory access operation 710 may include any of a variety of memory access operations while remaining within the scope of the present disclosure as well. - If, at
decision block 506, it is determined that the processor memory-centric access request provides a remote memory access, themethod 500 proceeds to block 510 where the clustered memory management subsystem generates memory access information for performing remote memory access operation. In an embodiment, atdecision block 506 the MMU sub-engine 404 b in the clusteredmemory management engine 404 that is provided by the hardware subsystem and included in theSCP subsystem 304/400 in thecomputing system 202 a/300 may determine that the CPU request to perform the memory access is directed to an SCP “remote” memory subsystem (e.g., the processor memory-centric access request points to a memory address included in theaddress spaces 612 a-612 c provided by the memory subsystems included in thecomputing systems - However, in some examples and in response to determining that the CPU request to perform the memory access is directed to an SCP “remote” memory subsystem, the clustered
memory management engine 404 in theSCP subsystem 304/400 in thecomputing system 202 a/300 may determine whether data associated with the memory access operation identified in that CPU request (e.g., a memory read operation) in stored in thecache device 405 and, if so, may perform the memory access operation (e.g., reading the data) using thecache device 405 and without the need to proceed to block 510. As such, some remote memory access operations may be accomplished via the “local” cache device and, in the case of memory read operations, may include the clusteredmemory management engine 404 in theSCP subsystem 304/400 in thecomputing system 202 a/300 determining whether the latest read data satisfying the remote memory access operations is located in thecache device 405 and, if so, transmitting that read data via the central processing subsystem coupling 408 in theSCP subsystem 304/400 in thecomputing system 202 a/300 and to theapplication 309 via thecentral processing subsystem 308 in thecomputing system 202 a/300. As such, one of skill in the art in possession of the present disclosure will appreciate how theSCP subsystem 304/400 in thecomputing system 202 a/300 may operate to manage cache coherency in some embodiments, while in other embodiments cache coherency management may be performed using application-level software coherency mapping techniques that would be apparent to one of skill in the art in possession of the present disclosure. - With reference to
FIG. 7F , in response to determining that the CPU request to perform the memory access is directed to an SCP “remote” memory subsystem, the MMU sub-engine 404 b in the clusteredmemory management engine 404 that is provided by the hardware subsystem and included in theSCP subsystem 304/400 in thecomputing system 202 a/300 may perform memoryrequest transmission operations 712 to transmit the CPU request to perform the memory access (e.g., the processor memory-centric access request) to the remote memory access sub-engine 404 a in the clusteredmemory management engine 404 that is provided by the hardware subsystem (e.g., a high speed state machine) included in theSCP subsystem 304/400 in thecomputing system 202 a/300. - With reference to
FIG. 7G , in response to receiving the CPU request to perform the memory access (e.g., the processor memory-centric access request), the remote memory access sub-engine 404 a in the clusteredmemory management engine 404 that is provided by the hardware subsystem and included in theSCP subsystem 304/400 in thecomputing system 202 a/300 may perform remote memoryaccess generation operations 714 to generate memory access information for performing remote memory access operation, and store that memory access information in the clustered memory management database 406 (e.g., provided by SRAM device(s)). For example, the remote memoryaccess generation operations 704 may include the use of any information included in the CPU request to perform the memory access (e.g., the processor memory-centric access request) in order to generate an Ethernet descriptor for thecontroller 412 a (e.g., an Ethernet controller) in thenetworking device 412 that identifies memory address information, memory access operations, and/or any other remote memory access information that one of skill in the art in possession of the present disclosure would recognize as allowing for the remote memory access operations discussed below. - In a specific example, the remote memory
access generation operations 704 may include the utilization of the global memory address map and the CPU request in order to generate RDMA descriptors and/or any of a variety of RDMA information and/or instructions that allow for RDMA communications that enable the remote memory access operations discussed below. Furthermore, as discussed above, memory subsystems offsets may be utilized in the generation of the memory access information in order to address the use of different/inconsistent addressing information for the memory subsystems in thecomputing systems computing system 202 a. As will be appreciated by one of skill in the art in possession of the present disclosure, the generation of RDMA information for the performance of RDMA communications in the present disclosure may be performed by the remote memory access sub-engine 404 a provided by the hardware subsystem, and without the conventional middleware, libraries, sockets, and/or other software required by the x86 host processor in conventional clustered memory systems. However, while RDMA communications provided according to the RDMA protocol are described, one of skill in the art in possession of the present disclosure will appreciate that the remote memory access operations discussed below may be enabled using Ethernet communications according to Ethernet protocols and/or other protocols while remaining within the scope of the present disclosure as well. - The
method 500 then proceeds to block 512 where the clustered memory management subsystem instructs a networking device to utilize the memory access information to perform the remote memory access operation. With reference toFIG. 7H , in an embodiment of block 512, the remote memory access sub-engine 404 a in the clusteredmemory management engine 404 that is provided by the hardware subsystem and included in theSCP subsystem 304/400 in thecomputing system 202 a/300 may perform memory access informationutilization instruction operations 716 that include transmitting an instruction to thecontroller 412 a in thenetworking device 412 to utilize the memory access information in the clusteredmemory management database 406 to perform a remote memory access operation. - The
method 500 then proceeds to block 514 where the networking device retrieves the memory access information and performs the remote memory access operation. With reference toFIG. 71 , in an embodiment ofblock 514 and in response to receiving the instruction to utilize the memory access information in the clusteredmemory management database 406 to perform a remote memory access operation, thecontroller 412 a (e.g., an Ethernet controller) in thenetworking device 412 in theSCP subsystem 304/400 in thecomputing system 202 a/300 may perform remote memory accessinformation access operations 718 to access the remote memory access information in the clusteredmemory management database 406 and use that remote memory access information to perform remote memory access operations. For example, atblock 514, thecontroller 412 a in thenetworking device 412 in theSCP subsystem 304/400 in thecomputing system 202 a/300 may generate RDMA or Ethernet communications using the remote memory access information in SRAM provided by the clusteredmemory management database 406. - With reference to
FIG. 7J , thecontroller 412 a in thenetworking device 412 in theSCP subsystem 304/400 in thecomputing system 202 a/300 may perform remotememory access operations 720 that include transmitting remote memory access communications (e.g., the RDMA or Ethernet communications discussed above) via thenetwork 204 to thecomputing system 202 b/300. As illustrated inFIG. 7K , thenetworking device 412 in theSCP subsystem 304/400 in thecomputing system 202 b/300 may receive those remote memory access communications via thenetwork 204 and, in response, perform remote memory accesscommunication transmission operations 722 that include transmitting the remote memory access communications to the clusteredmemory management engine 404 in theSCP subsystem 304/400 in thecomputing system 202 b/300. - The clustered
memory management engine 404 in theSCP subsystem 304/400 in thecomputing system 202 b/300 may receive the remote memory access communications and, in response, determine whether that remote memory access communication is directed to an SCP “local” memory subsystem (e.g., using the example of the globalmemory address map 600 discussed above, the remote memory access communication identifies a memory address included in theaddress spaces SCP subsystem 304/400 in thecomputing system 202 b/300), or a computing system “local ” memory subsystem (e.g., using the example of the globalmemory address map 600 discussed above, the remote memory access communication identifies a memory address included in theaddress space 604 a provided by the memory subsystems 310 a-310 c included in the computing system 202/300). As illustrated inFIG. 7L , if it is determined that the remote memory access communication is directed to an SCP “local” memory subsystem, the clusteredmemory management engine 404 in theSCP subsystem 304/400 in thecomputing system 202 b/300 may performmemory access operations 724 specified in those remote memory access communications via the memory port(s) 414 (e.g., CxL host ports) and on the memory subsystem(s) 416 in theSCP subsystem 304/400 in thecomputing system 202 b/300. - As discussed above, in some examples the
memory access operation 724 specified in the remote memory access communication may include a memory write operation, and thememory access operations 724 may include writing data included in that remote memory access communication to the memory subsystem(s) 416 in thecomputing system 202 b/300. As also discussed above, in some examples thememory access operation 724 specified in the remote memory access communication may include a memory read operation, and thememory access operation 724 may include reading data from the memory subsystem(s) 416 in theSCP subsystem 304/400 in thecomputing system 202 b/300 and returning it to theapplication 309 in thecomputing system 202 a/300 (e.g., via thenetworking device 412 in theSCP subsystem 304/400 in thecomputing system 202 b/300 and though thenetwork 204 to theSCP subsystem 304/400 in thecomputing system 202 a/300, via thenetworking device 412, the clusteredmemory management engine 404, and the central processing subsystem coupling 408 in theSCP subsystem 304/400 in thecomputing system 202 a/300, and via thecentral processing subsystem 308 in thecomputing system 202 a/300). However, while two specific examples are provided, one of skill in the art in possession of the present disclosure will appreciate that thememory access operation 724 may include any of a variety of memory access operations while remaining within the scope of the present disclosure as well. - As illustrated in
FIGS. 7M and 7N , if it is determined that the remote memory access communication is directed to a computing system “local” memory subsystem, the clusteredmemory management engine 404 in theSCP subsystem 304/400 in thecomputing system 202 b/300 may performmemory access operations 726 specified in those remote memory access communications via the central processing subsystem coupling 408 in theSCP subsystem 304/400 in thecomputing system 202 b/300 and thecentral processing subsystem 308 in thecomputing system 202 b/300, and on thememory subsystem 310 b in thecomputing system 202 b/300. As discussed above, in some examples thememory access operation 726 specified in the remote memory access communication may include a memory write operation, and thememory access operations 726 may include writing data included in that remote memory access communication to thememory subsystem 310 b in thecomputing system 202 b/300. As also discussed above, in some examples thememory access operation 726 specified in the remote memory access communication may include a memory read operation, and thememory access operation 726 may include reading data from thememory subsystem 310 b in thecomputing system 202 b/300 and returning it to theapplication 309 in thecomputing system 202 a/300 (e.g., theSCP subsystem 304/400 in thecomputing system 202 b/300 and though thenetwork 204 to theSCP subsystem 304/400 in thecomputing system 202 a/300, via thenetworking device 412, the clusteredmemory management engine 404, and the central processing subsystem coupling 408 in theSCP subsystem 304/400 in thecomputing system 202 a/300, and via thecentral processing subsystem 308 in thecomputing system 202 a/300). However, while two specific examples are provided, one of skill in the art in possession of the present disclosure will appreciate that thememory access operation 726 may include any of a variety of memory access operations while remaining within the scope of the present disclosure as well. - The
method 500 then proceeds to block 516 where the clustered memory management subsystem determines a memory access operation completion status and reports it to the central processing subsystem. In an embodiment, atblock 516, the clusteredmemory management engine 404 in theSCP subsystem 304/400 in thecomputing system 202 b/300 may report a completion status (e.g., a successful/completed status, an unsuccessful/incomplete status, etc.) of its memory access operation to the clusteredmemory management engine 404 in theSCP subsystem 304/400 in thecomputing system 202 a/300, and the clusteredmemory management engine 404 in theSCP subsystem 304/400 in thecomputing system 202 a/300 may report that completion status to thecentral processing subsystem 308 in thecomputing system 202 a/300. For example, with reference toFIG. 7O , completion status reporting operations may include theSCP subsystem 304 in thecomputing system 202 a/300 receive the completion status for the remote memory access operations from theSCP subsystem 304 in thecomputing system 202 b/300 and transmitting that completion status to thecentral processing subsystem 308 in thecomputing system 202 a/300, which may then transmit that completion status to theapplication 309 provided in thecomputing system 202 a/300. As such, theapplication 308 may be informed of the completion status (e.g., a successful/completed status, an unsuccessful/incomplete status, etc.) of its memory access request. - Thus, systems and methods have been described that provide an SCP subsystem that utilizes hardware subsystems to identify processor memory-centric access requests that are directed to remote memory subsystems, and generate remote memory access information for use by a networking device to perform remote memory access operations. For example, the clustered memory system of the present disclosure may include a first server device that is coupled to a second server device via a network, and that includes a SCP subsystem coupled to a central processing subsystem and a networking device. The SCP subsystem receives a processor memory-centric access request associated with a memory access operation from the central processing subsystem, and uses memory management hardware to determine that the processor memory-centric access request is directed to a second memory subsystem in the second server device. The SCP subsystem then uses remote memory access hardware to generate memory access information for performing the memory access operation at the second memory subsystem, and instructs the networking device to utilize the memory access information to transmit at least one memory access communication that performs the memory access operation at the second memory subsystem. Thus, a global memory address space may be presented to the operating system and/or other applications in a server device, while allowing memory access operations requested for those applications by a central processing subsystem via processor memory-centric access requests to be converted to remote memory access operations via high speed, dedicated hardware and without the use of remote memory access software (e.g., middleware/libraries/sockets) executed by that central processing subsystem.
- Although illustrative embodiments have been shown and described, a wide range of modification, change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. Accordingly, it is appropriate that the appended claims be construed broadly and in a manner consistent with the scope of the embodiments disclosed herein.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/085,133 US11334245B1 (en) | 2020-10-30 | 2020-10-30 | Native memory semantic remote memory access system |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/085,133 US11334245B1 (en) | 2020-10-30 | 2020-10-30 | Native memory semantic remote memory access system |
Publications (2)
Publication Number | Publication Date |
---|---|
US20220137816A1 true US20220137816A1 (en) | 2022-05-05 |
US11334245B1 US11334245B1 (en) | 2022-05-17 |
Family
ID=81379980
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/085,133 Active US11334245B1 (en) | 2020-10-30 | 2020-10-30 | Native memory semantic remote memory access system |
Country Status (1)
Country | Link |
---|---|
US (1) | US11334245B1 (en) |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7593284B2 (en) | 2007-10-17 | 2009-09-22 | Unity Semiconductor Corporation | Memory emulation using resistivity-sensitive memory |
US9626108B2 (en) | 2014-09-16 | 2017-04-18 | Kove Ip, Llc | Dynamically provisionable and allocatable external memory |
CN112615917B (en) * | 2017-12-26 | 2024-04-12 | 华为技术有限公司 | Storage device management method in storage system and storage system |
-
2020
- 2020-10-30 US US17/085,133 patent/US11334245B1/en active Active
Also Published As
Publication number | Publication date |
---|---|
US11334245B1 (en) | 2022-05-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10095639B2 (en) | Multi-processor startup system | |
JP6514330B2 (en) | System on chip with multiple compute subsystems | |
US10824574B2 (en) | Multi-port storage device multi-socket memory access system | |
JP2017519294A (en) | Multi-host power controller (MHPC) for flash memory-based storage devices | |
US11726797B2 (en) | Secondary processor device ownership system | |
US11157191B2 (en) | Intra-device notational data movement system | |
US12056066B2 (en) | System, device, and method for accessing memory based on multi-protocol | |
US11853803B2 (en) | Workload compliance governor system | |
US11334245B1 (en) | Native memory semantic remote memory access system | |
US11775465B2 (en) | Intra-chassis device multi-management domain system | |
US11601262B2 (en) | Distributed key management system | |
US11288008B1 (en) | Reflective memory system | |
US11501002B2 (en) | Protocol security system | |
US20220035550A1 (en) | Data mover selection system | |
US10936219B2 (en) | Controller-based inter-device notational data movement system | |
US11989579B2 (en) | Secondary processor proxied device ownership system | |
US11593120B1 (en) | Secondary processor device ownership assignment system | |
US11500747B2 (en) | Computer initialization debug message display system | |
US11301259B1 (en) | System control processor (SCP) cloning system | |
US11281612B2 (en) | Switch-based inter-device notational data movement system | |
US11734429B1 (en) | Secure bios-enabled passthrough system | |
US11822816B2 (en) | Networking device/storage device direct read/write system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: DELL PRODUCTS L.P., TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HORMUTH, ROBERT W.;PIKE, JIMMY D.;JREIJ, ELIE;AND OTHERS;SIGNING DATES FROM 20201023 TO 20201026;REEL/FRAME:054224/0456 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, NORTH CAROLINA Free format text: SECURITY AGREEMENT;ASSIGNORS:EMC IP HOLDING COMPANY LLC;DELL PRODUCTS L.P.;REEL/FRAME:054591/0471 Effective date: 20201112 |
|
AS | Assignment |
Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT, TEXAS Free format text: SECURITY INTEREST;ASSIGNORS:EMC IP HOLDING COMPANY LLC;DELL PRODUCTS L.P.;REEL/FRAME:054475/0523 Effective date: 20201113 Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS COLLATERAL AGENT, TEXAS Free format text: SECURITY INTEREST;ASSIGNORS:EMC IP HOLDING COMPANY LLC;DELL PRODUCTS L.P.;REEL/FRAME:054475/0609 Effective date: 20201113 Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT, TEXAS Free format text: SECURITY INTEREST;ASSIGNORS:EMC IP HOLDING COMPANY LLC;DELL PRODUCTS L.P.;REEL/FRAME:054475/0434 Effective date: 20201113 |
|
AS | Assignment |
Owner name: EMC IP HOLDING COMPANY LLC, TEXAS Free format text: RELEASE OF SECURITY INTEREST AT REEL 054591 FRAME 0471;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058001/0463 Effective date: 20211101 Owner name: DELL PRODUCTS L.P., TEXAS Free format text: RELEASE OF SECURITY INTEREST AT REEL 054591 FRAME 0471;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058001/0463 Effective date: 20211101 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: DELL PRODUCTS L.P., TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (054475/0609);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:062021/0570 Effective date: 20220329 Owner name: EMC IP HOLDING COMPANY LLC, TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (054475/0609);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:062021/0570 Effective date: 20220329 Owner name: DELL PRODUCTS L.P., TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (054475/0434);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:060332/0740 Effective date: 20220329 Owner name: EMC IP HOLDING COMPANY LLC, TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (054475/0434);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:060332/0740 Effective date: 20220329 Owner name: DELL PRODUCTS L.P., TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (054475/0523);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:060332/0664 Effective date: 20220329 Owner name: EMC IP HOLDING COMPANY LLC, TEXAS Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (054475/0523);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:060332/0664 Effective date: 20220329 |