DE102022205478A1 - INTERBUS MEMORY MAPPING - Google Patents

Abstract

Computerized system for efficient interaction between a host, the host having a first operating system and a second operating system, the system comprising a subsystem on the second operating system that extracts data directly from a buffer local to the host, the A system is operable to map memory from one bus associated with the first operating system to another bus associated with the second operating system and from which the memory is accessed, thereby establishing a connection between the first and second operating systems by emulating a cross-bus memory map.

Description

FIELD OF THE INVENTION

This invention relates generally to hardware storage devices and more particularly to networked hardware storage devices.

BACKGROUND OF THE INVENTION

Null copies are known within a single operating system, e.g. B. by using the Linux TCP stack.

PCIe (Peripheral Component Interconnect Express), also abbreviated as PCI-e, is an example of a high-speed serial computer expansion bus standard, described among others in the Wikipedia entry on PCI_Express.

The following article: lwn.net/Articles/752188, which is available online, describes a MSG ZEROCOPY function added in (Linux Kernel) 4.14 that "allows zero-copy transmission of data", and adds, that "the implementation of zero-copy reception must map (or allocate) packet buffers into user-space memory".

BlueField ^® -2 and ConnectX ^® -6 are well-known NVIDIA ^® products.

BlueField ^® -2 are next-generation cloud SmartNICs and I/O Processing Unit (IPU) solutions that deliver unprecedented security, performance, and efficiency for data centers at scale and for any workload. ConnectX ^® -6 Dx and BlueField ^® -2 continue Mellanox's innovation in high-performance cloud fabrics, making the impossible possible by introducing state-of-the-art hardware acceleration engines and best-in-class software programmability for mission-critical applications. They offer a new level of security and efficiency with the highest return on investment (ROI) for hyperscale, enterprise and telecom environments.

ConnectX ^® -6 Dx SmartNICs offer up to two ports of 25, 50 or 100Gb/s or a single port of 200Gb/s, Ethernet connectivity with 50Gb/s PAM4 SerDes technology and PCIe 4.0 host connectivity. The ConnectX ^® -6 Dx hardware offload engines include Ipsec and inline TLS data-in-motion cryptography, advanced network virtualization, RDMA over Converged Ethernet (RoCE), and NVMe over Fabrics (NVMe-oF) storage accelerators.

The BlueField ^® -2 IPU integrates all the advanced features of ConnectX ^® -6 Dx with a suite of powerful ARM processor cores, high-performance memory interfaces and flexible processing capabilities in a single System-on-Chip (SoC).

RDMA for data centers based on an Ethernet or mixed protocol fabric is described online here: Mellanox.com/sites/default/files/relateddocs/whitepapers/roce_in_the_data_center.pdf.

Wikipedia states: “In computer science, remote direct memory access (RDMA) is direct memory access from one computer's memory to another's without involving the operating system of one computer or the other. This enables high-throughput, low-latency networking, which is particularly useful in massively parallel computing clusters. RDMA supports zero-copy networking by allowing the network adapter to transfer data from the wire directly to application memory or from application memory directly to the wire, eliminating the need to copy data between application memory and data buffers in the operating system. Such transfers require no work by CPUs, caches, or context switches, and the transfers run in parallel with other system operations. This reduces latency in message transmission.”

However, Wikipedia adds that "this strategy introduces several problems related to not notifying the destination node of the completion of the request (one-way communication)".

The use of network offloads in data centers is described here, among others: Mellanox.com/flles/doc-2020/sb-mellanox-nvme-snap.pdf.

A "socket" typically comprises an endpoint (typically IP address and port number) of a two-way communications link between a first and second program running on the network. The socket can be bound to a port number that allows the TCP layer to identify the application to send data to.

NVMe/TCP is an example of a block storage protocol. NVMe is an example protocol, but references to it are not intended to be limiting; instead, generally any geeig nete local or PCI storage protocol can be used.

Transmission Control Protocol/Internet Protocol (TCP/IP) is an example of a network protocol that can be used by multiple computers to communicate with each other.

SUMMARY OF THE INVENTION

The invention is defined by the claims. Aspects and embodiments are described herein for purposes of illustration of the invention, which may or may not fall within the scope of the claims.

Certain embodiments aim to reduce the number of buffer copies for memory conditioning.

Certain embodiments aim to provide a single device (e.g., the ^ConnectX® card) with multiple PCIe buses, with software configured to map memory from one PCIe bus to another PCIe -Bus can access it, e.g. B. through cross-PCIe mapping and/or transactions. A storage key can be used to describe a buffer (memory); the key can be defined to represent a PCIe-spanning (GVMI) mapping, e.g. B. as described here. However, in a PCIe NTB (Non-Transparent Bridge) implementation, no storage key needs to be provided. In contrast, legacy RDMA bridges the PCIe boundaries through a pair of HW devices (ConnectX ^® cards or other) that define an "RDMA connection". Each device can only handle local PCIe buffers, but the RDMA connection can transfer buffers between devices.

Certain embodiments aim to enable cross-bus memory mapping.

In general, certain embodiments aim to improve IOPS (IO operations per second) and/or reduce latency.

• Ausführungsform 1. Ein computergestütztes System für eine effiziente Interaktion zwischen einem Host, wobei der Host ein erstes Betriebssystem und ein zweites Betriebssystem hat, wobei das System umfasst:
  • ein Subsystem, z.B. auf dem zweiten Betriebssystem, das Daten extrahiert, typischerweise direkt aus einem Puffer, der sich auf dem Host befindet, wobei das System betriebsfähig ist, um einen Speicher von einem Bus, der dem ersten Betriebssystem zugeordnet sein kann, auf einen anderen Bus abzubilden, der dem zweiten Betriebssystem zugeordnet sein kann und von dem aus typischerweise auf den Speicher zugegriffen wird, um dadurch eine Verbindung zwischen dem ersten und dem zweiten Betriebssystem zu emulieren, z.B. durch busübergreifende Speicherabbildung.
• Ausführungsform 2. System gemäß einer der vorangehenden Ausführungsformen, bei dem das erste Betriebssystem eine PCI-artige Konnektivität, einschließlich mehrerer Schichten, mit dem zweiten Betriebssystem aufweist.
• Ausführungsform 3. System gemäß einer der vorhergehenden Ausführungsformen, wobei das Teilsystem einen Netzwerk-Stapel umfasst.
• Ausführungsform 4. System gemäß einer der vorhergehenden Ausführungsformen, wobei der Netzwerk-Stapel so konfiguriert ist, dass er einen Puffer außerhalb des eigenen Speichers des zweiten Betriebssystems identifiziert, der es der Software des zweiten Betriebssystems ermöglicht, auf den Puffer des ersten Betriebssystems zuzugreifen.
• Ausführungsform 5. System gemäß einer der vorangehenden Ausführungsformen, bei dem sich der Puffer, der sich auf dem Host befindet, auf dem Host bleibt.
• Ausführungsform 6. Ein System nach einer der vorhergehenden Ausführungsformen, wobei das System auch eine Anwendung auf dem Host enthält, die:
  • ein Speichervolumen auf einem emulierten NVMe öffnet, um so eine geöffnete Datei bereitzustellen;
  • einen Speicherblock im lokalen Speicher zuweist und Daten als Puffer B in den Speicherblock schreibt;
  • in die geöffnete Datei schreibt, um dadurch einen Schreibaufruf zu definieren; und einen Systemkern-Treiber, der den Schreibaufruf verarbeitet und eine NVMe-Kapsel mit dem Puffer B vorbereitet, bereitstellt.
• Ausführungsform 7. System gemäß einer der vorangehenden Ausführungsformen, wobei das System NVMe-Emulationssoftware und einen TCP-Software-Stapel enthält, die zusammenarbeiten, um eine busübergreifende Zuordnung zu ermöglichen, indem sie es einem geräteübergreifenden Speicherschlüssel erlauben, mehrere Softwareschichten zu durchlaufen, damit Software auf dem zweiten Betriebssystem auf den lokalen Host-Puffer des ersten Betriebssystems zugreifen kann.
• Ausführungsform 8. System gemäß einer der vorhergehenden Ausführungsformen, wobei das erste Betriebssystem auch so konfiguriert ist, dass es Operationen steuert, um dadurch zu ermöglichen, dass der geräteübergreifende Speicherschlüssel die mehreren Softwareschichten durchläuft, wobei die Operationen Folgendes umfassen:
  • Operation 25. der NVMe-Treiber C des Hosts sendet eine Anforderung, die auf den Puffer B zeigt, an ein über die DPU verwaltetes emuliertes NVMe-Gerät D, und das emulierte NVMe-Gerät D stellt dem Host ein NVMe-Gerät (das D emuliert) zur Verfügung;
  • Operation 30. das emulierte NVMe-Gerät D teilt einem Manager E des emulierten NVMe-Geräts D auf einer DPU mit, dass eine NVMe-Anforderung bereit ist, und der Manager E erhält eine Kopie der NVMe-Anforderung im lokalen Speicher des Managers E;
  • Operation 35. Manager E versteht, z. B. durch Zugriff auf die Kopie der NVMe-Anfrage im lokalen Speicher von Manager E, dass die NVMe-Anfrage auf Puffer B verweist, der als Teil des Hostspeichers des Clients einen anderen PCIe-Bus oder eine andere Funktion hat;
  • Operation 40. Manager E startet eine neue Anforderung an einen NVMe-TCP-Initiatortreiber F, der auf dem DPU läuft, mit Attributen zur Ausführung dieser neuen Anforderung, und die neue Anforderung oder deren Attribute umfassen lokale Anforderungsinformationen und den Inhalt des Puffers (B), der entfernt ist;
  • Operation 50. der NVMe-TCP-Initiatortreiber F hält über einen TCP-Benutzerraum-Stapel G auf der DPU einen TCP-Socket mit einem Ziel verbunden;
  • Operation 53. der NVMe-TCP-Initiatortreiber F übergibt die NVMe-TCPformatierten Header der Anforderung im lokalen Speicher der DPU an den VMA-TCP-Benutzerraum-Stapel G, und die Header werden von F über eine BSD-Sende-API an G weitergeleitet;
  • Operation 57. der NVMe-TCP-Initiatortreiber F übergibt den Inhalt des Puffers B an den VMA-TCP-Benutzerraum-Stapel G, und der Inhalt von Puffer B wird über die erweiterte Sende-API von G an G weitergegeben;
  • Operation 60. der TCP-Benutzerraum-Stapel G verwaltet den TCP-Stapel und die HW-Sendewarteschlange, um den DPU-Speicherstromanteil und den Host-Speicherstromanteil ohne Kopieren des Puffers B oder der Speicherstromanteile an den lokalen PCIe-Funktionsbus zu senden;
  • Operation 70. der NVMe-TCP-Initiatortreiber F erhält vom Ziel eine NVMe-TCP-Antwort auf die gesendete NVMe-TCP-Anfrage;
  • Operation 75. als Reaktion darauf gibt der NVMe-TCP-Initiatortreiber F den Puffer B als den entsprechenden TCP-Stream-Teil im TCP-Benutzerraum-Stapel G frei, der mit einem ACK versehen wurde.
• Ausführungsform 9. System nach einer der vorhergehenden Ausführungsformen, wobei der Speicherschlüssel einen RDMA-Schlüssel umfasst.
• Ausführungsform 10. System nach einer der vorhergehenden Ausführungsformen, wobei der Speicherschlüssel einen DMA-Schlüssel umfasst.
• Ausführungsform 11. Verfahren zur effizienten Interaktion zwischen einem Host und einem entfernten Ziel, wobei das Verfahren umfasst:
  • Bereitstellen eines Hosts mit einem ersten Betriebssystem und eines entfernten Ziels mit einem zweiten Betriebssystem, das eine PCI-Typ-Verbindung mit dem ersten Betriebssystem hat; und
  • gemeinsames Nutzen mindestens eines Puffers, der sich auf dem Host befindet, mit dem entfernten Ziel, einschließlich der Bereitstellung eines Null-Kopie-Zugriffs für das Ziel auf den mindestens einen Puffer, um dadurch eine doppelte Null-Kopie über mehrere Betriebssysteme hinweg bereitzustellen.
• Ausführungsform 12: Verfahren gemäß einer der vorhergehenden Ausführungsformen, bei dem mindestens einer der Hosts und das Ziel eine nichttransparente Brücke (NTB) für die Inter-Domain-Kommunikation über PCIe-Schnittstellen verwenden.
• Ausführungsform 13: Verfahren gemäß einer der vorhergehenden Ausführungsformen, bei dem eine E/A-(bzw. I/O)-Verarbeitungseinheit (IPU) mit einem Array von ARM-Prozessorkernen zwischen dem Host und dem Netzwerk eingesetzt wird, und bei dem SNAP TCP Storage auf den ARM-Prozessorkernen den PCIe-Zugriff vom Host unter Verwendung mindestens eines XGVMI-Speicherschlüssels auf Null kopiert.
• Ausführungsform 14. Verfahren nach einer der vorangehenden Ausführungsformen, bei dem ein erster Kopiervorgang vom Host zur IPU und ein zweiter Kopiervorgang vom CPU-Speicher zu einem TCP-Stapel erfolgt.
• Ausführungsform 15. Verfahren gemäß einer der vorhergehenden Ausführungsformen, wobei das Verfahren die folgenden Operationen umfasst, um dadurch IPOS zu erhöhen und/oder die Latenzzeit zu verringern und/oder die CPU-Auslastung zu verringern, indem die Kopien des Speicherbereitstellungspuffers reduziert werden:
  • Operation 10. eine Anwendung auf einem Client-Host bereitet einen entfernten Datenpuffer B im lokalen Speicher des Hosts vor;
  • Operation 15. die Host-Anwendung fordert einen NVMe-Treiber C auf dem Client-Host auf, einen entfernten Puffer B in ein Speichervolumen zu schreiben;
  • Operation 20. der NVMe-Treiber C auf dem Client-Host bereitet eine NVMe-Anforderung vor, die auf den Puffer B verweist;
  • Operation 25. der NVMe-Treiber C des Hosts sendet die Anforderung, die auf den Puffer B zeigt, an ein über die DPU verwaltetes emuliertes NVMe-Gerät D, und das emulierte NVMe-Gerät D stellt dem Client-Host ein NVMe-Gerät zur Verfügung, das das emulierte NVMe-Gerät D emuliert;
  • Operation 30. das emulierte NVMe-Gerät D teilt einem Manager E des emulierten NVMe-Geräts D auf einer DPU mit, dass eine NVMe-Anfrage bereit ist, und Manager E erhält eine Kopie der NVMe-Anfrage im lokalen Speicher von Manager E;
  • Operation 35. Manager E versteht, dass die NVMe-Anfrage auf Puffer B verweist, der Teil des Client-Host-Speichers ist und daher einen anderen PCIe-Bus oder eine andere Funktion hat;
  • Operation 40. Manager E startet eine neue Anforderung auf einem NVMe-TCP-Initiatortreiber F, der auf der DPU läuft, mit Attributen zur Ausführung dieser neuen Anforderung, und die neue Anforderung oder ihre Attribute umfassen lokale Anforderungsinformationen und den Inhalt des entfernten Datenpuffers B;
  • Operation 50. NVMe-TCP-Initiatortreiber F hält über einen VMA-TCP-Benutzerraum-Stapel G auf der DPU einen TCP-Socket mit einem Ziel verbunden;
  • Operation 53. der NVMe-TCP-Initiatortreiber F übergibt die NVMe-TCPformatierten Header der Anforderung im lokalen Speicher der DPU an den VMA-TCP-Benutzerraum-Stapel G, und die Header können vom NVMe-TCP-Initiatortreiber F über eine BSD send()-API an den VMA-TCP-Benutzerraum-Stapel G übergeben werden;
  • Operation 57. der NVMe-TCP-Initiatortreiber F übergibt den Inhalt von Puffer B an den VMA-TCP-Benutzerraum-Stapel G, und der Inhalt von Puffer B wird über die erweiterte Sende-API vom NVMe-TCP-Initiatortreiber F an den VMA-TCP-Benutzerraum-Stapel G übergeben;
  • Operation 60. der TCP-Benutzerraum-Stapel G verwaltet den TCP-Stapel und die HW-Sendewarteschlange, um den DPU-Speicherstromanteil und den Host-Speicherstromanteil ohne Kopieren des Puffers B oder der Speicherstromanteile an einen lokalen PCIe-Funktionsbus zu senden;
  • Operation 70. der NVMe-TCP-Initiatortreiber F erhält eine NVMe-TCP-Antwort vom Ziel über die gesendete NVMe-TCP-Anfrage; und
  • Operation 75. als Reaktion darauf gibt der NVMe-TCP-Initiatortreiber F den Puffer B frei, da der betreffende TCP-Stream-Teil im VMA-TCP-Benutzerraum-Stapel G mit ACK bestätigt wurde.
• Ausführungsform 16: Verfahren gemäß einer der vorhergehenden Ausführungsformen, bei dem die IPU einen VMA umfasst und bei dem der Netzwerk-Stapel einen Puffer, der Daten speichert, die über das Netzwerk zu senden sind, unter Verwendung eines Speicherschlüssels zusätzlich zu einem Zeiger identifiziert.
• Ausführungsform 17. Verfahren nach einer der vorhergehenden Ausführungsformen, bei dem der Netzwerk-Stapel die über das Netzwerk zu sendenden Daten direkt aus dem Puffer auf dem Host extrahiert, der sich in einer anderen PCIe-Domäne befindet.
• Ausführungsform 18: Verfahren gemäß einer der vorangehenden Ausführungsformen, bei dem das entfernte Ziel über ein Netzwerk mit dem Host verbunden ist, von dem der Host nichts weiß.
• Ausführungsform 19. System gemäß einer der vorangehenden Ausführungsformen, bei dem eine NTB (Non-Transparent Bridge) verwendet wird, um Software auf dem zweiten Betriebssystem den Zugriff auf den lokalen Host-Puffer des ersten Betriebssystems zu ermöglichen.
• Ausführungsform 20. System nach einer der vorhergehenden Ausführungsformen, wobei die NTB (Non-Transparent Bridge) von einem VMA auf dem zweiten Betriebssystem verwendet wird.
• Ausführungsform 21. System gemäß einer der vorhergehenden Ausführungsformen, bei dem der Netzwerk-Stapel so konfiguriert ist, dass er den Puffer unter Verwendung eines Speicherschlüssels und nicht eines Zeigers identifiziert.
• Ausführungsform 22. System gemäß einer der vorhergehenden Ausführungsformen, bei dem ein NVMe-TCP-Initiatortreiber den Inhalt des Puffers an einen TCP-Benutzerraum-Stapel im Netzwerk-Stapel weitergibt.
• Ausführungsform 23. System gemäß einer der vorhergehenden Ausführungsformen, wobei das zweite Betriebssystem mehrere ARM-CPUs enthält.
• Ausführungsform 24. System gemäß einer der vorangegangenen Ausführungsformen, bei dem ARM-basierte NVMe-TCP-Emulationssoftware den Puffer an einen TCP-Socket im VMA weiterleitet.
• Ausführungsform 25. System gemäß einer der vorhergehenden Ausführungsformen, wobei das zweite Betriebssystem die über das Netz zu sendenden Daten direkt aus einem Puffer auf dem Host extrahiert, der sich in einer anderen PCIe-Domäne befindet.
• Ausführungsform 26. System nach einer der vorhergehenden Ausführungsformen, wobei der Netzwerk-Stapel einen VMA umfasst.

A computerized system for efficient interaction between a host is provided, the host having a first operating system and a second operating system, the system including a subsystem on the second operating system that extracts data directly from a buffer local to the host , the system being operable to map memory from one bus associated with the first operating system to another bus associated with the second operating system and from which other bus the memory is accessed to thereby establish a connection between to emulate the first and the second operating system by a memory map across the bus. The scope of the invention includes, inter alia, the following embodiments:

• Embodiment 1. A computerized system for efficient interaction between a host, the host having a first operating system and a second operating system, the system comprising:
  • a subsystem, eg on the second operating system, that extracts data, typically directly from a buffer located on the host, the system being operable to move memory from one bus that may be associated with the first operating system to another Map a bus that may be associated with the second operating system and from which memory is typically accessed, thereby emulating a connection between the first and second operating systems, eg, by cross-bus memory mapping.
• Embodiment 2. The system according to any of the preceding embodiments, wherein the first operating system has PCI-style connectivity, including multiple layers, with the second operating system.
• Embodiment 3. System according to any of the preceding embodiments, wherein the subsystem comprises a network stack.
• Embodiment 4. The system of any preceding embodiment, wherein the network stack is configured to identify a buffer outside of the second operating system's own memory that allows the second operating system's software to access the first operating system's buffer.
• Embodiment 5. System according to any of the preceding embodiments, in which the buffer that is on the host remains on the host.
• Embodiment 6. A system according to any preceding embodiment, the system also including an application on the host that:
  • opens a storage volume on an emulated NVMe so as to provide an open file;
  • allocates a memory block in local memory and writes data to the memory block as buffer B;
  • writes to the open file thereby defining a write call; and provides a kernel driver that processes the write request and prepares an NVMe capsule with buffer B.
• Embodiment 7. The system of any preceding embodiment, wherein the system includes NVMe emulation software and a TCP software stack that work together to enable cross-bus mapping by allowing a cross-device storage key to traverse multiple layers of software to allow software on the second operating system can access the local host buffer of the first operating system.
• Embodiment 8. The system of any preceding embodiment, wherein the first operating system is also configured to control operations to thereby allow the cross-device storage key to traverse the multiple software layers, the operations comprising:
  • Operation 25. The host's NVMe driver C sends a request pointing to the buffer B to a DPU-managed emulated NVMe device D, and the emulated NVMe device D presents the host with an NVMe device (the D emulated) available;
  • operation 30. the emulated NVMe device D notifies a manager E of the emulated NVMe device D on a DPU that an NVMe request is ready and the manager E obtains a copy of the NVMe request in the local storage of the manager E;
  • Operation 35. Manager E understands, e.g. B. by accessing manager E's local storage copy of the NVMe request, that the NVMe request points to buffer B, which as part of the client's host storage has a different PCIe bus or function;
  • Operation 40. Manager E starts a new request to an NVMe TCP initiator driver F running on the DPU with attributes to execute this new request, and the new request or its attributes include local request information and the content of the buffer (B) , which is removed;
  • Operation 50. The NVMe TCP initiator driver F holds a TCP socket connected to a target via a TCP user space stack G on the DPU;
  • Operation 53. The NVMe-TCP initiator driver F passes the NVMe-TCP formatted headers of the request in the DPU's local storage to the VMA-TCP user space stack G, and the headers are forwarded from F to G via a BSD send API ;
  • Operation 57. NVMe TCP initiator driver F commits the contents of buffer B to the VMA TCP user space stack G, and the contents of buffer B are passed from G to G via the extended send API;
  • Operation 60. the TCP user space stack G manages the TCP stack and the HW send queue to send the DPU memory stream share and the host memory stream share to the local PCIe function bus without copying the buffer B or the memory stream shares;
  • Operation 70. The NVMe TCP initiator driver F receives from the target an NVMe TCP Response to the sent NVMe TCP Request;
  • Operation 75. in response, the NVMe TCP initiator driver F releases the buffer B as the corresponding TCP stream part in the TCP user space stack G that has been ACKed.
• Embodiment 9. The system of any preceding embodiment, wherein the storage key comprises an RDMA key.
• Embodiment 10. The system of any preceding embodiment, wherein the storage key comprises a DMA key.
• Embodiment 11. A method for efficient interaction between a host and a remote target, the method comprising:
  • providing a host with a first operating system and a remote target with a second operating system having a PCI-type connection with the first operating system; and
  • sharing at least one buffer residing on the host with the remote target, including providing the target with null copy access to the at least one buffer, thereby providing a duplicate null copy across multiple operating systems.
• Embodiment 12 Method according to any of the preceding embodiments, in which at least one of the hosts and the target uses a non-transparent bridge (NTB) for inter-domain communication over PCIe interfaces.
• Embodiment 13: Method according to one of the preceding embodiments, at which uses an I/O (or I/O) processing unit (IPU) with an array of ARM processor cores between the host and the network, and where SNAP TCP storage on the ARM processor cores enables PCIe access from Host nulled using at least one XGVMI storage key.
• Embodiment 14. The method of any preceding embodiment, wherein a first copy is from host to IPU and a second copy is from CPU memory to a TCP stack.
• Embodiment 15. The method according to any of the preceding embodiments, wherein the method comprises the following operations to thereby increase IPOS and/or reduce latency and/or reduce CPU utilization by reducing the copies of the memory ready buffer:
  • Operation 10. an application on a client host prepares a remote data buffer B in the host's local memory;
  • Operation 15. The host application requests an NVMe driver C on the client host to write a remote buffer B to a storage volume;
  • Operation 20. NVMe driver C on the client host prepares an NVMe request pointing to buffer B;
  • Operation 25. The host's NVMe driver C sends the request pointing to the buffer B to an emulated NVMe device D managed via the DPU, and the emulated NVMe device D presents an NVMe device to the client host Available that emulates the emulated NVMe device D;
  • operation 30. the emulated NVMe device D notifies a manager E of the emulated NVMe device D on a DPU that an NVMe request is ready, and manager E obtains a copy of the NVMe request in manager E's local storage;
  • Operation 35. Manager E understands that the NVMe request references Buffer B, which is part of client host memory and therefore has a different PCIe bus or function;
  • Operation 40. Manager E starts a new request on an NVMe TCP initiator driver F running on the DPU with attributes to execute this new request, and the new request or its attributes include local request information and the content of the remote data buffer B;
  • Operation 50. NVMe TCP initiator driver F holds a TCP socket connected to a target via a VMA TCP user space stack G on the DPU;
  • Operation 53. The NVMe-TCP initiator driver F passes the NVMe-TCP formatted headers of the request in the DPU's local storage to the VMA-TCP user-space stack G, and the headers can be read by the NVMe-TCP initiator driver F via a BSD send( ) API are passed to the VMA TCP user space stack G;
  • Operation 57. The NVMe TCP initiator driver F commits the contents of buffer B to the VMA TCP user space stack G, and the contents of buffer B are sent from the NVMe TCP initiator driver F to the VMA via the extended send API - pass TCP user space stack G;
  • Operation 60. TCP user space stack G manages TCP stack and HW send queue to send DPU memory stream portion and host memory stream portion to a local PCIe function bus without copying buffer B or memory stream portions;
  • Operation 70. The NVMe TCP initiator driver F receives an NVMe TCP Response from the target via the sent NVMe TCP Request; and
  • Operation 75. in response to this, the NVMe TCP initiator driver F releases the buffer B since the relevant TCP stream part in the VMA TCP user space stack G has been ACKed.
• Embodiment 16: Method according to any of the preceding embodiments, in which the IPU comprises a VMA and in which the network stack identifies a buffer storing data to be sent over the network using a storage key in addition to a pointer.
• Embodiment 17. The method of any preceding embodiment, wherein the network stack extracts the data to be sent over the network directly from the buffer on the host located in a different PCIe domain.
• Embodiment 18: A method according to any one of the preceding embodiments, wherein the remote target is connected to the host via a network of which the host is unaware.
• Embodiment 19. System according to one of the preceding embodiments, in which an NTB (Non-Transparent Bridge) is used to transfer software on the second operating sys tem to allow access to the local host buffer of the first operating system.
• Embodiment 20. The system of any preceding embodiment, wherein the NTB (Non-Transparent Bridge) is used by a VMA on the second operating system.
• Embodiment 21. The system of any preceding embodiment, wherein the network stack is configured to identify the buffer using a storage key rather than a pointer.
• Embodiment 22. The system of any preceding embodiment, wherein an NVMe TCP initiator driver passes the contents of the buffer to a TCP user space stack in the network stack.
• Embodiment 23. The system according to any preceding embodiment, wherein the second operating system includes multiple ARM CPUs.
• Embodiment 24. System according to any of the preceding embodiments, wherein ARM-based NVMe TCP emulation software forwards the buffer to a TCP socket in the VMA.
• Embodiment 25. The system of any preceding embodiment, wherein the second operating system extracts the data to be sent over the network directly from a buffer on the host located in a different PCIe domain.
• Embodiment 26. The system of any preceding embodiment, wherein the network stack comprises a VMA.

Each feature of one aspect or embodiment may be applied to other aspects or embodiments, in any suitable combination. In particular, any feature of a method aspect or embodiment may be applied to an apparatus aspect or embodiment and vice versa.

Aspects and embodiments of the disclosure will now be described, by way of example only, with reference to the accompanying drawings.

character list

• 1 Figure 12 is a simplified block diagram representation of a system for memory mapping across buses according to an embodiment of the invention.
• the 2 - 5 are tables that can be used to create storage keys.

DETAILED DESCRIPTION OF EMBODIMENTS

The term "memory key" (mkey) as used herein is intended to provide a context with a pointer and additional attributes, e.g. the pointer, describing at least one of the following characteristics: how the memory pointer can be accessed (e.g. read, write, atomic, all), what entity or entities can access the pointer (e.g. protection key), what translations need to be made to enable a virtual to physical figure, and GVMI-wide information, e.g. B. as described here.

It is well known that pointers are known in computer programming, such as For example, as described in the Wikipedia entry about "Pointer_(computer_programming)", CPUs (and other peripherals) can process data when it is in registers or pointed to by a pointer in memory space. Storage keys are described e.g. in Mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_User_Manual_v4_3.pdf.

Mkeys and their creation are described, for example, in Mellanox.com/relateddocs/user_manuals/Ethernet_Adapters_Programming_Manual.pdf. For example, the CREATE_MKEY command described there creates new mkeys. The context of the new mkey object can be taken from an input mailbox. Normally the hardware supplies a mkey index as an output parameter. This index can be used by the software as an opener when accessing this object.

VHCA is a Virtual Host Channel Adapter. VHCAs and VHCA-IDs are described, inter alia, in the US patent US9397960B2 described, the disclosure of which is hereby incorporated by reference. Virtual host channel adapters with multiple contexts and single logic are described in this Cisco patent: https://patents.google.com/patent/ US20080123672 , the disclosure of which is hereby incorporated by reference. It is known that there are multiple Virtual Functions (VFs), each presenting itself to a host as an independent VHCA, while only a single HCA can be observed by the network, which is unaware of the VHCAs. Methods for sharing the same physical port among multiple VHCAs are known and are described, for example, in download.lenovo.com/server s/mig/2020/05/28/21936/ mlnx-lnvgy_dd_nic_cx.ib-5.0-2.1.8.0-0_rhel7_x86-64.pdf on page 284.

The term "GVMI" aka Guest Virtual Machine Identifier (GVMI) usually denotes a hardware structure; the GVMI is typically the name of a guest operating system for a virtual machine. An example of a GVMI is a vhca_id aka "VHCA ID" or VHCA identifier.

The term "Cross-GVMI-Mkey" refers to a hardware structure that e.g. B. in Connect ^® X6Dx exists and is described under the following www link: nvidia.com/enus/networking/ethernetlMellano-6-dx. The terms XGVMI, cross-GVMI, and cross-domain GVMI are generally used interchangeably herein. Network adapters like the Connect ^® X6Dx are known to have HW parts that support a cross-GVMI mkey and work in conjunction with the firmware running on the internal cores. This firmware typically has a proprietary command interface that is documented in the Programmer's Reference Manual (PRM) available here: Mellanox.com/related-docs/user_manuals/Ethernet_Adapters_Programming_Manual.pdf.

However, it is appreciated that any suitable Ethernet network adapter driver that supports a cross-GVMI mkey as described can be used. This firmware's code can take a command and map that command to HW configurations. This is just an example implementation that provides flexibility in terms of keeping the command even when a new hardware version comes out. In device APIs implemented by the above firmware, this can be represented as: "CREATE_MKEY with access_mode = 0x6:CROSSING_VHCA_MKEY"

• eine Client-Anwendung, auch bekannt als „Performance-Anwendung“ oder Perf, auf einem Host, auch bekannt als Client-Host. Es versteht sich von selbst, dass Software, die sich auf dem Host-Betriebssystem in 1 befindet, typischerweise die im Folgenden beschriebenen Vorgänge 10-75 steuert;
• eine DPU (z. B. BlueField^® )
• einen NVMe-Treiber auf dem Client-Host (auch bekannt als der NVMe-Treiber C des Hosts);
• NVMe-TCP-Initiatortreiber F, der auf dem DPU läuft;
• VMA TCP Benutzerraum-Stapel, G, auf der DPU;
• eine VHCA;
• GVMIs A und B, die in der Regel in Hardware implementiert sind, von denen eine oder beide durch VHCAs ersetzt werden können; GVMI A ist in der Regel als Netzwerkgerät definiert. GVMI B umfasst in der Regel ein emuliertes NVMe-Gerät (z. B. PCIe), das dem Client-Host ein NVMe-Gerät zur Verfügung stellt und über die DPU verwaltet wird;
• Software auf dem zweiten Betriebssystem (z. B. BlueField^® ), um auf den lokalen Puffer des Hosts vom ersten Betriebssystem zuzugreifen;
• TCP-Socket im VMA; und
• NVMe-TCP-Emulationssoftware, die einen Puffer B an den VMA-TCP-Socket übergibt, der auf oder als SNAP^® implementiert werden kann.

1 Figure 12 shows a system for bus-to-bus memory mapping that typically facilitates NVMe memory emulation via NVMe TCP acceleration according to an embodiment of the invention. The system of 1 typically includes all or an appropriate subset of the following components:

• a client application, also known as a "performance application" or perf, on a host, also known as a client host. It goes without saying that software located on the host operating system in 1 typically controls operations 10-75 described below;
• a DPU (e.g. BlueField ^® )
• an NVMe driver on the client host (aka the host's NVMe driver C);
• NVMe TCP initiator driver F running on the DPU;
• VMA TCP user space stack, G, on the DPU;
• a VHCA;
• GVMIs A and B, typically implemented in hardware, one or both of which can be replaced by VHCAs; GVMI A is typically defined as a network device. GVMI B typically includes an emulated NVMe device (e.g. PCIe) that presents an NVMe device to the client host and is managed through the DPU;
• software on the second operating system (e.g., ^BlueField® ) to access the host's local buffer from the first operating system;
• TCP socket in VMA; and
• NVMe TCP emulation software that passes a buffer B to the VMA TCP socket that can be implemented on or as SNAP ^® .

A cross-GVMI storage key, also called a cross-GVMI mkey, is configured to take a buffer from one GVMI, e.g. GVMI A, and use the same buffer on the other side with the second GVMI, e.g. GVMI B.

• eine softwareprogrammierbare Mehrkern-CPU, die eine ARM-Architektur aufweisen kann und in der Regel mit anderen SoC-Komponenten gekoppelt ist, und/oder
• eine Hochleistungsnetzschnittstelle zum Parsen, Verarbeiten und Übertragen von Daten, z. B. an GPUs und/oder CPUs, mit der Geschwindigkeit des übrigen Netzes (auch Leitungsrate genannt); und/oder
• programmierbare Beschleunigungs-Engines, die Anwendungen beispielsweise für KI und/oder maschinelles Lernen und/oder Sicherheit und/oder Telekommunikation und/oder Speicherung entlasten und dadurch deren Leistung verbessern.

A DPU is a possible example of a data processor, typically a system on a chip, also called a SoC, which typically includes:

• a software-programmable multi-core CPU, which may have an ARM architecture and is typically coupled with other SoC components, and/or
• a high-performance network interface for parsing, processing and transmitting data, e.g. to GPUs and/or CPUs, at the speed of the rest of the network (also called line rate); and or
• programmable acceleration engines that offload applications such as AI and/or machine learning and/or security and/or telecom and/or storage thereby improving their performance.

It is appreciated that certain embodiments herein are advantageous in reducing latency and CPU usage caused by the copy overhead of the NVMe payload from host memory (e.g., X86) to a TCP packet payload , created by the NVMe emulation soft ware of the DPU CPU cores is managed (e.g. SNAP ^® ).

Typically, NVMe emulation software (such as SNAP ^® ) configures the network interface card (NIC) hardware to send one or more NVMe TCP packets so that the TCP header is retrieved from local DPU memory , while fetching the TCP payload directly from host memory (X86) on a remote PCIe bus. This avoids or skips up to two copy and staging buffers.

A particular advantage of certain embodiments is that the network interface card (NIC) hardware typically collects a single packet from multiple memory addresses spread across multiple PCIe buses. Software is used to translate and/or map and/or manage these memory addresses from their respective PCIe buses and hardware is configured accordingly to send such packets.

The BlueField ^® -2 I/O Processing Unit (IPU) is a System-on-Chip (SoC) that includes a SmartNIC, e.g. B. ConnectX ^® -6 Dx, a range of ARM processor cores, high-performance memory interfaces and flexible processing capabilities. The SmartNIC can hardware offload engines with IPsec and inline TLS data-in-motion cryptography and/or network virtualization and/or RDMA over Converged Ethernet (RoCE) and/or NVMe over Fabrics (NVMe-oF) storage accelerators included, such as B. described online in: mellanox.com/news/press_release/mellanox-introducesrevolutionary-connectx-6-dx-and-bluefield-2-secure-cloud-smartnics-and-io. An ARM processor is a CPU based on the Advanced RISC Machines (ARM) RISC (Reduced Instruction Set Computer) architecture, e.g. B. as described online in: whatis.techtarget.com/definition/ARMprocessor#:~:text=An%20ARM%20processor%20is%20one,bit%20RISC%20multi%2Dcore %20processors.

The system off 1 may be used in conjunction with known technologies such as, without limitation, SNAP ^® and/or VMA and/or TCP Sockets and/or libibverbs and/or BlueField ^® . The second operating system in 1 may, for example, contain multiple ARM CPUs as described here, or a set of ARM processor cores (such as an NVIDIA ^® BlueField ^® IPU aka BlueField ^® (ARM)), or alternatively it may not be ARM-based at all and instead may be, for example, X86 or use another suitable CPU architecture. Mellanox NVMe SNAP ^® (Software-defined Network Accelerated Processing) is an example of a framework or technology that enables hardware-accelerated virtualization of NVMe storage, e.g. B. by emulating a block storage device on a host. For example, an NVMe drive can be emulated on a PCIe bus, making network storage look like a local NVMe SSD. A host OS/hypervisor can use its standard NVMe driver unaware that the communication is terminated by the NVMe SNAP ^® and not a physical drive. The logic can then be applied to the data via the NVMe SNAP ^® and transmitted over the network to a storage destination, e.g. B. via an Ethernet or InfiniBand protocol. Alternatively, any other suitable technology may be employed that implements hardware memory virtualization and/or enables optimization of memory resources in terms of CAPEX and OPEX savings and/or is configured to free up CPU resources of the compute node by swapping out both network as well as storage data paths are freed up, and preferably allows integration into any storage solution based on any network protocol.

Regarding [copy #2], it is estimated that this online link: Systemkern.org/doc/html/v4.15/networking/msg_zerocopy.html provides a Linux socket definition of the zerocopy flags. TCP Stack Implementation (VMA) can add support for broadcasts with MSG_ZEROCOPY flag. The implementation of the send() call with the MSG_ZEROCOPY flag can use the application buffer pointer as a reference without copying it to an intermediate buffer. The TCP stack can maintain the reference buffer pointer for sending the TCP packet and must not release the reference to the buffer until a TCP ACK covering the entire send buffer size has been received. The TCP socket owner (SPDK initiator) can call send() or a send with MSG_ZEROCOPY if supported by the TCP stack (VMA) implementation. The caller's implementation of send() with MSG_ZEROCOPY can set aside the sent buffer pending an indication that the TCP stack has completed the send and received an acknowledgment.

To [copy #1], which may include a zerocopy SNAP ^® (e.g. Host -> BlueField ^® ), an API can be added to the TCP stack (e.g. VMA) to enable the Provide cross-GVMI-Mkey so that VMA later recognize a marked physical address (aka "PA") as host storage and use this cross-domain GMVI storage key for those in HW collected PCIe can use. ^SNAP® preparing the cross-GVMI mkey can pass the cross-GVMI mkey to VMA at TCP socket initialization. The signaling of the TCP buffer as “external host memory”, for which the cross-GVMI mkey between ^SNAP® and (e.g.) VMA is required, can be done by appropriate marking of the external host buffer pointer. For example, the MSB can be set according to the physical address. Logic can be added in (for example) VMA to recognize the marked PA as an external host memory buffer and can use the cross GVMI VMA to pass the mkey) to the hardware with data buffer SGE on raw QP (SQ /TIS) with the cross GVMI mkey.

For simplicity, this specification refers to SNAP™ as an example. However, such references are not intended to be limiting and, in general, any storage device emulator (e.g., a smartNIC service) that emulates an NVMe (or virtio-blk) storage PCIe device for the host may be used by providing a Shares the storage PCIe device interface with a host (or allows the host to access the storage PCIe device).

1. A. eine Anwendung auf einem Client-Host (auch bekannt als Host-Anwendung A)
2. B. einen Datenpuffer im Speicher des Client-Hosts (auch Puffer B genannt)
3. C. einen NVMe-Treiber auf dem Client-Host (auch bekannt als NVMe-Treiber C des Hosts)
4. D. ein emuliertes NVMe-Gerät (stellt das NVMe-Gerät dem Client-Host zur Verfügung und wird über die DPU verwaltet)
5. E. NVMe-emulierter Gerätemanager auf DPU, z. B. SNAP^®
6. F. NVMe-TCP-Initiatortreiber F, der z. B. auf der DPU läuft
7. G. einen VMA (beispielsweise) TCP Benutzerraum-Stapel, G, der sich auf der DPU befinden kann.

The following procedure is either alone or in conjunction with the system of 1 or more generally provided with all or any subset of the following procedures:

1. A. an application on a client host (aka host application A)
2. B. a data buffer in client host memory (also called Buffer B)
3. C. An NVMe driver on the client host (aka the host's NVMe driver C)
4. D. An emulated NVMe device (makes the NVMe device available to the client host and managed by the DPU)
5. E. NVMe emulated device manager on DPU, e.g. B. ^SNAP®
6. F. NVMe TCP initiator driver F that supports e.g. B. running on the DPU
7. G. a VMA (e.g.) TCP user space stack, G which may reside on the DPU.

• Operation 10. eine Anwendung auf einem Client-Host bereitet einen entfernten Datenpuffer B im lokalen Speicher des Hosts vor;
• Operation 15. die Host-Anwendung fordert einen NVMe-Treiber C auf dem Client-Host auf, einen entfernten Puffer B in ein Speichervolumen zu schreiben;
• Operation 20. der NVMe-Treiber C des Hosts bereitet eine NVMe-Anforderung vor, die auf den Puffer B verweist;
• Operation 25. der NVMe-Treiber C des Hosts sendet die auf den Puffer B zeigende Anforderung an ein über die DPU verwaltetes emuliertes NVMe-Gerät D, und das emulierte NVMe-Gerät D stellt dem Client-Host ein NVMe-Gerät zur Verfügung, das das emulierte NVMe-Gerät D emuliert;
• Operation 30. das emulierte NVMe-Gerät D teilt einem Manager E des emulierten NVMe-Geräts D auf einer DPU mit, dass eine NVMe-Anforderung bereit ist, und Manager E erhält eine Kopie der NVMe-Anforderung im lokalen Speicher von Manager E;
• Operation 35. Manager E versteht, z. B. durch Zugriff auf die Kopie der NVMe-Anfrage im lokalen Speicher von Manager E, dass die NVMe-Anfrage auf Puffer B verweist, der Teil des Hostspeichers des Clients ist und daher einen anderen PCIe-Bus oder eine andere Funktion hat;
• Operation 40. Manager E startet eine neue Anforderung an einen NVMe-TCP-Initiatortreiber F, der auf der DPU läuft, mit Attributen zur Ausführung dieser neuen Anforderung, und die neue Anforderung oder ihre Attribute umfassen lokale Anforderungsinformationen und den Inhalt des Ferndatenpuffers B;
• Operation 50. der NVMe-TCP-Initiatortreiber F hält über einen VMA-TCP-Benutzerraum-Stapel G auf der DPU einen TCP-Socket mit einem Ziel verbunden;
• Operation 53. der NVMe-TCP-Initiatortreiber F übergibt die NVMe-TCPformatierten Header der Anforderung im lokalen Speicher der DPU an den VMA-TCP-Benutzerspace-Stapel G, und die Header können vom NVMe-TCP-Initiatortreiber F über eine BSD send()-API an den VMA-TCP-Benutzerspace-Stapel G übergeben werden;
• Operation 57. der NVMe-TCP-Initiatortreiber F übergibt den Inhalt des Puffers B an den VMA-TCP-Benutzerraum-Stapel G. Der Inhalt des Puffers B kann vom NVMe-TCP-Initiatortreiber F über die erweiterte Sende-API an den VMA-TCP-Benutzerraum-Stapel G übergeben werden; dadurch wird der VMA in die Lage versetzt, den Puffer zu identifizieren, indem er einen Speicherschlüssel anstelle eines Zeigers verwendet.
• Operation 60. der VMA-TCP-Benutzerraum-Stapel G verwaltet den TCP-Stapel und die HW-Sendewarteschlange, um den DPU-Speicherstromanteil und den Host-Speicherstromanteil ohne Kopieren des Puffers B oder der Speicherstromanteile an einen lokalen PCIe-Funktionsbus zu senden;
• Operation 70. der NVMe-TCP-Initiatortreiber F erhält eine NVMe-TCP-Antwort vom Ziel über die gesendete NVMe-TCP-Anfrage; und
• Operation 75. als Reaktion darauf, gibt der NVMe-TCP-Initiatortreiber F den Puffer B als den entsprechenden TCP-Stream-Teil im VMA-TCP-Benutzerraum-Stapel G frei, der mit einem ACK versehen wurde.

The method typically includes all or any subset of the following operations in an appropriate order, e.g. B. as follows:

• Operation 10. an application on a client host prepares a remote data buffer B in the host's local memory;
• Operation 15. The host application requests an NVMe driver C on the client host to write a remote buffer B to a storage volume;
• Operation 20. The host's NVMe driver C prepares an NVMe request that points to buffer B;
• Operation 25. The host's NVMe driver C sends the request pointing to the buffer B to an emulated NVMe device D managed via the DPU, and the emulated NVMe device D provides the client host with an NVMe device that the emulated NVMe device D emulates;
• Operation 30. The emulated NVMe device D notifies a manager E of the emulated NVMe device D on a DPU that an NVMe request is ready, and manager E obtains a copy of the NVMe request in manager E's local storage;
• Operation 35. Manager E understands, e.g. B. by accessing manager E's local memory copy of the NVMe request, that the NVMe request points to buffer B, which is part of the client's host memory and therefore has a different PCIe bus or function;
• Operation 40. Manager E starts a new request to an NVMe TCP initiator driver F running on the DPU with attributes to execute this new request, and the new request or its attributes include local request information and remote data buffer B content;
• Operation 50. The NVMe TCP initiator driver F holds a TCP socket connected to a target via a VMA TCP user space stack G on the DPU;
• Operation 53. The NVMe-TCP initiator driver F passes the NVMe-TCP formatted headers of the request in the DPU's local storage to the VMA-TCP userspace stack G, and the headers can be read by the NVMe-TCP initiator driver F via a BSD send( ) API are passed to the VMA TCP user space stack G;
• Operation 57. The NVMe TCP initiator driver F commits the content of buffer B to the VMA TCP user space stack G. The content of buffer B can be sent from the NVMe TCP initiator driver F to the VMA via the extended send API TCP user space stack G are passed; this enables the VMA to identify the buffer using a storage key instead of a pointer.
• Operation 60. the VMA TCP user space stack G manages the TCP stack and the HW send queue to send the DPU memory stream share and the host memory stream share to a local PCIe function bus without copying the buffer B or the memory stream shares;
• Operation 70. The NVMe TCP initiator driver F receives an NVMe TCP Response from the target via the sent NVMe TCP Request; and
• Operation 75. in response, the NVMe TCP initiator driver F releases the buffer B as the corresponding TCP stream part in the VMA TCP user space stack G that has been ACKed.

• Das hier beschriebene System und die Methoden können in jedes geeignete System wie z. B. SNAP, NVIDIA Mellanox Rivermax^® oder VMA integriert werden. Es ist jedoch klar, dass die Verwendung von VMA nur eine mögliche Ausführungsform ist. VMA ist eine mögliche TCP/IP-Stapel-Lösung, aber alternativ kann bei einer Nicht-VMA-Lösung ein TCP/IP-Stapel bereitgestellt werden, der z. B. wie hier beschrieben modifiziert wird, um den Cross-GVMI-Mkey zu handhaben. Es wird davon ausgegangen, dass VMA (oder Mellanox Messaging Accelerator) ein Beispiel für einen Netzwerk-Stapel ist; andere Beispiele umfassen, sind aber nicht beschränkt auf Linux TCP Stapel, mTCP, Seastar und andere Netzwerk-Stapels, die online beschrieben sind, z. B. auf der DPDK-Ökosystemseite, die online unter dem folgenden Link verfügbar ist: dpdk.org/ecosystem/.

In comparison to the embodiments described here only as examples, many variants are possible. To name just a few examples:

• The system and methods described here can be integrated into any suitable system such as e.g. B. SNAP, NVIDIA Mellanox Rivermax ^® or VMA can be integrated. However, it is clear that the use of VMA is only one possible embodiment. VMA is one possible TCP/IP stack solution, but alternatively, in a non-VMA solution, a TCP/IP stack can be provided, e.g. B. is modified as described here to handle the cross-GVMI Mkey. It is assumed that VMA (or Mellanox Messaging Accelerator) is an example of a network stack; other examples include but are not limited to Linux TCP stacks, mTCP, Seastar, and other network stacks described online, e.g. B. the DPDK ecosystem page, available online at the following link: dpdk.org/ecosystem/.

Additionally, this invention is not specific to PCI connectors (e.g., specifically PCI or PCIe connectors) and can be extended to busses other than PCI and PCIe.

Any other suitable method of creating storage keys can also be used. the 2 , 3 , 4 and 5 For example, Tables 226 through 229 in mellanox.com/related-docs/user_manuals/Ethernet_Adapters_Programming_Manual.pdf represent an example of the layout of the CREATE_MKEY entry structure ( 2 ), an example of the field descriptions of the CREATE_MKEY inbound structure ( 3 ), an example of the layout of the CREATE_MKEY initial structure ( 4 ) and an example for the field descriptions of the CREATE_MKEY initial structure ( 5 )

A method based on NTB (Non-Transparent Bridge) can also replace the use of a cross-domain GVMI storage key, also known as "cross GVMI Mkey", as described here, e.g. B. by providing software configured so that VMA can use NTB (as in the cross GVMI Mkey solution) so that software on the second operating system (e.g. BlueField ^® ) can access the local host buffer of the first operating system can access.

• Schritt a. Öffne das Speichervolumen (z. B.: open(„path“)) auf dem emulierten NVMe; und/oder
• Schritt b. Zuweisung von lokalem Speicher (z. B.: malloc()) und Schreiben der Daten in den zugewiesenen Speicherblock [z. B. wie hier unter Bezugnahme auf Operation 10 beschrieben] und/oder
• Schritt c. Schreibe() in die geöffnete Datei [z. B. wie hier unter Bezugnahme auf Operation 15 beschrieben]; und/oder
• Stufe d. Ein Systemkern-Treiber kann den Schreibe()-Aufruf verarbeiten und eine NVMe-Kapsel mit Puffer B vorbereiten, typischerweise ab Stufe b, z. B. wie hier unter Bezugnahme auf Operation 20 beschrieben.

Also, as described, in certain embodiments, the VMA extracts the data to be sent over the network directly from the buffer on the host located in a different PCI or PCIe domain. Any suitable implementation can be used to cause the VMA to do this. In a storage-aware solution, the application on the host can perform all or any appropriate subset of the following tasks:

• step a. Open the storage volume (e.g.: open("path")) on the emulated NVMe; and or
• step b. Allocating local memory (e.g.: malloc()) and writing the data to the allocated memory block [e.g. as described herein with reference to Operation 10] and/or
• step c. Write() to the opened file [e.g. as described herein with reference to Operation 15]; and or
• level d. A kernel driver can process the write() call and prepare an NVMe capsule with buffer B, typically from level b, e.g. B. as described herein with reference to operation 20.

The term "non-transparent" used here is in contrast to connectivity, which is "transparent". For example, a PCIe NTB (Non-Transparent Bridge) can be used to establish connectivity between multiple CPUs, GPUs and FPGAs. When using a (transparent) PCIe bridge, all PCIe buses are visible to all endpoints for a given root complex. In contrast, a non-transparent bridge routes PCIe traffic between separate PCIe buses, and each Root Complex, RC, sees the NTB (Non-Transparent Bridge) as the endpoint device, while the Root Complex and devices on the other side of the bridge are not visible to the RC, e.g. B. described at the following online address ben: missinglinkelectronics.com/devzone/index.php/fpga-debug-ntb.

VMA (Voltaire Messaging Accelerator) is an example of data acceleration technology. NVIDIA's VMA, released as open source, boosts the performance of message-based and streaming applications across a variety of industries, including: B. High-Frequency Trading (HFT) platforms and Web 2.0 clusters. VMA provides standard socket APIs with kernel bypass architecture, enabling user space networking for multicast, UDP unicast and TCP streaming. VMA can be used immediately, e.g. B. thanks to built-in preconfigured profiles for z. B. Latency and streaming. VMA is described at the following online locations: docs.mellanox.com/category/vma and docs.mellanox.com/display/VMAv911/Introduction+to +VMA; it is understood that other acceleration technologies may include only a subset of the features, properties and functionality of VMA.

The disclosures of all publications and patent documents mentioned in the specification, as well as publications and patent documents directly or indirectly cited therein, are hereby incorporated by reference, save for any disclaimer or disclaimer as to subject matter.

It should be understood that the software components of the present invention may be implemented in ROM (read only memory) form if desired. The software components can generally be implemented in firmware or hardware, if desired, using conventional techniques. In addition, the software components can be instantiated, for example, as a computer program product or on a tangible medium. In some cases it may be possible to instantiate the software components as a signal interpretable by an appropriate computer, although such instantiation may be precluded in certain embodiments of the present invention.

It goes without saying that various features of the invention, which are described in connection with individual embodiments for reasons of clarity, can also be provided in combination in a single embodiment. Conversely, various features of the invention that are described in the context of a single embodiment for the sake of brevity may also be provided separately or in any suitable sub-combination.

Those skilled in the art will appreciate that the present invention is not limited to what has been shown and described above. Rather, the scope of the invention is defined by the appended claims and their equivalents.

It is understood that the aspects and embodiments described above are exemplary only and that changes may be made in detail within the scope of the claims.

Each apparatus, method and feature disclosed in the description and (where appropriate) claims and drawings may be provided independently or in any suitable combination.

The reference numerals contained in the claims are for illustration only and have no limiting effect on the scope of the claims.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• US 9397960 B2 [0027]
• US20080123672 [0027]

Claims (26)

A computerized system for efficient interaction between a host, the host having a first operating system and a second operating system, the system comprising: a subsystem on the second operating system that extracts data directly from a buffer located locally on the host, the system being operable to map memory from one bus associated with the first operating system to another bus that associated with the second operating system and from which the memory is accessed, thereby emulating a connection between the first and second operating systems through a cross-bus memory map. system after claim 1 wherein the first operating system has PCI-style connectivity, including multiple tiers, with the second operating system. system after claim 2 , in which the second operating system extracts data to be sent over a network directly from a buffer on the host located in a different PCIe domain. system after claim 1 , 2 or 3 , wherein the subsystem comprises a network stack. system after claim 4 , wherein the network stack comprises a VMA. system after claim 4 or 5 wherein the network stack is configured to identify a buffer outside of the second operating system's own memory that allows the second operating system's software to access the first operating system's buffer. system after claim 6 , where the network stack is configured to identify the buffer using a storage key rather than a pointer. system after claim 7 , where an NVMe TCP initiator driver passes the contents of the buffer to a TCP user space stack in the network stack. system after claim 8 , with the second operating system containing multiple ARM CPUs. system after claim 9 , where ARM-based NVMe TCP emulation software forwards the buffer to a TCP socket in a VMA. A system as claimed in any preceding claim, wherein the buffer local to the host resides on the host. A system as claimed in any preceding claim, wherein the system also includes an application on the host which: opens a storage volume on an emulated NVMe so as to provide an open file; allocates a memory block in local memory and writes data to the memory block as buffer B; writes to the open file thereby defining a write call; and provides a kernel driver that handles the write request and prepares an NVMe capsule with buffer B. The system of any preceding claim, wherein the system includes NVMe emulation software and a TCP software stack that work together to enable cross-bus mapping by traversing a cross-device storage key through multiple layers of software to thereby allow software on the second operating system to access to the local host buffer from the first operating system. system after Claim 13 , wherein the first operating system is also configured to control operations to thereby allow the device-spanning storage key to traverse the multiple layers of software, the operations comprising: Operation 25. The host's NVMe driver C sends a request that points to the buffer B, to an emulated NVMe device D managed via the DPU, and the emulated NVMe device D provides an NVMe device (which emulates D) to the host; operation 30. the emulated NVMe device D notifies a manager E of the emulated NVMe device D on a DPU that an NVMe request is ready and the manager E obtains a copy of the NVMe request in the local storage of the manager E; Operation 35. Manager E understands, e.g. B. by accessing manager E's local storage copy of the NVMe request, that the NVMe request points to buffer B, which as part of the client's host storage has a different PCIe bus or function; Operation 40. Manager E starts a new request to an NVMe TCP initiator driver F running on the DPU with attributes to execute this new request, and the new request or its attributes include local request information and the content of the buffer (B) , which is removed; Operation 50. The NVMe TCP initiator driver F holds a TCP socket connected to a target via a TCP user space stack G on the DPU; Operation 53. The NVMe-TCP initiator driver F passes the NVMe-TCP formatted headers of the request in the DPU's local storage to the VMA-TCP user space stack G, and the headers are forwarded from F to G via a BSD send API ; Operation 57. NVMe TCP initiator driver F commits the contents of buffer B to the VMA TCP user space stack G, and the contents of buffer B are forwarded from G to G via the extended send API; Operation 60. the TCP user space stack G manages the TCP stack and the HW send queue to send the DPU memory stream share and the host memory stream share to the local PCIe function bus without copying the buffer B or the memory stream shares; Operation 70. The NVMe TCP initiator driver F receives from the target an NVMe TCP Response to the sent NVMe TCP Request; Operation 75. in response, the NVMe TCP initiator driver F releases the buffer B as the relevant TCP stream part in the TCP user space stack G that has been ACKed. system after Claim 13 or 14 , wherein the storage key comprises an RDMA key. system after Claim 13 or 14 , wherein the storage key comprises a DMA key. A system as claimed in any preceding claim, wherein an NTB (Non-Transparent Bridge) is used to allow software on the second operating system to access the local host buffer from the first operating system. system after Claim 17 , where the NTB (Non-Transparent Bridge) is used by a VMA on the second operating system. A method for efficient interaction between a host and a remote target, the method comprising: providing a host with a first operating system and a remote target with a second operating system having a PCI-type connection with the first operating system; and sharing at least one buffer residing on the host with the remote target, including providing the target with null copy access to the at least one buffer, thereby providing a duplicate null copy across multiple operating systems. procedure after claim 19 , where at least one of the host and target uses a non-transparent bridge (NTB) for inter-domain communication over PCIe interfaces. procedure after claim 19 or 20 , where an I/O processing unit (IPU) with a set of ARM processor cores is deployed between the host and a network, and where SNAP TCP memory on the ARM processor cores enables PCIe access from the host using at least one XGVMI memory key zero copied. procedure after Claim 21 , where a first copy is from host to IPU, and a second copy is from CPU memory to a TCP stack. procedure after Claim 22 , the method comprising the following operations to thereby increase IPOS and/or reduce latency and/or reduce CPU utilization by reducing copies of the memory provisioning buffer: operation 10. prepares an application on a client host provides a remote data buffer B in the host's local memory; Operation 15. The host application requests an NVMe driver C on the client host to write a remote buffer B to a storage volume; Operation 20. NVMe driver C on the client host prepares an NVMe request pointing to buffer B; Operation 25. The host's NVMe driver C sends the request pointing to the buffer B to an emulated NVMe device D managed via the DPU, and the emulated NVMe device D presents an NVMe to the client host -Device available that emulates the emulated NVMe device D; operation 30. the emulated NVMe device D notifies a manager E of the emulated NVMe device D on a DPU that an NVMe request is ready, and manager E obtains a copy of the NVMe request in manager E's local storage; Operation 35. Manager E understands that the NVMe request references Buffer B, which is part of client host memory and therefore has a different PCIe bus or function; Operation 40. Manager E starts a new request to an NVMe TCP initiator driver F running on the DPU with attributes to execute this new request, and the new request or its attributes include local request information and remote data buffer B content; Operation 50. NVMe TCP initiator driver F holds a TCP socket connected to a target via a VMA TCP user space stack G on the DPU; Operation 53. The NVMe-TCP initiator driver F passes the NVMe-TCP formatted headers of the request in the DPU's local storage to the VMA-TCP user-space stack G, and the headers can be read by the NVMe-TCP initiator driver F via a BSD send( ) API are passed to the VMA TCP user space stack G; Operation 57. The NVMe TCP initiator driver F commits the contents of buffer B to the VMA TCP user space stack G, and the contents of buffer B are sent from the NVMe TCP initiator driver F to the VMA via the extended send API - pass TCP user space stack G; Operation 60. TCP user space stack G manages TCP stack and HW send queue to send DPU memory stream portion and host memory stream portion to a local PCIe function bus without copying buffer B or memory stream portions; Operation 70. The NVMe TCP initiator driver F receives an NVMe TCP response from the target to the NVMe TCP request that was sent; and operation 75. in response, the NVMe TCP initiator driver F releases the buffer B since the relevant TCP stream part in the VMA TCP user space stack G has been ACKed. Procedure according to one of Claims 21 until 23 wherein the IPU comprises a VMA and wherein a network stack identifies a buffer storing data to be sent over the network using a storage key in addition to a pointer. procedure after Claim 24 , where the network stack extracts the data to be sent over the network directly from the buffer on the host, which is in a different PCIe domain. Procedure according to one of claims 19 until 25 , where the remote target is connected to the host over a network unknown to the host.

Images