CN118312337A - Remote memory access method suitable for multi-tenant data center - Google Patents

Abstract

The invention belongs to a data center network, and discloses a remote memory access method suitable for a multi-tenant data center, which comprises the steps that a client computer acquires necessary information of a server computer for accessing a relevant remote memory area by executing out-of-band remote procedure call; the client computer generates a client command and sends the client command to the server computer; the server side computer performs data access on the target memory based on the client side command, and feeds back the command completion condition to the client side computer after the data access; and the client computer judges whether fault recovery and congestion control are needed according to the received command completion condition. The invention can ensure that the high priority application is not affected by the low priority application, can provide predictable delay even under high load and fault conditions, can almost immediately converge to fair bandwidth sharing under the condition of competing applications, and shortens the unavailable time during encryption key rotation.

Description

Remote memory access method suitable for multi-tenant data center

Technical Field

The invention belongs to a data center network, and particularly relates to a remote memory access method suitable for a multi-tenant data center.

Background

Modern data center applications such as search, machine learning, etc. require the network to support high bandwidth and operating rates while achieving low tail latency. Remote Direct Memory Access (RDMA) is an attractive option for such distributed systems because single-sided read-write provides latency and operational speed advantages. Since these operations do not involve a remote CPU, the performance provided is limited only by hardware.

Industry standard RDMA evolved from supercomputer environments and deployment in commercial data centers has been challenging. RDMA assumes a low latency, reliable, orderly network, while supercomputing architecture achieves these expectations through switch-forced lossless link-level flow control, which allows RDMA-enabled network cards (NICs) to implement simple congestion control and loss recovery schemes to react to congestion and packet loss. These structures are typically single tenants, while RDMA solutions for authorization, access control, failure recovery, and privacy protection also reflect the single tenant's expectations.

In contrast, modern very large scale data centers are characterized by multiple tenants, i.e., uncoordinated large scale distributed applications, sharing a common infrastructure. The combination of diverse, time-varying applications can lead to rapid changes in network traffic patterns, which require strong privacy and authentication.

But these requirements are contrary to the design choice of standard RDMA. Standard RDMA provides a hardware connection, a capability that is well integrated with early RDMA applications, but has fundamental limitations on scale isolation, performance, and fault tolerance. With modern service and storage systems running on a scale exceeding ten thousand servers, the hardware resources of each connection are easily exhausted. The congestion control algorithm needs to iterate continuously depending on deployment and application conditions, and the standard direct memory access RNIC embeds a significant portion of the congestion response into the hardware, which leaves little opportunity for adjustment after deployment.

Since applications and infrastructure are not trusted with each other, multi-tenants require line rate encryption and application support to manage the source of encryption keys. Although modern RNICs provide encryption functionality, practical challenges remain: encryption is connection-dependent, applications must trust a lower level stack to manage keys and do not support security-dependent management operations such as encryption key rotation.

Disclosure of Invention

In order to solve the technical problems, the invention provides a remote memory access method suitable for a multi-tenant data center, which provides performance advantages of standard single-side RDMA, namely high bandwidth, high operation rate and low delay, and also provides predictable tail performance, scalability, fault tolerance, isolation, safety characteristics and convenience of quick iteration after deployment, so that the limit of the integrated multi-tenant data center is better met.

In order to achieve the above purpose, the invention is realized by the following technical scheme:

The invention relates to a remote memory access method suitable for a multi-tenant data center, which comprises the following steps:

step 1, a client computer acquires necessary information of a server computer for accessing a related remote memory area by executing out-of-band remote procedure call;

Step 2, after the client computer obtains the information required by remote access, generating a client command, writing the client command into a client computer card, and sending the client command to the server computer;

step 3, the server computer performs data access on the target memory based on the client command, and feeds back the command completion condition to the client computer after the data access;

And step 4, the client computer judges whether fault recovery and congestion control are needed according to the received command completion condition.

The invention further improves that: in the step 1, the necessary information of the relevant remote memory area acquired by the client computer through executing the out-of-band remote procedure call includes: encryption key k_d, architecture name RegionId of the memory to be accessed.

The invention further improves that: in the step 2, the network card includes a registration area table, a command slot and a command slot table, a request window and a fifo arbiter, where the registration area table is a memory conversion table accessed by remote memory, the architecture name RegionId of the memory to be accessed indexes the memory conversion table to display the computer memory range where the network card is located and all corresponding metadata, i.e. area keys k_r, PCIe addresses, boundaries and permissions, each time the server computer memory is accessed, one memory area in the registration area table is represented, the command slot represents an operation in execution, and can be reused after the operation is completed, the command slot table is composed of a fixed number of slots in the network card, each slot is uniquely identified by its command slot number, the network card encodes the slot when the operation is completed, the encoding indicates to the client computer operating system which command is completed, because the operation may be completed out of order, the capacity in the request window is dynamically shared by a pair of fifos, i.e. each internal remote memory access service class is selected ready in the registration area table by a command of the fifo arbiter.

The invention further improves that: in the step 2, the client computer obtains the necessary information required by remote access and then generates a client command specifically as follows: the client computer writes a client command into the network card through PCIe according to the necessary information required by remote access, and writes MMIO storage into a command slot on the network card, wherein the command slot queues enqueues to wait for service operation, thereby starting the required RMA operation, and the starting the required RMA operation specifically comprises: enabling the command slot to enter service, enabling the RMA operation to wait for the capacity of a request window in the network card to meet the requirement of the user, sending a request to a server-side computer, borrowing the request window according to the actual size by the network card after the RMA operation enters service, signing the request sent to the server-side computer by the RMA operation by using an encryption key K_d, wherein the encryption key K_d is provided in the RMA operation command and sent to a network.

The invention further improves that: in the step 3, the data access of the server computer to the target memory based on the client command is specifically: the request reaches the network card of the server computer, the network card refers to the on-chip table with fixed size, namely the table integrated in the chip and capable of inquiring the position of the memory RegionId, searches the key information of the architecture name RegionId of the memory to be accessed contained in the request, deduces the encryption key K_d, verifies the inbound data packet, accesses the memory through PCIe after verification is successful, and executes the client command.

The invention further improves that: in the step 3, the feedback command completion situation from the server computer to the client computer after the data access specifically includes: the server computer card returns a completion indicator to the client computer designated completion area that initiated the command, wherein the completion indicator includes the command slot number to complete the command slot, the time required to perform the operation, the time required to request access to initiator services, and an operation status code that includes successful completion, NACK remote congestion, TIMEOUT.

The invention further improves that: in the step 4, the step of determining, by the client computer, whether the fault recovery is required according to the received command completion condition is specifically: the client computer generates accurate and fast feedback in the form of an explicit fault code based on the time required to perform the operation, the time required to request access to the initiator service, and the operation status code received in the completion indicator, indicating to the client computer which fault mode occurred and the appropriate action the client computer took when such a fault was encountered.

The invention further improves that: in the step 4, the client computer determines whether congestion control is required according to the received command completion condition, including: the client computer makes policy decisions to adjust the operation release rate based on the time required to perform the operation, the time required to request access to the initiator service, and the failure results in the received completion indicator.

The invention further improves that: remote access between a client computer and a server computer is guided using an RPC that includes secure exchange of keys, in particular: in memory registration, a region is assigned a frame name RegionId of the memory to be accessed, an application of a client computer designates a region key k_r, protecting the corresponding memory region, k_r is a 128-bit value from which an encryption key k_d is derived, the encryption k_d forms the basis of the security of the RMA operation, and protects a single transmission, and neither the region key k_r nor the encryption key k_d is sent over the network as part of a remote memory access operation or its response.

The invention further improves that: the derived encryption key k_d is used to generate a message authentication code, sign all protocol messages, and encrypt all data related to transmission, and the calculation formula is as follows:

K_d＝AES(Key＝K_r,Contents＝Address_Initiator,PID_Initiator,OperationType)。

The present invention is quite different from standard Remote Direct Memory Access (RDMA), in which the network card hardware is very simple, focusing only on fast, fixed functional primitives, unlike RDMA, which provides an infinite resource illusion. The invention manages explicit limited hardware resources in the client computer, which also implements the functions of fault recovery, congestion control and inter-operation sequencing for ease of fast iteration.

The remote memory access method (SRMA) of the present invention is connectionless and the state of the client computer card does not grow as the number of endpoint pairs increases. Because connection semantics are eliminated, a network card (NIC) may treat each operation as independent of the other operations, allowing the client computer operating system to process the order between operations as needed. The remote memory access method distributes the tasks of retry and fault recovery of each operation to the operating system of the client computer, provides simpler fault quick recovery behavior, ensures timely completion of the client computer card and directly provides quick and simple operation fault notification for the application program of the client computer.

The remote memory access method of the invention is small-scale, and takes the request as the basis of all transmission, each SRMA operation transmits a payload of 4KB at most, thereby realizing isolation and priority ordering; furthermore, all data transfer requires a request: SRMA does not initiate a transfer unless it guarantees access to the network card NIC SRAM of the present invention. The client computer card forced request design can prevent large inconsistencies from forming, combine requests with small operations, can implement congestion control for quick response and limit network queuing phenomena.

The client computer card may assist the client computer operating system in congestion control, in contrast to existing RDMA schemes that embed congestion control algorithms into the client computer card. The invention provides delay information and explicit fault quick notification of each operation for the congestion event, so that software can easily distinguish local congestion from remote congestion, and accurate operation is adopted to minimize congestion, overtime and packet loss.

The invention allocates resources based on service priority and sets clear limit to the work that the client computer operating system can provide each time, converting the limited resource pool into advantages. These limitations, in combination with small scale operations, can prevent low priority applications from monopolizing the network and planning operations in an explicit finite resource pool can simplify client computer cards as the illusion of providing near infinite resources is not required.

The transmission of the invention adopts the linear rate AES-GCM to encrypt and sign in the client computer card, and the design is consistent with the connectionless architecture of the invention. The present invention allows applications of client computers to directly manage encryption keys without extending trust in the infrastructure software and supporting frequent encryption key rotation with minimal interruption in availability.

The beneficial effects of the invention are as follows: the remote memory access method of the invention divides large operation into 4KB operation and realizes congestion control and operation management in the operation system of the client computer, thus only 0.5 kernel is needed to drive the line speed of 100Gbps, even under the conditions of high load and fault, the memory access method of the invention can provide predictable delay, ensure that high priority application is not affected by low priority application, the congestion control can almost immediately converge to fair bandwidth sharing under the condition of competing application, and shorten the unavailable time during encryption key rotation.

Drawings

FIG. 1 is a flow chart of a remote memory access method of the present invention.

FIG. 2 is a schematic diagram of the client computer operating system of the present invention.

Fig. 3is a schematic diagram of main components of a network card of the remote memory access method of the present invention.

FIG. 4 is a diagram illustrating the overall operation of the remote memory access method of the present invention for performing a 2KB read operation.

FIG. 5 is a schematic diagram illustrating the generation and distribution of security keys for a remote memory access method according to the present invention.

Detailed Description

Embodiments of the invention are disclosed in the drawings, and for purposes of explanation, numerous practical details are set forth in the following description. However, it should be understood that these practical details are not to be taken as limiting the invention. That is, in some embodiments of the invention, these practical details are unnecessary.

The invention provides a remote memory access method suitable for a multi-tenant data center, which is used for dividing responsibilities of hardware and software.

As shown in fig. 2, the client computer operating system of the remote memory access method of the present invention provides a number of transport abstractions and congestion management. At the bottom level, the command entry object manages a set of command slots and a memory region configured to accept completions, providing a familiar command/completion queue structure. Since the command slots are memory mapped registers in the network card NIC, the command entry will process the details of mmap () to insert these registers into the client computer's application memory. On the basis of the command entry, the next layer of command executors of the software stack support arbitrarily sized transmissions that are transparently blocked into operations up to 4KB in size and are affected by congestion control software adjustments. The application software layer of client computing above the command executor is not responsible for blocking, pacing or congestion control. The application may choose to manage the failure according to personal needs. Since the network card of the present invention does not preserve the state of each destination, each command fully encodes all metadata required for operation.

As shown in fig. 3, the network card of the present invention includes a registration area table, command slots and a command slot table, a request window and a fifo arbiter, where the registration area table is a memory conversion table accessed by remote memory, the architecture name RegionId of the memory to be accessed indexes the memory conversion table to display the memory range of the network card and all corresponding metadata, i.e. area keys k_r, PCIe addresses, boundaries, rights, each time the server computer memory is accessed, one memory area in the registration area table is represented, the command slots represent an operation in execution, and can be reused after the operation is completed, the command slot table is composed of a fixed number of slots in the network card, each slot is uniquely identified by its command slot number, the network card encodes the slots when the operation is completed, the encoding indicates to the client computer operating system which command is completed, because the operation may be completed out of order, the capacity in the request window is dynamically shared by a pair of fifos, i.e. each internal remote memory access service class is selected ready by a command in the registration area table.

When an application program sends an operation to a command slot, a CPU performs MMIO write operation to a corresponding hardware register through PCIe. This write may cause the SRMA hardware pipeline to queue in hardware first-in-first-out, waiting for scheduling, depending on the class of service of the operation. The queue head waits for the capacity in the request window and then sends the request to the far end, which is specified by the address information in the command itself.

Taking a 2KB read operation as an example, as shown in fig. 1, the remote memory access method suitable for a multi-tenant data center of the present invention specifically includes the following steps:

step 1, a client computer acquires necessary information of a server computer for accessing a related remote memory area by executing out-of-band remote procedure call.

Before performing the RMA operation, the client computer is started to perform an out-of-band remote procedure call (out-of-band RPC) to obtain the necessary information required for the remote memory region to be accessed, including: the encryption key k_d is an encryption security key bound to the initiating client, is difficult to guess and is not transferable, and RegionId is an architecture name of the memory to be accessed established at the time of memory registration on the server. All protocol messages are signed using the information verification code generated by the encryption key k_d. Similarly, all data is encrypted using encryption key k_d.

And step 2, after the client computer obtains the information required by remote access, generating a client command, writing the client command into a client computer card, and sending the client command to the server computer.

After obtaining the information required for remote access, the client initiates its required RMA operation, i.e. a 2KN read operation, which in this example is performed by writing a client command over PCIe and storing MMIO into a command slot on the network card NIC that queues the enqueue waiting for service operation. Thus initiating the required RMA operation, requesting to wait for execution according to the bidding rules of the SRMA. To enter service, the operation needs to get 4KB of available space in the on-chip request window, which is an SRAM buffer for storing inbound payloads. The 4KB needs to be arbitrated regardless of the operation size, so that small operations can be prevented from occupying space of large operations, and the large operations are starved.

After the RMA operation enters the service, the network card deducts according to the actual size, namely 2KB borrowing request window, and signs a request sent to the server computer by the RMA operation by using an encryption key k_d, wherein the encryption key k_d is provided in an RMA operation command and is sent to the network.

And 3, the server computer performs data access on the target memory based on the client command, and feeds back the command completion condition to the client computer after the data access. The server computer card returns a completion indicator to the client computer designated completion area that initiated the command, wherein the completion indicator includes the command slot number to complete the command slot, the time required to perform the operation, the time required to request access to initiator services, and an operation status code that includes successful completion, NACK remote congestion, TIMEOUT.

Normally, when the request arrives at the network card of the server computer, the network card refers to the on-chip table with a fixed size, that is, the table integrated in the chip and capable of searching the location of the memory RegionId, so as to find RegionId included in the request, derive the encryption key k_d, and verify the inbound data packet. After verification is successful, the network card accesses the memory through PCIe and executes the client command. The NIC then reads the requested data over PCIe and takes the read data stream as a separate network response stream. Each response is encrypted and signed with the previously derived key information. These responses travel through the network due to adaptive routing, but out of order arrival situations may occur.

Upon arrival at the initiator, each response is individually validated and decrypted and then transferred to the initiator's memory via the PCIe write stream, with the offset of each response encoded in the inbound response. To avoid out-of-order responses, the NIC tracks byte arrivals, and in the absence of errors, writes a successful completion message to the boot software once all bytes have arrived. The operation completion also includes a hardware delay metric value showing the time it takes to perform the operation (total_delay) and the time it takes to request to enter the initiator service (issue_delay).

Although a fault-free condition has been described, the above-described operation may experience various faults. For example, a read operation may not enter service at the initiator due to severe local congestion such as a large number of queuing operations. When a request arrives at an overloaded server, the serving NIC may discard or "NACK" requests, such as lengthy inbound request queues. Finally, the server-to-client response may be dropped or delayed in the network.

And step 4, the client computer judges whether fault recovery and congestion control are needed according to the received command completion condition.

The invention generates accurate and rapid feedback in the form of an explicit fault code, and indicates which fault mode occurs to the start-up software. Upon encountering such a failure, the application software may take appropriate action, such as immediately retry the action.

Given that the objects of the present invention are deployed at mutually untrusted endpoints and mutually untrusted network structures, all essential aspects of the remote memory access method of the present invention include security.

As shown in fig. 5, the security of the present invention is as follows:

Similar to standard RDMA, the present invention uses an RPC that includes secure exchange of keys to direct remote access between a client computer and a server computer by out-of-band encryption, specifically: in memory registration, a region is assigned a frame name RegionId of the memory to be accessed, an application of a client computer designates a region key k_r, protecting the corresponding memory region, k_r is a 128-bit value from which an encryption key k_d is derived, the encryption k_d forms the basis of the security of the RMA operation, and protects a single transmission, and neither the region key k_r nor the encryption key k_d is sent over the network as part of a remote memory access operation or its response.

The derived encryption key k_d is used to generate a message authentication code, sign all protocol messages, and encrypt all data related to transmission, and the calculation formula is as follows:

K_d＝AES(Key＝K_r,Contents＝Address_Initiator,PID_Initiator,OperationType)。

The key calculated by this function is for each boot process (identified by the host address of the process and PID of the process) and the type of operation, and is rooted in a single shared process-K _ r that is easily placed in each zone table of NIC memory. When an application of the client computer allocates a command slot to start an RMA, the SRMA driver always stores the associated pid_initiator in the configuration of the slot. The SRMA hardware always includes pid_initiator in the request packet, regionId. Thus, the server-side computer card can derive an assigned encryption key k_d for each inbound request. In addition, key derivation can be easily accomplished at the server side computer operating system, which can calculate k_d for any potential initiator and send k_d to the initiator in an RPC response without the need to keep k_d on any host or NIC table. The present invention adds a message counter that is incremented NIC-by-NIC for the encryption process for protecting key integrity and preventing replay. It is critical that the security-related state on the NIC does not grow with the number of communication endpoint pairs.

A process attacker can easily guess RegionId in the face of a malicious user attempting to access a remote memory area owned by another tenant, but it is difficult to obtain a k_d to match hosts and processes, lacking root level attacks. In the face of an attacker who has full access to the network links and switches, they can observe the ciphertext during transmission, but cannot easily decrypt it without destroying the participating hosts. It is noted that the present invention also prevents duplicate attacks, and although the remote memory access method of the present invention does not prevent duplicate read requests from being accommodated, such attacks generate new encrypted ciphertexts, so that the attacker still needs the correct cipher k_d to decrypt the resulting response.

The security model of the present invention does not change the behavior of hosts with root level hazards, and a root level attacker can emulate a process on a host, so that authentication can be performed as one of these users in any way it chooses. The main defensive measure against such threats is to accelerate the detection, e.g. by performing frequent encryption key rotations, forcing repeated authentication steps, which can be recorded and checked. The present invention provides explicit support for encryption key rotation to help achieve this goal.

If the verification fails, the network card discards the response of the verification failure. In contrast, an inbound request that fails authentication will immediately send a FAILURE notification in response, and use a known reserved key signature, resulting in the REMOTE_ AUTHENTICATION _FAILURE signal. Discarding such requests may make the remote memory access of the present invention more powerful, forcing an attacker to face time-out rather than timely making a negative response. But this will have an impact on non-attack situations because the remote authentication failure error code is easily identified as a side effect of the encryption key rotation and thus the recovery step is obvious to the client software. In contrast, discarded requests appear to timeout, which does not immediately indicate that an encryption key rotation may occur.

The invention can ensure that the high priority application is not affected by the low priority application, can provide predictable delay even under high load and fault conditions, can almost immediately converge to fair bandwidth sharing under the condition of competing applications, and shortens the unavailable time during encryption key rotation.

The foregoing description is only illustrative of the invention and is not to be construed as limiting the invention. Various modifications and variations of the present invention will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, or the like, which is within the spirit and principles of the present invention, should be included in the scope of the claims of the present invention.

Claims (10)

1. A remote memory access method suitable for a multi-tenant data center is characterized in that: the remote memory access method specifically comprises the following steps:

step 1, a client computer acquires necessary information of a server computer for accessing a related remote memory area by executing out-of-band remote procedure call;

Step 2, after the client computer obtains the information required by remote access, generating a client command, writing the client command into a client computer card, and sending the client command to the server computer;

step 3, the server computer performs data access on the target memory based on the client command, and feeds back the command completion condition to the client computer after the data access;

And step 4, the client computer judges whether fault recovery and congestion control are needed according to the received command completion condition.

2. The method for remote memory access for a multi-tenant data center of claim 1, wherein: in the step 1, the necessary information of the relevant remote memory area acquired by the client computer through executing the out-of-band remote procedure call includes: encryption key k_d, architecture name RegionId of the memory to be accessed.

3. The method for remote memory access for a multi-tenant data center of claim 1, wherein: in step 1, a remote access between a client computer and a server computer is guided by an out-of-band remote procedure call comprising a secure exchange of keys, in particular: in memory registration, a region is assigned a frame name RegionId of the memory to be accessed, an application of a client computer designates a region key k_r, protecting the corresponding memory region, k_r is a 128-bit value from which an encryption key k_d is derived, the encryption k_d forms the basis of the security of the RMA operation, and protects a single transmission, and neither the region key k_r nor the encryption key k_d is sent over the network as part of a remote memory access operation or its response.

4. A remote memory access method for a multi-tenant data center according to claim 3, wherein: the derived encryption key k_d is used to generate a message authentication code, sign all protocol messages, and encrypt all data related to transmission, and the calculation formula is as follows:

K_d＝AES(Key＝K_r,Contents＝Address_Initiator,PID_Initiator,OperationType)。

5. the method for remote memory access for a multi-tenant data center of claim 1, wherein: in the step2, the network card includes a registration area table, a command slot and a command slot table, a request window and a fifo arbiter, where the registration area table is a memory conversion table accessed by remote memory, the architecture name RegionId of the memory to be accessed indexes the memory conversion table to display the computer memory range where the network card is located and all corresponding metadata, i.e., area keys k_r, PCIe addresses, boundaries and permissions, each time the server computer memory is accessed, the network card represents a memory area in the registration area table, the command slot represents an operation in execution, the operation can be reused after completion, the command slot table is composed of a fixed number of slots in the network card, each slot is uniquely identified by its command slot number, the network card encodes the arbitration slot when the operation is completed, the encoding indicates to the client computer operating system which command has been completed, and the capacity in the request window is dynamically shared by a pair of fifos, i.e., each internal remote memory access service class is selected by a fifo in the ready command in the registration area table.

6. The method for remote memory access for a multi-tenant data center of claim 5, wherein: in the step 2, the client computer obtains the necessary information required by remote access and then generates a client command specifically as follows: the client computer writes a client command into the network card through PCIe according to the necessary information required by remote access, and writes MMIO storage into a command slot on the network card, wherein the command slot queues enqueues to wait for service operation, thereby starting the required RMA operation, and the starting the required RMA operation specifically comprises: enabling the command slot to enter service, enabling the RMA operation to wait for the capacity of a request window in the network card to meet the requirement of the user, sending a request to a server-side computer, borrowing the request window according to the actual size by the network card after the RMA operation enters service, signing the request sent to the server-side computer by the RMA operation by using an encryption key K_d, wherein the encryption key K_d is provided in the RMA operation command and sent to a network.

7. The method for remote memory access for a multi-tenant data center of claim 1, wherein: in the step 3, the data access of the server computer to the target memory based on the client command is specifically: the request reaches the network card of the server computer, the network card refers to the on-chip table with fixed size, the key information of the architecture name RegionId of the memory to be accessed contained in the request is searched, the encryption key K_d is deduced, the inbound data packet is verified, after the verification is successful, the network card accesses the memory through PCIe, and the client command is executed.

8. The method for remote memory access for a multi-tenant data center of claim 7, wherein: in the step 3, the feedback command completion situation from the server computer to the client computer after the data access specifically includes: the server computer card returns a completion indicator to the client computer designated completion area that initiated the command, wherein the completion indicator includes the command slot number to complete the command slot, the time required to perform the operation, the time required to request access to initiator services, and an operation status code that includes successful completion, NACK remote congestion, TIMEOUT.

9. The method for remote memory access for a multi-tenant data center of claim 8, wherein: in the step 4, the step of determining, by the client computer, whether the fault recovery is required according to the received command completion condition is specifically: the client computer generates accurate and fast feedback in the form of an explicit fault code based on the time required to perform the operation, the time required to request access to the initiator service, and the operation status code received in the completion indicator, indicating to the client computer which fault mode occurred and the appropriate action the client computer took when such a fault was encountered.

10. The method for remote memory access for a multi-tenant data center of claim 9, wherein: in the step 4, the client computer determines whether congestion control is required according to the received command completion condition, including: the client computer makes policy decisions to adjust the operation release rate based on the time required to perform the operation, the time required to request access to the initiator service, and the failure results in the received completion indicator.