WO2020001459A1

WO2020001459A1 - Data processing method, remote direct memory access network card, and device

Info

Publication number: WO2020001459A1
Application number: PCT/CN2019/092937
Authority: WO
Inventors: 孙贝磊; 周超; 李涛
Original assignee: 华为技术有限公司
Priority date: 2018-06-26
Filing date: 2019-06-26
Publication date: 2020-01-02
Also published as: CN110647480B; CN110647480A

Abstract

The present application provides a data processing method, an RNIC, and a device. The method comprises: a first RNIC receiving a remote direct memory access (RDMA) write request sent by a second RNIC, the RDMA write request comprising first data and a data persistence flag, the data persistence flag being used to indicate that the first data is data to be persisted; according to the RDMA write request, the first RNIC sending a DMA write request to a first processor, to instruct the first processor to write the first data into a first device, the first RNIC and the first processor belonging to the first device, the second RNIC belonging to a second device, and the two devices communicating on the basis of an RDMA approach; and according to the data persistence flag, the first RNIC determining that the first data is the data to be persisted, and the first RNIC instructing the first processor to save the first data into a non-volatile memory of the first device. Thus, the present invention reduces the load of an RDMA network during remote memory persistence.

Description

Data processing method, remote direct access storage network card and equipment

Technical field

The present application relates to the field of computer technology, and in particular, to a data processing method, a remote direct access storage network card, and a device.

Background technique

Storage class memory (SCM), such as 3D Xpoint, is a new type of non-volatile memory (NVM), which is a combination of traditional storage media (such as mechanical hard drives, solid state Hard disks, etc.) and storage (such as dynamic random access memory). The SCM can be embedded into the motherboard's slot like a dynamic random access memory. Compared to a dynamic random access memory, the SCM can still save data without interruption in the state of power loss, and has the characteristics of power-off saving. SCM can provide faster read and write speeds than flash memory, and is cheaper than dynamic random access memory. In some computing device system architectures, SCM is used as memory usage. In some solutions, multiple SCM-based computing devices are connected in an interconnected manner to form an SCM resource pool to expand the capacity of the SCM, thereby achieving redundant backup of data.

In the SCM resource pool composed of multiple computing devices with SCM as memory, any two computing devices can communicate and transfer data based on remote direct memory access (RDMA) technology. Remote direct memory access can be referred to as For remote direct fetch. Unlike traditional network transmission technologies, RDMA technology can directly transfer data from the memory of one computing device to the memory of another computing device without the intervention of the operating systems or kernels of the two computing devices. For the SCM resource pool based on RDMA technology for communication and data transmission, it is necessary to solve the problem of remote memory persistence. Memory persistence refers to writing data from the volatile storage medium of the computing device back to the non-volatile storage medium of the computing device; while remote memory persistence refers to the SCM of a computing device After writing the data to another computing device, the data is stored in the SCM of the other computing device. In a storage system that uses SCM as a memory, data in the SCM of computing device A (the computing device that initiates the remote operation request) is to be saved to the SCM of computing device B (the computing device that receives the remote operation request). The data requires After passing through the processor of the computing device B, and because there is a cache in the processor of the computing device B, the data may be temporarily cached in the cache, and there is a risk of losing power. In the current solution, computing device A generally writes data to computing device B through an RDMA write request, and then initiates an RDMA read request or RDMA send request to The data written by the RDMA write request is written back to the SCM of the computing device B. The RDMA read request or RDMA send request initiated by the computing device A can be regarded as a remote memory persistence request. This solution has the following problems: Since the computing device A must initiate a remote memory persistence request after each RDMA write request, the network load of the RDMA network is increased.

Summary of the invention

This application provides a data processing method, a remote direct access memory card and a device, and solves the problem of a large network load caused by an additional remote memory persistence request.

In a first aspect, a data processing method is provided, including:

The first remote direct memory card (remote direct memory access network interface card, RNIC) receives the remote direct memory access (RDMA) write request sent by the second RNIC. The RDMA write request includes the first data and data persistence. Flag, the RDMA write request is used to request the first data to be written to the first device, and the data persistence flag is used to indicate that the first data is data to be persisted, where the first RNIC is the RNIC of the first device, that is, The RNIC of the device receiving the RDMA request. The second RNIC is the RNIC of the second device, that is, the RNIC of the device sending the RDMA request. The first device and the second device communicate based on the RDMA method. According to the RDMA write request, the first RNIC determines The first data needs to be written to the first device. The first RNIC sends a direct memory access (DMA) write request to the first processor. The DMA write request includes the first data to instruct the first processor to send the first data to the first processor. A data is written into the first device, wherein the first processor is the processor of the first device, and the first RNIC and the first processor communicate in a DMA-based manner; according to the data Persistence flag. The first RNIC determines that the first data is data to be persisted, and the first RNIC instructs the first processor to save the first data to a non-volatile memory of the first device.

The non-volatile memory may be an SCM. SCM can be phase-change random access memory (PRAM), PRAM can be 3D Xpoint, resistive random access memory (ReRAM), magnetic random access memory (magnetic random access memory) memory, MRAM), and so on. A device that receives an RDMA request can be called a remote device, and a device that sends an RDMA request can be called a local device.

In the above solution, after the RNIC of the remote device determines that the first data is to be persisted according to the data persistence flag in the RDMA write request, it directly instructs the processor of the remote device to save the first data to the first device. To complete the memory persistence of the first data in the non-volatile memory, without the RNIC of the local device initiating a remote memory persistence request. Because the local device does not need to initiate a remote memory persistence request after initiating an RDMA write request, the load on the RDMA network is reduced; in addition, the data persistence flag is carried in the RDMA write request to enable remote write operations and remote memory persistence operations It becomes an operation that is performed continuously to ensure that data can be saved to non-volatile memory after being written to the remote device to avoid data inconsistencies.

In a possible implementation manner, the data persistence mark may be a write persistence instruction. In the case where the data is marked as a write-persistent instruction, the RDMA write request can also be called an RDMA write-persistence request, such as rdma write durable. By adding a write persistence instruction to the original RDMA operation instruction, the RNIC of the remote device parses the RDMA write request to obtain the write persistence instruction, and the RNIC of the remote device can determine that the first data in the RDMA write request is pending. Persistent data.

In another possible implementation manner, the data persistence mark is a destination storage address corresponding to the first data, and a storage space corresponding to the destination storage address is used to store the first data, and the destination storage address is in the first device. The persistent storage address, which is used to store the data to be persisted. Here, the destination storage address corresponding to the first data is a storage address corresponding to a storage space for storing the first data in the first device designated by the second device. The remote device allocates storage space for storing data to be persisted in advance, and determines the destination storage address corresponding to the first data is the storage corresponding to the storage space allocated for storing data to be persisted in the remote device in advance In the case of an address, the RNIC of the remote device may determine that the first data in the RDMA write request is data to be persisted.

In another possible implementation manner, the first RNIC may instruct the first processor to save the first data to the non-volatile memory of the first device in the following manner: The first RNIC receives the data corresponding to the second RNIC. A DMA least significant bit (LSB) read request is added to the queue (receive queue, SQ); the first RNIC sends the DMA LSB read request to the first processor, and the DMA LSB read request is used to instruct the first processor Write all data buffered in the peripheral bus link of the first device to the non-volatile memory of the first device. This method may be applicable to the case where the data does not pass through the last level cache (LLC) of the first processor during the data writing process of the first RNIC. Because the data does not pass through the LLC of the first processor, the data may not yet be written and cached in the cache of the processor's input / output (I / O) controller. According to the peripheral component interconnect standard (PCIe) protocol stipulates that before any read operation is performed, all write operations need to be completed, so by adding a DMA LSB read request to the receive queue, and then sending the DMA LSB read request to the processor, the processor executes the DMA. The read operation corresponding to the LSB read request. The read operation can write the data that has not been written on the peripheral bus to the non-volatile memory, and then can save the first data that may also be cached on the peripheral bus. To non-volatile memory.

In another possible implementation manner, the first RNIC may instruct the first processor to write all data buffered in the peripheral bus link of the first device to the first by sending another DMA read request to the processor. The device's non-volatile memory. The above DMA read request may be a read request to read an arbitrary address in the SCM, and the address read by the read request may be any address in the SCM. For example, the address is a starting address range used to store the first data. Start address, end address, or any address other than start address and end address. Optionally, the address to be read by the read request may also be an address segment. For example, the address segment is an address range for storing the first data, or any one of the storage ranges in the SCM.

In another possible implementation manner, the first RNIC may instruct the first processor to save the first data to the non-volatile memory of the first device in the following manner: the first RNIC generates RDMA reception according to the first data (receive) After requesting a corresponding work queue entry (work queue entry, WQE), the WQE is cleared, and the RDMA receive request is used to receive an RDMA send request initiated by the second device; after clearing the WQE, the first RNIC Generate a completion queue entry (CQE) corresponding to the RDMA request, to instruct the first processor to store first data buffered in a volatile storage medium of the first processor to a non-volatile memory of the first device Volatile memory. The volatile storage medium of the first processor may be LLC. This method can be applicable to the case where the data passes through the LLC of the processor of the remote device during the data writing process of the RNIC of the remote device. Because the first data passes through the LLC of the first processor, by generating and clearing the WQE corresponding to the RDMA reception request and generating the CQE corresponding to the RDMA reception request according to the first data, when the CQE is obtained by the first processor, an interrupt can be generated and the The first processor is caused to write the data of the corresponding address back to the non-volatile memory. WQE and CQE are both generated based on the first data, which corresponds to the first data. The first data can be stored in the non-volatile memory by writing the data of the corresponding address back to the non-volatile memory.

In another possible implementation manner, after the first RNIC instructs the first processor to save the first data in the non-volatile memory of the first device, the first RNIC may also send the persistence corresponding to the first data to the second RNIC. A confirmation message. The persistence confirmation message includes a second packet sequence number (packet sequence number, PSN), and the second PSN is a sequence number of the first data. The persistence confirmation message is used to indicate that the first data memory is persistent. . By sending a persistent confirmation message to the second RNIC, the second RNIC can determine that the first data is stored in the non-volatile memory of the first device according to the second PSN.

In another possible implementation manner, after the first RNIC receives the RDMA write request sent by the second RNIC, it sends a reception confirmation message corresponding to the first data to the second RNIC, and the reception confirmation message corresponding to the first data includes the first PSN The first PSN is a serial number of the first data, and the reception confirmation message is used to indicate that the first RNIC receives the first data. By sending a reception confirmation message to the second RNIC, the second RNIC can determine that the transmission of the first data is completed according to the first PSN.

In a second aspect, another data processing method is provided, including: the second RNIC receives an RDMA write persistence request from a second processor, the RDMA write persistence request includes a data persistence flag, and the RDMA write persistence request is used for The first data is requested to be stored in the non-volatile memory of the first device, and the data persistence flag is used to indicate that the first data is data to be persisted, where the second RNIC and the second processor are second The device's RNIC and processor, the second RNIC and the second processor are based on DMA communication, the second device is the device initiating the RDMA request, the first device is the device receiving the RDMA request, and the second device and the first device are based on RDMA Mode communication; the second RNIC generates an RDMA write request according to the RDMA write persistence request, the RDMA write request includes the first data and a data persistence flag; the second RNIC sends the RDMA write request to the first RNIC, and the RDMA write request is used Request to write the first data to the first device, and the first RNC is the RNIC of the first device.

The device receiving the RDMA request may be called a remote device, and the device sending the RDMA request may be called a local device, that is, the first device is a remote device, and the second device is a local device.

In the above solution, when the RNIC of the local device receives the RDMA write persistence request initiated by the processor of the local device, the RDMA write request is generated according to the RDMA write persistence request. The meaning of the RDMA write request itself can make the remote device's The RNIC writes the first data to the remote device. The data persistence mark in the RDMA write request can make the RNIC of the remote device know that the first data is data to be persisted, so the RNIC of the remote device can write the first data. After writing to the remote device, the first data is stored in the non-volatile memory of the first device to complete the memory persistence of the first data. This solution is equivalent to combining the write request and the memory persistence request in one request. The local device does not need to send additional memory persistence requests, which reduces the load of the RDMA network. In addition, the data persistence mark is carried in the RDMA write. In the request, the remote write operation and the remote memory persistence operation are continuously performed operations, ensuring that data can be saved to the non-volatile memory after being written to the remote device, thereby avoiding the problem of data inconsistency.

In a possible implementation manner, the foregoing data persistence is marked as a write persistence instruction. In the case where the data is marked as a write-persistent instruction, the RDMA request can also be called an RDMA write-persistence request, such as rdma write durable. By adding a write persistence instruction to the original RDMA operation instruction, the RNIC of the remote device can determine that the first data in the RDMA write request is to be persisted after parsing the RDMA write request to obtain the write persistence instruction. data.

In another possible implementation manner, the data persistence mark is a destination virtual memory address corresponding to the first data, and the destination storage address corresponding to the first data is a persistent storage address in the first device, and the persistent storage address corresponds to Of storage space is used to store data to be persisted. The destination storage address corresponding to the first data refers to a storage address in the first device designated by the second device for storing the first data. By setting the destination storage address corresponding to the first data to the storage address corresponding to the storage space for storing the data to be persisted in the remote device, the RNIC of the remote device can determine the first data according to the destination storage address For data to be persisted.

In another possible implementation manner, after the second RNIC sends an RDMA write request to the first RNIC, after receiving the persistence confirmation message corresponding to the first data sent by the first RNIC, the second RNIC generates RDMA The CQE corresponding to the write persistence request. The persistence confirmation message corresponding to the first data is used to indicate the completion of the memory persistence of the first data. The CQE corresponding to the RDMA write persistence request is used to notify the second processor of the memory persistence completion. . This method can be applied to the case that the data does not pass through the LLC of the processor of the remote device during the process of writing data by the RNIC of the remote device. Upon receiving the persistence confirmation message sent by the first RNIC, the second RNIC generates a CQE corresponding to the RDMA write persistence request. When the second processor obtains the CQE, the first number of memory persistences can be determined according to the CQE After the modification is completed, the first processor does not need to initiate a remote memory persistence request again, which reduces the load of the RDMA network.

In another possible implementation manner, before the second RNIC generates the CQE corresponding to the RDMA write persistence request, the second RNIC buffers the first data in a case where the second RNIC receives a reception confirmation corresponding to the first data sent by the first RNIC. A corresponding reception confirmation message, the reception confirmation message corresponding to the first data includes a first PSN, the first PSN is a serial number of the first data, and the reception confirmation message corresponding to the first data is used to instruct the first RNIC to receive the first data; The second RNIC receives the first confirmation message sent by the first RNIC, and the first confirmation message includes the second PSN. In the case where the second PSN is the same as the first PSN, the second RNIC determines to receive the persistent confirmation corresponding to the first data. Message. In the RDMA protocol, an acknowledgement (acknowledgement, ACK) message is only used to indicate the meaning of acknowledgement, and the ACK message is specifically used to indicate that the confirmation of what request needs to be based on the request sent before the acknowledgement message is received or the received ACK The order of the messages is judged. In this design solution, because the write operation occurs before the memory persistence operation, the first confirmation message carrying the serial number of the first data must be the confirmation of the reception of the first data. Message, after receiving the reception confirmation message of the first data, by buffering the reception confirmation message, the RNIC of the second device does not need to wait for the persistent confirmation message of the first data to be sent to the first device RNIC. For the next data of the data, by comparing the PSN method, the memory persistence of one data and the write operation of the next data of the data can be performed in parallel, which improves the efficiency of data memory persistence and reduces the delay.

In another possible implementation manner, after the second RNIC sends an RDMA write request to the first RNIC, upon receiving a reception confirmation message corresponding to the first data sent by the first RNIC, the second RNIC generates an RDMA write persistence The CQE corresponding to the request is received. The reception confirmation message corresponding to the first data is used to indicate that the first RNIC receives the first data. The CQE corresponding to the RDMA write persistence request is used to notify the second processor that the first data has been written to the first. Device; the second RNIC receives the RDMA reception request sent by the second processor; in the case of receiving the persistence confirmation message corresponding to the first data, the second RNIC generates a CQE corresponding to the RDMA request and the persistence confirmation corresponding to the first data The message is used to indicate that the first number of memory persistence is completed, and the CQE corresponding to the RDMA receiving request is used to notify the second processor that the memory persistence of the first data is completed. This method can be applicable to the case where the data passes through the LLC of the processor of the remote device during the data writing process of the RNIC of the remote device. Since the first data passes through the LLC of the processor of the first device, the CQE corresponding to the RDMA write persistence request is generated when the reception confirmation message corresponding to the first data is received, so that the second processor can write persistence according to the RDMA The CQE corresponding to the request initiates an RDMA reception request, eliminating the process of the RDMA sending request initiated by the second processor, and reducing the load of the RDMA network.

In another possible implementation manner, before the second RNIC generates the CQE corresponding to the RDMA request, the second RNIC may also buffer the reception confirmation message corresponding to the first data, and the reception confirmation message corresponding to the first data includes the first PSN, the first One PSN is the serial number of the first data; the second RNIC receives the first confirmation message sent by the first RNIC, and the first confirmation message includes the second PSN; when the second PSN is the same as the first PSN, the second RNIC It is determined that a persistence confirmation message corresponding to the first data is received. By buffering the reception confirmation message of the first data, the second device can send the next data of the first data to the first device without waiting for receiving the persistent confirmation of the first data, and a data can be made by comparing the PSN The memory persistence and the next data write operation of the data are performed in parallel, improving the efficiency of data memory persistence and reducing the delay.

According to a third aspect, another data processing method is provided, including: the second processor sends an RDMA write persistence request to the second RNIC, the RDMA write persistence request includes a data persistence flag, and the RDMA write persistence request is used for the request The first data is stored in the non-volatile memory of the first device, and the data persistence flag is used to indicate that the first data is data to be persisted, wherein the second RNIC and the second processor are the data of the second device, respectively. The RNIC and the processor, the second RNIC and the second processor communicate in a DMA-based manner, the second device is a device initiating an RDMA request, the first device is a device receiving an RDMA request, and the first device and the second device communicate in an RDMA In the case of obtaining the CQE corresponding to the RDMA write persistence request, the second processor determines that the first data has been stored in the non-volatile memory of the first device.

The device that receives the RDMA request may be called a remote device, and the device that sends the RDMA request may be called a local device. That is, the first device is a remote device and the second device is a local device.

This solution can be applied to the case that the data does not pass through the LLC of the processor of the remote device during the process of writing data by the RNIC of the remote device. The processor of the local device only needs to send an RDMA write request to write and store the first data in the remote device. It does not need to initiate a remote memory persistence request after initiating an RDMA write request, which reduces the RDMA. The load of the network; in addition, carrying the data persistence tag in an RDMA request makes the remote write operation and remote memory persistence operation be continuously performed operations, ensuring that data can be saved to non-volatile after writing to the remote device To avoid data inconsistency in the memory.

In another possible implementation manner, the foregoing data persistence is marked as a write persistence instruction. By adding an RDMA write persistence instruction to the original RDMA operation instruction, the RNIC of the remote device can determine that the first data in the RDMA request is to be persisted after parsing the RDMA request to obtain the RDMA write persistence instruction. data.

In another possible implementation manner, the data persistence mark is a destination storage address corresponding to the first data, the destination storage address corresponding to the first data is a persistent storage address in the first device, and the persistent storage address corresponds to Storage space is used to store data to be persisted. The destination storage address corresponding to the first data refers to a storage address in the first device designated by the second device for storing the first data. By setting the destination storage address corresponding to the first data to the storage address corresponding to the storage space for storing the data to be persisted in the remote device, the RNIC of the remote device can determine the first data according to the destination storage address For data to be persisted.

According to a fourth aspect, another data processing method is provided, including: the second processor sends an RDMA write persistence request to the second RNIC, the RDMA write persistence request includes a data persistence flag, and the RDMA write persistence request is used for the request Storing first data in a non-volatile memory of the first device, and a data persistence flag is used to indicate that the first data is data to be persisted, wherein the second RNIC and the second processor are second The device's RNIC and processor, the second RNIC and the second processor communicate in a DMA-based manner, the second device is the device that initiates the RDMA request, the first device is the device that receives the RDMA request, and the first device and the second device are based on RDMA Mode communication; when the CQE corresponding to the RDMA write persistent request is acquired, the second processor sends an RDMA reception request to the second RNIC; when the CQE corresponding to the RDMA reception request sent by the second RNIC is acquired, the second The processor determines that the first data has been stored in a non-volatile memory of the first device.

This solution can be applied to the case that the data does not pass through the LLC of the processor of the remote device during the data writing process of the RNIC of the remote device. Eliminates the need for the local device to send an RDMA transmission request, reducing the load on the RDMA network.

In a possible implementation manner, the foregoing data persistence is marked as a write persistence instruction. By adding an RDMA write persistence instruction to the original RDMA operation instruction, the RNIC of the remote device can determine that the first data in the RDMA request is to be persisted after parsing the RDMA request to obtain the RDMA write persistence instruction. data.

According to a fifth aspect, an RNIC is provided, including various modules for executing the data processing method in the first aspect or any possible implementation manner of the first aspect.

According to a sixth aspect, another RNIC is provided, including various modules for executing the data processing method in the second aspect or any one of the possible implementation manners of the second aspect.

According to a seventh aspect, a processor is provided to execute part or all of the processes involved in the third aspect or the fourth aspect.

According to an eighth aspect, a first device is provided, including a processor, a non-volatile memory, and an RNIC, and the RNIC is configured to execute the operation steps in the method flow described in the first aspect.

According to a ninth aspect, a second device is provided, including a processor, a non-volatile memory, and an RNIC. The RNIC is configured to execute the operation steps in the method flow described in the second aspect, and the processor is configured to execute the first Operation steps in the method flow described in the third aspect or the fourth aspect.

According to a tenth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions that, when run on a computer, cause the computer to execute the methods described in the above aspects.

According to an eleventh aspect, a computer program product containing instructions is provided, which when executed on a computer, causes the computer to perform the methods described in the above aspects.

On the basis of the implementation manners provided in the above aspects, this application may be further combined to provide more implementation manners.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic structural diagram of a computing device according to an embodiment of the present application; FIG.

2 is a schematic flowchart of remote memory persistence provided by an embodiment of the present application;

3 is a schematic flowchart of another remote memory persistence process provided by an embodiment of the present application;

4 is a schematic diagram of a computing device network that communicates through RDMA technology according to an embodiment of the present application;

5 is a schematic diagram of a communication system composed of a local device and a remote device according to an embodiment of the present application;

6 is a schematic structural diagram of an implementation manner of an RNIC according to an embodiment of the present application;

7 is another schematic structural diagram of an RNIC according to an embodiment of the present application;

8 is a schematic diagram of a channel between two computing devices according to an embodiment of the present application;

9 is a schematic flowchart of a data processing method according to an embodiment of the present application;

10 is a schematic flowchart of another data processing method according to an embodiment of the present application;

11 is a schematic flowchart of another data processing method according to an embodiment of the present application;

FIG. 12 is a schematic structural diagram of a computing device according to an embodiment of the present application; FIG.

FIG. 13 is a schematic structural diagram of another computing device according to an embodiment of the present application.

detailed description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

First, a conventional data processing method is described with reference to FIGS. 1 to 3.

FIG. 1 is a schematic diagram of a computing device. As shown in FIG. 1, the computing device 10 includes an RNIC 101, a processor 102, and an SCM 103. The RNIC, the processor 102, and the SCM 103 are connected through a bus 104. The bus 104 includes, but is not limited to, a peripheral bus (such as a peripheral component interconnect). (PCI) bus, PCIe bus, etc.), system bus; the processor may be a central processing unit (central processing unit, CPU). The processor includes 102 including an integrated input / output controller (IIO), a last level cache (LLC), an integrated memory controller (iMC), and one or more processor cores (core). In the processor, IIO is used to process messages that interact with peripherals such as RNICs. The messages can be PCI messages, PCIe messages, and so on. IIO, LLC, iMC, and processor cores can be connected through the chip's internal bus. For example, if the processor is a chip of an advanced reduced instruction set machine (ARMC) architecture, then IIO, LLC, iMC, and the processor core are advanced through High performance bus (advanced high performance bus, AHB) connection.

In a computing device that uses SCM as a memory, to write data in RNIC to SCM, the data needs to pass through the processor of the computing device. Data written in RNIC to SCM can have two types of data as shown in Figure 1. Flow direction. The first type of data flow includes: ① data is written to the processor's IIO by the RNIC, ② the processor's IIO writes the data to the processor's iMC, and the processor's iMC caches the data in the asynchronous dynamic random storage of the iMC Take the memory refresh area (asynchronous DRAM refresh, ADR), ③ the processor's iMC writes the data to the SCM. The second type of data flow includes: ① data is written to the processor's IIO by RNIC, ② the processor's IIO writes the data to the processor's LLC, and ③ the data in the processor's LLC is flushed to the processor's In iMC, the processor's iMC caches the data in the iMC's ADR, and ④ the processor's iMC writes the data to the SCM. Because iMC's ADR has the feature of not losing after power failure, under normal circumstances, after data is written into the ADR, it can be considered as completing the memory persistence.

It can be known from FIG. 1 that there are two possible data flows in the processor. Among them, the flow of data in the processor is determined by the characteristics of the processor architecture. In some processor architectures, the flow of data in the processor is related to the I / O data direct I / O (DDIO) function. If the DDIO function in IIO is enabled, the data is in the processor. The flow direction is the above-mentioned second data flow direction; if the DDIO function in IIO is not enabled, the data flow in the processor is the above-mentioned first data flow direction. In other processor architectures (such as Intel ’s Skylake architecture), the data flow in the processor is related to the DDIO function in IIO and the message sent by RNIC to IIO. If the message sent by RNIC to IIO is The no-snoop (NS) flag bit is 1 or the DDIO function in IIO is not enabled, the data flow in the processor is the first data flow direction described above; if the RNIC sends an NS flag in the message to IIO If the bit is not 1, and the DDIO function of the processor is enabled, the data flow in the processor is the second data flow direction described above.

A storage system formed by a computing device using SCM as a memory may include multiple computing devices using SCM as a memory. In this storage system, remote access between two computing devices is implemented by the RNIC of the computing device. Two computing devices can be divided into two roles, local device and remote device. Among them, "local" and "remote" are two opposite concepts. Local device refers to the computing device that initiates the RDMA request, that is, to request access. Computing device of another computing device. A remote device refers to a computing device that receives an RDMA request, that is, a computing device that is accessed by another computing device. The access of the local device to the remote device can be the writing of data from the local device to the remote device. Specifically, the local device transmits the data in the local device to the RNIC of the remote device through the RNIC of the local device. The RNIC of the end device receives the data, thereby transmitting the data in the local device to the remote device. The access of the local device to the remote device may also be that the local device reads data from the remote device. Specifically, the local device can read the data in the SCM of the remote device through the RNIC of the local device. The RNIC of the device sends the data to be read by the local device to the RNIC of the local device, and the RNIC of the local device receives the data to complete reading the data in the remote device.

Remote memory persistence is implemented in two computing devices. For the above two different data flows, in some design schemes, there are different remote memory persistence processes.

For the first type of data flow, the data does not pass through the LLC of the processor. The remote memory persistence process is shown in Figure 2, including the following steps:

S201. The processor of the local device initiates a first RDMA write request to the RNIC of the local device.

S202. The RNIC of the local device processes the first RDMA write request, and generates a second RDMA write request according to the first RDMA write request.

S203: The RNIC of the local device sends a second RDMA write request to the RNIC of the remote device, and the RNIC of the remote device receives the second RDMA write request.

S204. The RNIC of the remote device writes the data in the second RDMA write request to the SCM in a DMA manner.

Here, the data flow direction of the data in the second RDMA data write request in the remote device is the first data flow direction. Because the data is transmitted on the bus, there is a certain delay. Due to the bus being occupied, some data may be The buffer is in the buffer of the intermediate medium such as IIO on the peripheral bus link.

S205: The RNIC of the remote device sends an ACK to the RNIC of the local device, and the RNIC of the local device receives the ACK.

S206. The RNIC of the local device generates a CQE corresponding to the first RDMA write request.

S207: The processor of the local device obtains a CQE corresponding to the first RDMA write request, and determines that data writing is completed.

S208. The processor of the local device initiates a first RDMA read request to the RNIC of the local device.

S209. The RNIC of the local device processes the first RDMA read request, and generates a second RDMA read request according to the first RDMA read request.

S210: The RNIC of the local device sends a second RDMA read request to the RNIC of the remote device, and the RNIC of the remote device receives the second RDMA read request.

S211. The RNIC of the remote device reads data from the SCM through DMA.

As specified in the peripheral bus protocol (such as the PCI protocol and the PCIe protocol), before any read operation is performed, all write operations before the read operation need to be completed. Therefore, reading data from the SCM through DMA can cache the data. The data in the intermediate medium on the peripheral bus link such as IIO is written into the SCM. Then, through the DMA read operation, data in the first RDMA write request that may also be buffered in the intermediate medium may be written into the SCM.

S212: RNIC of the remote device The RNIC of the remote device sends an ACK to the RNIC of the local device, and the RNIC of the local device receives the ACK.

S213: The RNIC of the local device generates a CQE corresponding to the first RDMA read request.

S214. The processor of the local device obtains a CQE corresponding to the first RDMA read request, and determines that the data persistence is completed.

For the second type of data flow, the data passes through the LLC of the processor, and the remote memory persistence process is shown in Figure 3, including the following steps:

S301. The processor of the local device initiates a first RDMA write request to the RNIC of the local device.

S302. The RNIC of the local device processes the first RDMA write request, and generates a second RDMA write request according to the first RDMA write request.

S303. The RNIC of the local device sends a second RDMA write request to the RNIC of the remote device, and the RNIC of the remote device receives the second RDMA write request.

S304. The RNIC of the remote device writes the data in the second RDMA write request to the SCM through DMA.

Data in the second RDMA write request The data flow direction of the remote device is the second data flow direction. Since the data passes through the LLC, the data is buffered in the LLC during the process of writing the data to the SCM.

S305: The RNIC of the remote device sends an ACK to the RNIC of the local device, and the RNIC of the local device receives the ACK.

S306. The RNIC of the local device generates a CQE corresponding to the first RDMA write request.

S307. The processor of the local device obtains a CQE corresponding to the first RDMA write request, and determines that data writing is completed.

S308. The processor of the local device initiates a first RDMA transmission request to the RNIC of the local device, and the first RDMA transmission request carries a persistence flag.

S309: The RNIC of the local device processes the first RDMA transmission request, generates a second RDMA transmission request according to the first RDMA transmission request, and the second RDMA request carries a persistent mark and a destination virtual memory address of the data.

S310: The RNIC of the local device sends a second RDMA sending request to the RNIC of the remote device.

S311. The processor of the remote device initiates a first RDMA reception request to the RNIC of the remote device.

S312: The RNIC of the remote device generates a WQE corresponding to the first RDMA reception request.

S313. The RNIC of the remote device generates a CQE corresponding to the first RDMA reception request according to the second RDMA transmission request.

S314: The RNIC of the remote device sends an ACK to the RNIC of the local device, and the RNIC of the local device receives the ACK.

S315. The RNIC of the local device generates a CQE corresponding to the first RDMA transmission request.

S316. The processor of the local device obtains a CQE corresponding to the first RDMA transmission request.

S317. The processor of the local device initiates a second RDMA reception request to the RNIC of the local device.

S318: The RNIC of the local device generates a WQE corresponding to the second RDMA reception request.

S319: The processor of the remote device obtains a CQE corresponding to the first RDMA reception request, and determines that data corresponding to the first RDMA reception request needs to be persisted.

S320. The processor of the remote device flashes the data of the corresponding address back to the AMC of the iMC.

S321. The processor of the remote device initiates a third RDMA transmission request to the RNIC, and the third RDMA request carries an indication of data flushing completion.

S322. The RNIC of the remote device processes the third RDMA transmission request, generates a fourth RDMA transmission request according to the third RDMA transmission request, and the fourth RDMA transmission request carries an indication of data flushing completion.

S323: The RNIC of the remote device sends a fourth RDMA transmission request to the RNIC of the local device, and the RNIC of the local device receives the fourth RDMA transmission request.

S324. The RNIC of the local device generates a CQE corresponding to the second RDMA reception request according to an indication of data flushing completion in the fourth RDMA transmission request.

S325. The processor of the local device obtains a CQE corresponding to the second RDMA receiving request, and determines that the data persistence is completed.

It can be seen that in the remote memory persistence processes shown in Figures 2 and 3, after the local device initiates an RDMA write request, in order to ensure that the data in the RDMA write request is written to the SCM, the local device needs to initiate an RDMA write After the request, an RDMA read request or an RDMA send request is initiated to write data that may not have been written to the SCM back to the SCM. The RDMA read request and RDMA send request can be called RDMA persistence requests. Since the local device initiates an RDMA persistence request after each RDMA write request, each RDMA request will occupy a certain bandwidth, and the additional RDMA persistence request increases the network load of the RDMA network.

The embodiments of the present application provide a data processing method, an RNIC, and a device to solve the problem of large network load in the remote memory persistence process shown in FIG. 2 and FIG. 3.

The embodiments of the present application can be applied to a computer network that interconnects computing devices and communicates through RDMA technology. The computer network can be as shown in FIG. 4. The computing device and the computing device can be connected and communicated through a wired network (such as Ethernet). . In this computer network, there are two roles, local device and remote device. The embodiments of the present application can be specifically applied to the communication system formed by the local device and the remote device. The communication system formed by the local device and the remote device can be as shown in FIG. 5, and the structure of the local device and the remote device can refer to the computing device shown in FIG. 1. Each computing device includes an RNIC, a processor, and an SCM. Among them, The definition of the local device and the remote device can be seen in the foregoing description. The local device and the remote device perform data transmission with the peer computing device through their respective RNICs.

The embodiments of the present application reduce the network load required by the remote memory persistence process by improving the structure of the RNIC and the remote memory persistence process.

First, an implementation manner of the RNIC provided by the embodiment of the present application is introduced. Compared with a general RNIC, the RNIC of the embodiment of the present application further includes a persistent memory (PM) module. The functions implemented by the PM module can be The hardware circuit composed of the gate array can also be implemented by a software program running in the RNIC. In the case where the control method of the control component in RNIC is a microprogram control method with micro storage as the core, the functions implemented by the PM module can be implemented by a software program running in the RNIC; the control method of the control component in the RNIC In the case of a control method mainly based on a logical wiring structure, the functions implemented by the PM module can be implemented by a hardware circuit composed of a gate array. The PM module is used to perform operations related to memory persistence. Among them, the PM module can determine whether the data in the RDMA write request needs to be persisted according to the data and parameters carried in the received RDMA write request, and determine that the data in the RDMA write request needs to be persisted according to the parameters. In the case of data storage, the processor is directly instructed to write the data in the RDMA write request to the non-volatile memory of the computing device to complete the memory persistence of the data.

In the embodiment of the present application, the non-volatile memory may be an SCM. The SCM may specifically be PRAM, and PRAM may be, for example, 3D Xpoint, ReRAM, MRAM, and so on.

FIG. 6 is a schematic structural diagram of an implementation manner of an RNIC 60 provided by an embodiment of the present application. The RNIC can be used as an RNIC of a remote device or an RNIC of a local device. As shown in FIG. 2, the RNIC 60 may include a receiving module 601, a scheduling module 602, a sending module 603, and a persistent memory module 604.

The receiving module 601 is configured to receive a message sent by an external computing device. In the case that the RNIC 60 is the RNIC of the local device, the receiving module 601 may be configured to receive a request or data sent by the RNIC of the remote device. In the embodiment of the present application, when the RNIC 60 is used as the RNIC of the local device, the receiving module 601 may be configured to execute the second interaction process between the second RNIC and the first RNIC in the method embodiment shown in FIG. 9 to FIG. 11. Receive operation performed by RNIC. In a case where the RNIC 60 is an RNIC of a remote device, the receiving module 601 may be configured to receive a request or data sent by the RNIC of the local device. In the embodiment of the present application, when the RNIC is used as the RNIC of the remote device, the receiving module 601 may be used to execute the first interaction process between the first RNIC and the second RNIC in the method embodiment shown in FIG. 9 to FIG. 11. A receive operation performed by an RNIC.

The sending module 603 is configured to send a message to an external computing device. In the case that the RNIC 60 is the RNIC of the local device, the sending module 603 may be configured to send a request or data to the RNIC of the remote device. In the embodiment of the present application, when the RNIC60 is used as the RNIC of the local device, the sending module 603 may be used to execute the second RNIC in the interaction process between the second RNIC and the first RNIC in the method embodiment shown in FIG. 9 to FIG. 11. The send operation performed. When the RNIC 60 is used as the RNIC of the remote device, the sending module 603 may be used to send a request or data to the RNIC of the local device. In the embodiment of the present application, when the RNIC60 is used as the RNIC of the remote device, the sending module 603 may be used to execute the first interaction process between the second RNIC and the first RNIC in the method embodiment shown in FIG. 9 to FIG. 11. Send operation performed by RNIC.

The scheduling module 602 is configured to communicate with a processor of a computing device and perform corresponding data processing. When the RNIC 60 is an RNIC of a local device, the scheduling module 602 is configured to communicate with a processor of the local device and perform data processing. In the embodiment of the present application, when the RNIC 60 is used as the RNIC of the local device, the scheduling module 602 may be configured to perform an operation performed by the second RNIC in the method embodiment shown in FIG. 9 to FIG. 11 to interact with the second processor or Operations related to scheduling. When the RNIC 60 is used as the RNIC of the remote device, the scheduling module 602 is configured to communicate with the processor of the remote device and perform data processing. In the embodiment of the present application, when the RNIC60 is used as the RNIC of the remote device, the scheduling module 602 may be configured to perform operations performed by the first RNIC in the method embodiments shown in FIG. 9 to FIG. 11 to interact with the first processor. Or scheduling-related operations.

The PM module 604 is used to perform operations related to memory persistence. In the embodiment of the present application, for the above-mentioned first data flow, the PM module 604 is configured to determine whether the first data in the RDMA write request is data to be persisted when the receiving module 601 receives the RDMA write request. When it is determined that the first data in the RDMA write request is data to be persisted, an RDMA LSB read request is added to the SQ corresponding to the RDMA write request, so that the scheduling module 603 can send the RDMA LSB read request to the processing A processor, so that the processor writes the data buffered on the peripheral bus link into the non-volatile memory, thereby writing the first data into the non-volatile memory. In the embodiment of the present application, for the above-mentioned second data flow, the PM module 604 is configured to determine whether the first data in the RDMA write request is data to be persisted when the receiving module 601 receives the RDMA write request. When it is determined that the first data in the RDMA write request is data to be persisted, a WQE corresponding to the RDMA reception request is generated according to the first data in the RDMA write request, and then the WQE is cleared, and then the RDMA reception is generated The corresponding CQE is requested, so that the processor corresponding to the RNIC 60 (that is, the processor of the remote device) can write the first data buffered in the volatile storage medium back to the non-volatile memory according to the CQE.

In a possible implementation manner, the functions implemented by the receiving module 601, the scheduling module 602, the sending module 603, and the persistent memory module 604 may be performed by the operation logic component 701, the register 702, the control component 703, and the input / output interface 704. In cooperation, as shown in FIG. 7, the operation logic component 701, the register 702, the control component 703, and the input / output interface 704 may be connected through one or more internal buses 705. The operation logic unit 701 can be used to execute operation commands, such as addition, subtraction, multiplication, division, and so on; the operation logic unit 701 can also be used to obtain logical commands such as OR logic commands, AND logic commands, and non-logic Commands, etc .; the logic operation unit 701 may be further configured to obtain a control signal from the control unit 703, obtain data corresponding to the control signal from the register 702 according to the obtained control signal, and perform a corresponding operation. Register 702 is a kind of memory with a small storage space. Register 702 can be used to store various instructions. Register 702 can also be used to store register operands and intermediate or final operation results temporarily stored during instruction execution. It may be used to store data used by the logic operation unit 701 to complete the task requested by the control unit 703. The control unit 703 is used to decode the instructions stored in the register and send out control signals to complete each operation to be performed by each instruction. There are two control modes of the control unit 703. One is to use micro memory as the core. The micro-program control method can be stored in the register 702, and the other is a hardware control method mainly based on a logical hard-wired structure. The control unit 703 can be composed of various AND-OR arrays, for example. The input-output interface 704 is used to send or receive data. There may be multiple input-output interfaces 704, which can be used to receive data sent by the processor or send data to the processor, or used to receive data or send data from an external computing device. Data to external computing devices.

Optionally, the RNIC may further include a crystal oscillator, a media access controller, a physical interface transceiver, and the like, which are not limited in the embodiments of the present application.

Based on the RNIC described in the foregoing embodiments corresponding to FIG. 6 and FIG. 7, the data processing method in this embodiment of the present application can be implemented.

Before introducing the data processing method of the embodiment of the present application, for ease of understanding, some concepts involved in the embodiment of the present application are first introduced.

1.The concept of RDMA operation

1) RDMA unilateral operation

RDMA unilateral operation means that when the application on the local device accesses the memory of the remote device, only the processor of the local device participates, and the processor of the remote device is not required to participate, that is, only one side of the processor is working. As long as the local device knows the source and destination addresses of the data, it can complete the reading and writing of data in the memory of the remote device. In the embodiment of the present application, the RDMA unilateral operation mainly involves an RDMA write operation (RDMA-Write).

2) RDMA bilateral operation

RDMA bilateral operation means that when the application on the local device accesses the memory of the remote device, the processor of the local device and the processor of the remote device are required to participate, that is, the processors of both devices are working. In the embodiment of the present application, the RDMA bilateral operations mainly involved include an RDMA send operation (RDMA-Send) and an RDMA receive operation (RDMA-Receive). If the local device is to transmit data to the memory of the remote device through the RDMA send operation, the remote device must first initiate an RDMA receive operation to receive the RDMA send operation initiated by the local device.

2.The concept of queues

The queue in the embodiment of the present application is similar to the concept of a message queue in socket communication. It can be understood that the queue is a container for storing various information or data for asynchronous processing.

1) Work queue (work queue, WQ)

In RDMA technology, when two computing devices need to communicate, a channel connection is established between the RNICs of the two computing devices, and the end and end of each channel are two pairs of queues (QPs). Exemplarily, the channel between the RNICs of the two computing devices may be as shown in FIG. 8. Each pair of QPs is composed of a send queue (SQ) and RQ. SQ and RQ manage various types of messages. The QP is directly mapped to the virtual address space of the application (client) of the computing device, so that the application in the computing device can directly access the RNIC through it. Both SQ and RQ can be called WQ. For a computing device to send data, WQ is SQ; for a computing device to receive data, WQ is RQ.

An application in a computing device can create a work request (WR) to notify a certain WQ in QP by using WR. WR describes the remote operation request of the application (such as remote read operation request, remote write operation request, etc.), so that The RNIC of the computing device can determine the operations to be scheduled and executed. In WQ, WR is converted to WQE format, waiting for RNIC to schedule and analyze it. For example, the application of computing device A wants to transfer the content stored in address A to address B (address A is the address in computer A, address B is the address in computer B), and the application uses address WR to address A and address B and the write instruction inform the RNIC of computer A. The RNIC of computer A adds WQE to the SQ. The WQE includes the address A, address B, and the write instruction.

An application of a computing device may send an RDMA request as a WR to the RNIC of the computing device. After receiving the RDMA request, the RNIC of the computing device adds a WQE corresponding to the RDMA request to the WQ. For RDMA requests of the transmission type (such as RDMA read requests, RDMA write requests, and RDMA transmission requests), the WQE corresponding to the RDMA requests of this transmission type is added to the SQ, as shown in Figure 8; for RDMA requests of the reception type (such as RDMA reception request), the WQE corresponding to the RDMA request of the reception type is added to the RQ.

2) completion queue (completion queue entry, CQ)

In addition to QP, there is a queue in RDMA technology. For completion queue (completion queue entry, CQ), CQ is used to store the completion event corresponding to an operation to inform the upper-layer application. For example, the application of computer A wants to transfer the content stored in address A to address B (address A is the address in computer A, address B is the address in computer B), and the RNIC of computer A sends the content of address A to Computer B ’s RNIC and determined that computer B ’s RNIC received the content, computer A ’s RNIC generates a CQE, and after computer A ’s application obtains the CQE, it determines that the content stored in address A is transferred to address B. .

Next, a data processing method according to an embodiment of the present application is described. Referring to FIG. 9, FIG. 9 is a schematic flowchart of a data processing method according to an embodiment of the present application. The first RNIC and the first processor are the RNIC and the processor of the first device, respectively, and the first device is the remote device. ; The second RNIC and the second processor are the RNIC and the processor of the second device, respectively, and the second device is a local device. As shown, the method includes:

S801. The second processor sends an RDMA write persistence request to the second RNIC. The second RNIC receives the RDMA write persistence request. The RDMA write persistence request includes a data persistence flag.

The RDMA write persistence request can be understood as a WR created by the second processor and is used to describe the remote operation request of the second processor. In the embodiment of the present application, the RDMA write persistence request is used to request that the first data be stored in the non-volatile memory of the first device to complete the memory persistence of the first data.

The RDMA write persistence request may include an RDMA operation instruction, a source virtual memory address of the first data, and a destination virtual memory address of the first data. The source virtual memory address of the first data is a virtual memory address of the first data in the second device, and a storage space corresponding to the source virtual memory address is used to store the first data in the second device. The destination virtual memory address of the first data is a virtual memory address in the first device, and a storage space corresponding to the destination virtual memory address is used to store the first data after the first data is written to the first device. The destination virtual memory address is a virtual memory address registered by the first processor through a virtual memory address registration process. For example, the source virtual memory address and the destination virtual memory address are described. For example, the first data is stored in a storage space corresponding to the virtual memory address A of the first device, and the second processor saves the first data to the first In the storage space corresponding to the virtual address memory address B of the two devices, the source virtual address of the first data is the virtual memory address A in the first device, and the destination virtual memory address of the second data is the virtual memory in the second device. Address B.

Here, the virtual memory address is a logical storage address that has a mapping relationship with a physical address in the memory of the computing device, and is used to implement isolation between programs in the computing device and ensure the normal operation of the program. In a computing device, after an application is compiled, it will form multiple subroutines. The addresses of these subroutines usually start with "0", and other addresses in the subroutine are relative to the starting address (that is, " 0 ”) is calculated, the address range formed by these addresses is called the address space, and the addresses in the address space are logical storage addresses; these addresses correspond to the addresses of the storage space of the computing device ’s memory storage space The resulting address range is called memory space, and the addresses in the memory space are physical addresses. In the case of multiple subprograms running at the same time, the addresses of these subprograms need to be loaded from address "0". Because there is only one physical address of 0 in the computer, some subprograms cannot be loaded from "0". In this way, the logical storage address in the address space and the physical address in the memory space are inconsistent. For example, if subroutine A needs to be loaded from logical storage address 0, but actually loads from physical address 10, then Through the address mapping, the logical storage address in the address space is converted into the corresponding physical address in the memory space.

In RDMA technology, the first processor allocates a piece of storage space in memory in advance. This storage space is used to store data related to RDMA operations, and then sends the target page table to the RNIC through the virtual address address registration process. The correspondence between the virtual memory address and the physical memory address corresponding to the storage space allows the RNIC to determine the virtual memory address and the physical memory address corresponding to the storage space used to store data related to the RDMA operation according to the target page table. The RNIC can also determine the physical memory address corresponding to a virtual memory address according to the target page table. For example, the processor uses the storage space in the memory space with a physical address of 1001 to 3000 to store data related to the RDMA operation, and the corresponding virtual memory address is 1 to 2000. For example, the processor will be as shown in Table 1. The page table shown is sent to RNIC.

Table 1

虚拟内存地址Virtual memory address	物理地址Physical address
11	10011001
22	10011001
…...	…...
20002000	30003000

In the process of registering the virtual memory address, in addition to sending the target page table to the RNIC, the processor also specifies the access mark of the registered address space. The access mark is used to indicate the attributes of the storage space corresponding to the address space, such as the address space. The attribute of the corresponding storage space is remote read, that is, the storage space is a storage space that can read data, and if the attribute of the corresponding storage space of the address space is remote writable, that is, the storage Space is a storage space where data can be written, and the storage space corresponding to the address space is remotely readable and writable, that is, the storage space is a storage space that can both read data and write data, etc. Wait. During the process of registering the virtual memory address, the processor sends the initial virtual memory address of the address space, the length of the address space, and the access mark of the address space to the RNIC, so as to inform the RNIC of the attributes of the storage space corresponding to each address space and its contents. Virtual memory address. For example, for example, the processor uses the storage space in the memory space with a physical address of 1001 to 3000 to store data related to the RDMA operation, and the corresponding virtual memory address is 1 to 2000. A storage space corresponding to an address space with a memory address of 1 to 500 is designated as a readable storage space, a storage space corresponding to an address space of 501 to 1000 is designated as a writable storage space, and a storage space corresponding to an address space of 1001 to 1500 is designated The space is designated as a readable and writable storage space, and the processor may send the information shown in Table 2 to the RNIC.

Table 2

起始虚拟内存地址Starting virtual memory address	地址空间的长度Address space length	访问标记Access token
11	500500	可读Readable
501501	500500	可写Writable
10011001	500500	可读并且可写Readable and writable

After the virtual memory address registration process is performed, the first device may send the related information of the address space registered during the virtual memory address registration process to the second device through an RDMA sending operation, so that the second device can learn the use of the first device. A virtual memory address corresponding to a storage space storing data related to the RMDA operation. The related information of the address space includes a starting virtual memory address of the registered address space, a length of the address space, and an access mark.

In the embodiment of the present application, the data persistence flag is used to indicate that the first data is data to be persisted. According to the network characteristics of the RDMA network, data persistence marking can have the following two possible situations:

1) In the RDMA network, the RNIC of the computing device performs the corresponding operation according to the RDMA operation instruction obtained by the analysis. Based on this setting, in the first possible implementation of the embodiment of the present application, a write persistence instruction may be added to the RDMA operation instruction, and the write persistence instruction indicates that the first data needs to be written and stored in memory Persistent data, that is, data persistence is marked as a write persistence instruction.

When the data persistence is marked as a write persistence instruction, the RDMA write persistence request may include a write persistence instruction, a source virtual memory address of the first data, and a destination virtual memory address of the first data.

2) In a second possible implementation of the embodiment of the present application, the data persistence flag may also be a destination storage address corresponding to the first data, where a storage space corresponding to the destination storage address is used to store the first data, The destination storage address is a persistent storage address in the first device, and a storage space corresponding to the persistent storage address is used to store data to be persisted.

As mentioned earlier, the processor of the first device specifies the address space corresponding to the storage space for storing data related to the RDMA operation and the access mark of the address space through the virtual memory address registration process, and performs virtual address registration After the process, the first device sends the information registered in the virtual memory address registration process to the second device through an RDMA sending operation. Based on this setting, a remote write persistence mark can be added to the access mark. The remote write persistence mark indicates that the storage space corresponding to the address space is used to store data to be persisted. Then, the destination storage address may be a destination virtual memory address, and the destination virtual memory address is a logical storage address in an address space where an access mark registered by the first processor through the virtual memory address registration process is a write persistence mark. Because the destination storage address is a logical storage address in the address space registered by the first processor through the virtual memory address registration process and the access mark is a write-persistent mark, the storage space corresponding to the address space marked by the write-persistent mark is used For storing the data to be persisted, the storage space corresponding to the destination virtual memory address is used to store the data to be persisted.

When the data persistence is marked as the destination virtual memory address of the first data, the RDMA write persistence request may include a write instruction, a source virtual memory address of the first data, and a destination virtual memory address of the first data. The memory address is a logical storage address in an address space registered as a write-persistent mark for an access mark registered by the first processor through a virtual memory address registration process.

S802. The second RNIC generates an RDMA write request according to the RDMA write persistence request. The RDMA write request includes the first data and a data persistence mark.

In a specific implementation manner, the second RNIC may create a WQE corresponding to the RDMA write persistence request in the SQ, and the WQE corresponding to the RDMA write persistence request may include a source virtual memory address of the first data and an RDMA operation instruction. And the destination virtual memory address of the first data; then when dispatching to the WQE corresponding to the RDMA write persistence request, obtaining the first data from the source virtual memory address according to the source virtual memory address of the first data, The data, the destination virtual memory address of the first data, and the RDMA operation instruction are encapsulated in an RDMA transmission message to form an RDMA write request. The RDMA transmission message refers to a message transmitted between the first RNIC and the second RNIC. The operation of generating a RDMA write request by the second RNIC according to the RDMA write persistence request may be specifically performed by a scheduling module of the second RNIC.

If the data persistence is marked as a write persistence instruction, the RDMA write request may include a write persistence instruction, a destination virtual memory address of the first data, and the first data, and the RDMA write request may also be referred to as a write persistence request; If the data persistence is marked as the destination virtual memory address of the first data, the RDMA write request may include a write instruction, the destination virtual memory address of the first data, and the first data, and the destination virtual memory address of the first data is the first The persistent virtual memory address that the processor registers with the virtual memory address.

Optionally, the RDMA write request may further include a serial number of the first data and a QP serial number of the first device. The serial number of the first data packet is used to uniquely identify the first data during the transmission between the first device and the second device, which is convenient for detecting missing or duplicate data packets; the QP serial number of the remote device is used to identify the local device And the only channel between the remote device.

S803: The second RNIC sends an RDMA write request to the first RNIC, and the first RNIC receives the RDMA write request.

In the embodiment of the present application, the second RNIC may send an RDMA write request to the first RNIC based on an InfiniBand (IB) protocol; the second RNIC may also be based on a converged Ethernet remote direct memory access (RDMA over Converged Ethernet, RoCE) The protocol sends the RDMA write request to the first RNIC; the second RNIC may also send the RDMA write request to the first RNIC based on the remote direct memory access (iWARP) protocol of the transmission control protocol.

In specific implementation, the second RNIC sends the RDMA write request to the first RNIC through the sending module of the second RNIC, and the first RNIC receives the RDMA write request through the receiving module of the first RNIC.

After receiving the RDMA write request, the receiving module of the first RNIC puts the RDMA write request into the RQ corresponding to the first RNIC, and waits for the scheduling module of the first RNIC to schedule it. When the scheduling module of the first RNIC dispatches the RDMA write request, the scheduling module of the first RNIC parses the RDMA write request to obtain first data, a destination virtual memory address of the first data, and an RDMA operation instruction.

When the data persistence is marked as a write persistence instruction, the scheduling module of the first RNIC may send the write persistence instruction to the PM module of the first RNIC, and the PM module of the first RNIC determines the first data according to the write persistence instruction. For data to be persisted, step S805 is performed; the scheduling module of the first RNIC determines that the first data needs to be written to the destination virtual memory address according to the destination virtual memory address of the first data and the write persistence instruction. The scheduling module of the RNIC executes step S804.

In the case where the data persistence is marked as the destination virtual memory address of the first data, the scheduling module of the first RNIC may send the destination virtual memory address of the first data to the PM module, because the destination virtual memory address of the first data is the first The access token registered by a processor through the virtual memory address registration process is a logical storage address in the address space of the remote write persistent tag. The PM module of the first RNIC determines that the first data is data to be persisted, and executes step S805; first The scheduling module of the RNIC determines that the first data needs to be written into the destination virtual memory address according to the destination virtual memory address and the write instruction of the first data, and the scheduling module of the first RNIC executes step S804.

S804. The first RNIC sends a DMA write request to the first processor. The first processor receives the DMA write request, and the DMA write request includes the first data.

The operation that the first RNIC sends a DMA write request to the first processor is performed by a scheduling module that can be the first RNIC.

Here, when the first RNIC sends a DMA write request to the first processor, the first data is written into the non-volatile memory of the first device in a DMA manner.

S805. The first RNIC instructs the first processor to save the first data to a non-volatile memory of the first device.

It can be seen from the steps shown in FIG. 9 that after the RNIC of the remote device receives the RDMA write request, it is determined that the data in the RDMA write request is data for memory persistence according to the data persistence mark in the RDMA write request. Directly instructs the first process to save the first data to the non-volatile memory of the first device, which is equivalent to combining the RDMA write request and the RDMA persistence request into one request, eliminating the need for the remote device to initiate RDMA The operation of persistent requests, because each request requires a certain amount of network bandwidth, eliminating the need for the remote device to initiate an RDMA persistent request operation, which saves the bandwidth occupied by a request and reduces the load on the RDMA network. Write requests and RDMA persistence requests are combined into one request, which is equivalent to turning data write operations and memory persistence operations into two operations that need to be performed continuously, ensuring that data can be saved to the remote end after it is written to the remote device The non-volatile memory of the device avoids data inconsistencies.

It can be known from the foregoing that the data received by an RNIC is written into the non-volatile memory of the computing device corresponding to the RNIC, and the data passes through the processor corresponding to the RNIC. Due to the bus being occupied and other reasons, the data may be Temporarily cached in all levels of the processor's cache. Due to different architecture characteristics of the processor, there are two types of data flow in the processor, that is, the first data has two different data flows in the first processor. In the embodiment of the present application, in combination with different data flows described in FIG. 1, the first RNIC instructs the first processor to save the first data to the non-volatile memory of the first device in different ways. The flow of two different data flow corresponding data processing methods will be specifically introduced. See Figures 10-11.

FIG. 10 is a schematic flowchart of another data processing method according to an embodiment of the present application. The process is applicable to a case where the data flow is the first data flow direction described above, where the first RNIC and the first processor are the first device, respectively. The first device is a remote device; the second RNIC and the second processor are the RNIC and the processor of the second device, and the second device is a local device. As shown, the method includes the following processes:

S901. The second processor sends an RDMA write persistence request to the second RNIC. The second RNIC receives the RDMA write persistence request. The RDMA write persistence request includes a data persistence flag.

S902. The second RNIC generates an RDMA write request according to the RDMA write persistence request. The RDMA write request includes the first data and a data persistence mark.

S903. The second RNIC sends an RDMA write request to the first RNIC, and the first RNIC receives the RDMA write request.

S904. The first RNIC sends a DMA write request to the first processor. The first processor receives the DMA write request, and the DMA write request includes the first data.

Here, for the implementation of steps S901 to S904, reference may be made to the description of steps S801 to S804, and details are not described herein again. The PM module of the first RNIC determines that the first data is data to be persisted according to the persistence flag, and the PM module of the first RNIC performs step S905.

S905. The first RNIC adds a DMA LSB read request to the RQ corresponding to the second RNIC.

Here, the PM module of the first RNIC may determine the RQ corresponding to the second RNIC according to the QP of the first device in the RDMA write request, and then add a DMA LSB read request to the RQ. A DMA LSB read request is a request corresponding to a DMA read operation, and the virtual memory address corresponding to the DMA LSB read request may be a logical storage address in any segment of the address space registered by the first processor through the virtual memory address registration process. The virtual memory address corresponding to the DMA LSB read request refers to the virtual memory address corresponding to the storage space to be read by the DMA LSB read operation corresponding to the DMA LSB read request.

As a possible embodiment, in addition to the first RNIC instructing the first processor to persist the data through a DMA LSB read request, the first RNIC may also instruct the first processor to send another DMA read request to the first processor. All data buffered in the peripheral bus link of the first device is written to the non-volatile memory of the first device. The above DMA read request may be a read request to read an arbitrary address in the SCM, and the address read by the read request may be any address in the SCM. For example, the address is a starting address range used to store the first data. Start address, end address, or any address other than start address and end address. Optionally, the address to be read by the read request may also be an address segment. For example, the address segment is an address range for storing the first data, or any one of the storage ranges in the SCM.

After acquiring the DAM LSB read request, the scheduling module of the first RNIC executes step S906.

S906. The first RNIC sends a DMA LSB read request to the first processor, and the first processor receives the DMA LSB read request.

Here, if the architecture of the first processor is the flow of data in the processor to the processor architecture related to the DDIO function in IIO and the message sent by RNIC to IIO, the first RNIC sets the NS flag in the DMA LSB read request. Is 1.

By issuing a DMA LSB read request to the first processor, all write operations of the first processor before the DMA LSB read request can be completed, so that all data that has not been written to the non-volatile memory is written to non-volatile memory. In the volatile memory, it is guaranteed that the first data written by the DMA write request can be written into the non-volatile memory before the DMA LSB read request is initiated, and the memory persistence of the first data is completed.

Further, after completing the memory persistence of the first data, the first RNIC may notify the second RNIC of the completion of the memory persistence of the first data by sending a confirmation message. The method shown in FIG. 10 may further include:

S907: The first RNIC sends a persistence confirmation message corresponding to the first data to the second RNIC, and the second RNIC receives the persistence confirmation message corresponding to the first data.

The persistence confirmation message corresponding to the first data includes a second PSN, and the second PSN is a sequence number of the first data.

In specific implementation, the first RNIC sends a persistence confirmation message corresponding to the first data to the second RNIC through the sending module of the first RNIC, and the second RNIC receives the persistence confirmation corresponding to the first data through the receiving module of the second RNIC. Message.

The scheduling module of the first RNIC obtains the serial number of the first data from the RDMA write request, and as the second PSN, encapsulates the second PSN into the RDMA transmission message to form a confirmation message, and sends it to the sending module of the first RNIC to Second RNIC. After receiving the confirmation message, the receiving module of the second RNIC sends the confirmation message to the scheduling module of the second RNIC. The scheduling module of the second RNIC parses the confirmation message to obtain a second PSN, and determines the confirmation message according to the second PSN. A persistent confirmation message corresponding to the first data.

S908. The second RNIC generates a CQE corresponding to the RDMA write persistence request.

Here, the scheduling module of the second RNIC determines that the first data has been stored in the non-volatile memory of the first device according to the persistence confirmation message corresponding to the first data, and the scheduling module of the second RNIC determines the first according to the second PSN WQE corresponding to the data, obtain the content in the WQE, and generate the CQE corresponding to the RDMA write persistence request. The CQE corresponding to the RDMA write persistence request may include the source virtual memory address of the first data and / or the second PSN and / or the destination. Virtual memory address.

S909. The second processor obtains a CQE corresponding to the RDMA write persistence request, and determines that the first data has been stored in the non-volatile memory of the first device.

Here, the second processor may determine that the first data has been stored in the non-volatile memory of the first device according to the content in the CQE. For example, the content of the CQE is the source virtual address of the first data, and the second processor determines that the data stored in the source virtual memory address has been stored in the non-volatile memory of the first device, that is, the first data has been stored in In the non-volatile memory of the first device. After the first data is stored in the non-volatile memory of the first device, the memory persistence of the first data is completed.

As can be seen from the steps shown in FIG. 10, in the embodiment of the present application, for the case where the data does not pass through the LLC of the processor, the embodiment of the present application only needs to initiate an RDMA write request to the RDMA write request in the embodiment of the present application. The data is written to the non-volatile memory of the remote device. Compared with the process shown in Figure 2, one unilateral operation is omitted, the load of the processor of the local device is reduced, and the initiation is saved. The bandwidth occupied by RDMA persistent requests reduces the network load.

FIG. 11 is a schematic flowchart of another data processing method according to an embodiment of the present application. The process is applicable to a case where the data flow is the second data flow direction, where the first RNIC and the first processor are the first device, respectively. The first device is a remote device; the second RNIC and the second processor are the RNIC and the processor of the second device, and the second device is a local device. As shown, the method includes the following processes:

S1001. The second processor sends an RDMA write persistence request to the second RNIC. The second RNIC receives the RDMA write persistence request. The RDMA write persistence request includes a data persistence flag.

S1002. The second RNIC generates an RDMA write request according to the RDMA write persistence request. The RDMA write request includes the first data and a data persistence mark.

S1003. The second RNIC sends an RDMA write request to the first RNIC, and the first RNIC receives the RDMA write request.

S1004. The first RNIC sends a DMA write request to the first processor. The first processor receives the DMA write request, and the DMA write request includes the first data.

Here, for the implementation of steps S1001 to S1004, reference may be made to the description of steps S801 to 804 above, and details are not described herein again. After the PM module of the first RNIC determines that the first data is data to be persisted according to the data persistence flag, steps S1006 to S1007 are performed.

S1005. The first processor sends a first RDMA reception request to the first RNIC, and the first RNIC receives the first RDMA reception request.

Because the data passes through the LLC in the first processor, the first processor is required to write the data in the LLC back to the non-volatile memory, that is, the first processor is required to participate, and the first processor is a remote device Processor, writing data from LLC back to non-volatile memory involves RDMA bilateral operations. For the RDMA bilateral operation, the remote device must first initiate an RDMA receive operation, then the first processor sends a first RDMA receive request to the second RNIC for receiving the RDMA send request sent by the second device.

S1006. The second RNIC generates a WQE corresponding to the first RDMA reception request according to the first data.

In a possible implementation manner, the WQE corresponding to the first RDMA reception request may include a destination virtual memory address of the first data; in another possible implementation manner, the CQE corresponding to the first RDMA reception request may include The serial number of the first data; in another possible implementation manner, the CQE corresponding to the first RDMA reception request may include a destination virtual memory address of the first data and a serial number of the first data.

S1007. The second RNIC clears the WQE corresponding to the first RDMA reception request, and generates a CQE corresponding to the first RDMA reception request.

In a possible implementation manner, the PM module of the second RNIC may obtain the WQE content from the WQE corresponding to the first RDMA reception request, and then generate the CQE corresponding to the first RDMA reception request according to the acquired WQE content. The CQE corresponding to the first RDMA reception request may include a destination virtual memory address of the first data and / or a serial number of the first data.

In steps S1006 to S1007, the operation performed by the PM module of the second RNIC replaces the operation of the RDMA transmission request initiated by the second device, and the effect achieved is the same as that of the second device by initiating the RDMA transmission request to instruct the first data to be stored in The effect in the non-volatile memory of the first device is the same.

S1008. The first processor obtains a CQE corresponding to the first RDMA reception request, and stores the first data in a non-volatile memory of the first device.

In the case that the CQE corresponding to the first RDMA reception request includes the destination virtual memory address of the first data, the first processor may find the first data in the LLC according to the destination virtual memory address of the first data, thereby converting the first data Data is stored in the non-volatile memory of the first device; in the case that the CQE corresponding to the first RDMA reception request includes the serial number of the first data, the first processor may be in the LLC according to the serial number of the first data The first data is found, thereby storing the first data in a non-volatile memory of the first device.

In an optional implementation manner, after receiving the first data, the first RNIC may send a confirmation message to the second RNIC to inform the second RNIC that the first data is received, and after step S1003, the method further includes:

S1009: The first RNIC sends a reception confirmation message corresponding to the first data to the second RNIC, and the second RNIC receives a reception confirmation message corresponding to the first data.

Here, the reception confirmation message corresponding to the first data includes a first PSN, the first PSN is a serial number of the first data, and the reception confirmation message corresponding to the first data is used to indicate that the first RNIC receives the first data.

In specific implementation, the first RNIC sends a reception confirmation message corresponding to the first data to the second RNIC through the sending module of the first RNIC, and the second RNIC receives the reception confirmation message corresponding to the first data through the receiving module of the second RNIC.

The scheduling module of the first RNIC obtains the serial number of the first data from the RDMA write request. As the first PSN, the first PSN is encapsulated into an RDMA transmission message to form a confirmation message, and is sent to the first RNIC sending module to Second RNIC. After receiving the confirmation message, the receiving module of the second RNIC sends the confirmation message to the scheduling module of the second RNIC. The scheduling module of the second RNIC parses the confirmation message to obtain the first PSN, and the scheduling module of the second RNIC according to the first A PSN determines that the confirmation message is a reception confirmation message corresponding to the first data.

S1010. The second RNIC generates a CQE corresponding to the RDMA write persistence request.

S1011: The second processor obtains a CQE corresponding to the RDMA write persistence request, and determines that the first data transmission is completed.

For specific implementations of steps S1010 to S1011, reference may be made to the description of steps S908 to S909, and details are not described herein again.

In the foregoing steps, the first RNIC initiates an RDMA transmission request operation instead of the second device, and then the second processor may initiate an RDMA reception request after determining that the first data transmission is completed, so as to receive the RDMA transmission request initiated by the first processor. , After step S1011, the method may further include:

S1012: The second processor initiates a second RDMA reception request to the second RNIC, and the second RNIC receives the second RDMA reception request.

S1013: The second RNIC generates a WQE corresponding to the second RDMA reception request.

In an optional implementation manner, after the first processor stores the first data in the non-volatile memory of the first device, the first RNIC may notify the second RNIC by sending a confirmation message to inform that the memory of the first data is persistent. After the change is completed, after step S1008, the method may further include:

S1014: The first processor sends an RDMA transmission request to the first RNIC, and the first RNIC receives the RDMA transmission request.

Here, the RDMA transmission request is used to indicate that the first data has been stored in the non-volatile memory of the first device, and the first RDMA transmission request may include a serial number of the first data.

S1015: The first RNIC generates a WQE corresponding to the RDMA transmission request.

S1016: The first RNIC sends a persistence confirmation message corresponding to the first data to the second RNIC, and the second RNIC receives the persistence confirmation message corresponding to the first data.

Here, the persistence confirmation message corresponding to the first data includes a second PSN, and the second PSN is a sequence number of the first data.

The scheduling module of the first RNIC obtains the sequence number of the first data packet from the first RDMA transmission request as the second PSN, and then encapsulates the second PSN into the RDMA transmission message to form a confirmation message, and passes the first RNIC's The sending module sends to the second RNIC. After receiving the confirmation message, the receiving module of the second RNIC sends the confirmation message to the scheduling module of the second RNIC. The scheduling module of the second RNIC parses the confirmation message to obtain a second PSN. The scheduling module of the second RNIC determines the The confirmation message is a persistent confirmation message corresponding to the first data.

When receiving the persistence confirmation message corresponding to the first data sent by the first RNIC, the second RNIC executes step S1018.

S1017: The first RNIC clears the WQE corresponding to the RDMA transmission request, and generates a CQE corresponding to the RDMA transmission request.

S1018: The second RNIC generates a CQE corresponding to the second RDMA reception request.

S1019: The second processor obtains a CQE corresponding to the second RDMA reception request, and determines that the first data has been stored in the non-volatile memory of the first device.

After the first data is stored in the non-volatile memory of the first device, the memory persistence of the first data is completed.

It can be seen from the steps shown in FIG. 11 that in the embodiment of the present application, for the case where data passes through the LLC of the processor of the remote device, the processor of the local device only needs to send the bilateral operation once after sending the unilateral operation The data can be saved to the non-volatile memory of the remote device. Compared with the process shown in Figure 3, a bilateral operation is omitted, the load on the processor of the local device is reduced, and the initiation is saved. The bandwidth occupied by RDMA persistent requests reduces the network load.

The processes shown in FIG. 10 and FIG. 11 above are related to a process of saving one data to the non-volatile memory of the first device. The above solution can also be used to save multiple data to the first device. Volatile memory. The RDMA transmission protocol stipulates that the RNIC of the local device must send the next data to the RNIC of the remote device only after determining that the previous data transmission process has been completed, that is, the RNIC of the local device receives the RNIC sent from the remote device. Only the ACK message for the previous data will send the next data to the remote device. In a possible implementation manner, the memory persistence process of the former data and the write process of the latter data of the two data can be performed in parallel to improve the transmission efficiency.

A specific implementation method for making the memory persistence process of the former data and the write process of the latter data of the two data in parallel can be: after receiving the RDMA write request, the first RNIC sends a second data to the second RNIC. Confirmation message, the second confirmation message carries the PSN of the first data; after receiving the second confirmation message, the second RNIC receives the confirmation message of the PSN whose PSN is the first data for the first time, and the second RNIC according to the The PSN of the first data determines that a reception confirmation message corresponding to the first data is received, and further determines that the first RNIC receives the first data, and the second RNIC sends an RDMA write request corresponding to the next data of the first data to the first RNIC; In addition, the second RNIC buffers the second confirmation message. When receiving the first confirmation message of the PSN of which the PSN is the first data, the second RNIC caches the PSN of the first data in the first confirmation message and the previous cache. The PSN of the first data in the second confirmation message is determined to be the second time that the PSN for which the PSN is the first data is received, and the second RNIC determines to receive the persistent confirmation corresponding to the first data, and executes Said step S908 or S1018.

In the RDMA transmission protocol, the acknowledgment message is only used to indicate the meaning of acknowledgment, and the acknowledgment message is specifically used to indicate that the confirmation of what request needs to be performed according to the request sent or the order of the received confirmation message It is judged that because the write operation occurs before the memory persistence operation, the first confirmation message carrying the serial number of the first data must be the first data reception confirmation message. After receiving the first data reception After the acknowledgement message is buffered, the RNIC of the second device can send the next data of the first data to the first device RNIC without waiting for the persistent acknowledgement message of the first data to be received. The way of confirming the PSN in the message can make one data's memory persistent and the next data's write operation can be performed in parallel, improving the efficiency of saving multiple data in the non-volatile memory of the remote device, Reduce latency.

As a possible embodiment, the RDMA write request sent by the second device may not include a data persistence flag. When the first device receives the RDMA write request, the first RNIC in the first device is communicating with the second device. A DMA read request is added to the receiving queue corresponding to the second RNIC, and the DMA read request is sent to the first processor, instructing the first processor to store the first data to the SCM. All write operations of the first processor before the DMA read request can be completed, so that all data that has not been written to the non-volatile memory is written to the non-volatile memory, and guaranteed to pass before the read request is initiated The first data written by the DMA write request can be written into the non-volatile memory, and the memory persistence of the first data is completed.

The foregoing method may be implemented on a RNIC and a processor of a computing device. In order to facilitate better implementation of the foregoing method in the embodiment of the present application, the embodiment of the present application further provides a corresponding computing device.

12, FIG. 12 is a schematic structural diagram of a computing device according to an embodiment of the present application. The computing device 130 includes an RNIC 131, a processor 132, and a non-volatile memory 133. The structure of the RNIC 131 may be as shown in FIG. 6, and the structure of the processor 132 may be shown as the processor 102 in FIG. 1. The non-volatile memory 133 may be an SCM.

The RNIC 131 is configured to execute the steps performed by the first RNIC in the method embodiments shown in FIG. 9 to FIG. 11, and the processor 132 is configured to perform the steps performed by the first processor in the method embodiments shown in FIG. 9 to FIG. 11. A step of.

Referring to FIG. 13, FIG. 13 is a schematic structural diagram of a computing device according to an embodiment of the present application. The computing device 140 includes an RNIC 141, a processor 142, and a non-volatile memory 143. The structure of the RNIC 141 may be as shown in FIG. 6, and the structure of the processor 142 may be shown as the processor 102 in FIG. 1. The non-volatile memory 133 includes SCM, NVRAM, and NVDIMM.

The RNIC 141 is configured to execute the steps performed by the second RNIC in the method embodiments shown in FIG. 9 to FIG. 11, and the processor 142 is configured to execute the steps performed by the second processor in the method embodiments shown in FIG. 9 to FIG. 11. A step of.

An embodiment of the present application further provides a processor. The structure of the processor may be as shown in the processor in FIG. 1, and the processor is configured to execute the second processor in the method embodiment shown in FIG. 9 to FIG. 11. Steps to perform.

The above embodiments may be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the processes or functions according to the embodiments of the present application are wholly or partially generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be from a website site, a computer, a server, or a data center. Transmission via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server, a data center, and the like, including one or more sets of available media. The usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium may be a solid state drive (SSD).

It should be noted that the first, second, third, fourth, and various numerical numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application.

The above is only a specific implementation of this application, but the scope of protection of this application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in this application. It should be covered by the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

A data processing method, comprising:

The first remote direct access memory card RNIC receives a remote direct access RDMA write request sent by the second RNIC. The RDMA write request includes first data and a data persistence flag, and the RDMA write request is used to request that the first Data is written to a first device, the data persistence flag is used to indicate that the first data is data to be persisted, the first RNIC is an RNIC of the first device, and the second RNIC is a second RNIC of the device, the first device and the second device communicate based on an RDMA mode;

The first RNIC sends a direct memory access DMA write request to a first processor, the DMA write request includes the first data, and the DMA write request is used to instruct the first processor to send the first data Write to the first device, the first processor is a processor of the first device, and the first RNIC and the first processor communicate based on a DMA method;

The first RNIC instructs the first processor to store the first data in a non-volatile memory of the first device according to the data persistence flag.
The method according to claim 1, wherein the data persistence mark is a write persistence instruction.
The method according to claim 1, wherein the data persistence mark is a destination storage address corresponding to the first data, and a storage space corresponding to the destination storage address is used to store the first data, so The destination storage address is a persistent storage address in the first device, and a storage space corresponding to the persistent storage address is used to store data to be persisted.
The method according to any one of claims 1-3, wherein the first RNIC instructs the first processor to store the first data to the first device according to the data persistence flag The non-volatile memory includes:

Adding, by the first RNIC, a DMA least significant bit read request in a receiving queue corresponding to the second RNIC according to the data persistence flag;

Sending, by the first RNIC, the DMA least significant bit read request to the first processor, where the DMA least significant bit read request is used to instruct the first processor to cache in a peripheral of the first device All data in the bus link is stored in a non-volatile memory of the first device.
The method according to any one of claims 1-3, wherein the first RNIC instructs the first processor to store the first data to the first device according to the data persistence flag The non-volatile memory includes:

After the first RNIC generates a work queue item WQE corresponding to an RDMA reception request according to the data persistence flag and the first data, the first RNIC clears the WQE, and the RDMA reception request is used to receive the second device initiated RDMA send request;

The first RNIC generates a completion queue item CQE corresponding to the RDMA reception request, and the CQE corresponding to the RDMA reception request is used to instruct the first processor to cache in a volatile storage medium of the first processor The first data is stored in a non-volatile memory of the first device.
A data processing method, comprising:

The second remote direct access memory card RNIC receives the remote direct access RDMA write persistence request sent by the second processor. The RDMA write persistence request includes a data persistence tag, and the RDMA write persistence request is used to request that the first A data is stored in a non-volatile memory of a first device, the data persistence flag is used to indicate that the first data is data to be persisted, the second RNIC is an RNIC of a second device, and the A second processor is a processor of the second device, the first device and the second device communicate based on an RDMA method, and the second RNIC and the second processor communicate based on a DMA method;

Generating, by the second RNIC, an RDMA write request according to the RDMA write persistence request, where the RDMA write request includes the first data and the data persistence flag;

Sending, by the second RNIC, the RDMA write request to the first RNIC, where the RDMA write request is used to request the first data to be written to the first device, and the first RNIC is the first device's RNIC.
The method according to claim 6, wherein the data persistence mark is a write persistence instruction.
The method according to claim 6, wherein the data persistence mark is a destination storage address corresponding to the first data, and a storage space corresponding to the destination storage address is used to store the first data, so The destination storage address is a persistent storage address in the first device, and a storage space corresponding to the persistent storage address is used to store data to be persisted.
The method according to any one of claims 6-8, wherein after the second RNIC sends the RDMA write request to the first RNIC, the method further comprises:

When receiving the reception confirmation message corresponding to the first data sent by the first RNIC, the second RNIC buffers the reception confirmation message corresponding to the first data, and the reception confirmation message corresponding to the first data includes A first data packet sequence number PSN, the first PSN is a sequence number of the first data, and a reception confirmation message corresponding to the first data is used to indicate that the first RNIC receives the first data;

Receiving, by the second RNIC, a first confirmation message sent by the first RNIC, where the first confirmation message includes a second PSN;

When the first PSN is the same as the second PSN, the second RNIC generates a completion item CQE corresponding to the RDMA write persistence request, and the CQE corresponding to the RDMA write persistence request is used to notify the first The first data of the second processor has been stored in a non-volatile memory of the first device.
The method according to any one of claims 6-8, wherein after the second RNIC sends the RDMA write request to the first RNIC, the method further comprises:

When receiving the reception confirmation message corresponding to the first data sent by the first RNIC, the second RNIC buffers the reception confirmation message corresponding to the first data, and the reception confirmation message corresponding to the first data includes A first PSN, where the first PSN is a serial number of the first data, and a reception confirmation message corresponding to the first data is used to instruct the first RNIC to receive the first data;

Generating, by the second RNIC, a CQE corresponding to the RDMA write persistence request, and the CQE corresponding to the RDMA write persistence request is used to notify the second processor that the first data has been written to the first device;

Receiving, by the second RNIC, an RDMA reception request sent by the second processor;

Receiving, by the second RNIC, a first confirmation message sent by the first RNIC, where the first confirmation message includes a second PSN;

When the second PSN is the same as the first PSN, the second RNIC generates a CQE corresponding to the RDMA reception request, and the CQE corresponding to the RDMA reception request is used to notify the second processor that the first A data has been stored in a non-volatile memory of the first device.
A remote direct access memory card RNIC is characterized in that it includes:

A receiving module, configured to receive a remote direct access RDMA write request sent by a second RNIC, the RDMA write request includes first data and a data persistence flag, and the RDMA write request is used to request that the first data is written to The first device, the data persistence flag is used to indicate that the first data is data to be persisted, the RNIC is the RNIC of the first device, and the second RNIC is the RNIC of the second device, so The first device and the second device communicate based on an RDMA manner;

A scheduling module, configured to send a direct memory access DMA write request to a first processor, where the DMA write request includes the first data, and the DMA write request is used to instruct the first processor to send the first data Write to the first device, the first processor is a processor of the first device, and the RNIC and the first processor communicate based on a DMA method;

A persistent memory module, configured to instruct the first processor to store the first data in a non-volatile memory of the first device according to the data persistence flag.
The RNIC according to claim 11, wherein the data persistence mark is a write persistence instruction.
The RNIC according to claim 11, wherein the data persistence mark is a destination storage address corresponding to the first data, and a storage space corresponding to the destination storage address is used to store the first data, so The destination storage address is a persistent storage address in the first device, and a storage space corresponding to the persistent storage address is used to store data to be persisted.
The RNIC according to any one of claims 11-13, wherein the persistent memory module is specifically configured to:

Adding a DMA least significant bit read request to a receiving queue corresponding to the second RNIC according to the data persistence flag;

Sending the DMA least significant bit read request to the first processor, where the DMA least significant bit read request is used to instruct the first processor to buffer in a peripheral bus link of the first device All data is written into a non-volatile memory of the first device.
The RNIC according to any one of claims 11-13, wherein the persistent memory module is specifically configured to:

After the first RNIC corresponding to generating the RDMA request according to the data persistence flag and the first data generates the work queue item WQE corresponding to the RDMA receiving request according to the first data, clearing the WQE, the The RDMA receiving request is used to receive an RDMA sending request initiated by the second device;

The first RNIC generates a completion queue item CQE corresponding to the RDMA reception request, and the CQE corresponding to the RDMA reception request is used to instruct the first processor to cache in a volatile storage medium of the first processor The first data is stored in a non-volatile memory of the first device.
A remote direct access memory card RNIC is characterized in that it includes:

A scheduling module, configured to receive a remote direct access memory RDMA write persistence request sent by a second processor, the RDMA write persistence request includes a data persistence tag, and the RDMA write persistence request is used to request to store the first data In the non-volatile memory to the first device, the data persistence flag is used to indicate that the first data is data to be persisted, the RNIC is an RNIC of a second device, and the second processor is A processor of the second device, the first device and the second device communicate based on an RDMA method, and the RNIC and the second processor communicate based on a DMA method;

The scheduling module is further configured to generate an RDMA write request according to the RDMA write persistence request, where the RDMA write request includes the first data and the data persistence flag;

A sending module, configured to send the RDMA write request to a first RNIC, where the RDMA write request is used to request the first data to be written to the first device, and the first RNIC is a RNIC.
The RNIC according to claim 16, wherein the data persistence flag is a write persistence instruction.
The RNIC according to claim 16, wherein the data persistence mark is a destination storage address corresponding to the first data, and a storage space corresponding to the destination storage address is used to store the first data, so The destination storage address is a persistent storage address in the first device, and a storage space corresponding to the persistent storage address is used to store data to be persisted.
The RNIC according to any one of claims 16 to 18, wherein the RNIC further comprises a receiving module, and the scheduling module is further configured to:

When the receiving module receives a reception confirmation message corresponding to the first data sent by the first RNIC, buffering the reception confirmation message corresponding to the first data, and the reception confirmation message corresponding to the first data Including a first data packet sequence number PSN, where the first PSN is a sequence number of the first data, and a reception confirmation message corresponding to the first data is used to instruct the first RNIC to receive the first data;

The receiving module is further configured to receive a first confirmation message sent by the first RNIC, where the first confirmation message includes a second PSN;

The scheduling module is further configured to: when the first PSN is the same as the second PSN, generate a completion item CQE corresponding to the RDMA write persistence request, and the CQE corresponding to the RDMA write persistence request is For notifying the second processor that the first data has been stored in the non-volatile memory of the first device.
The RNIC according to any one of claims 16 to 18, further comprising a receiving module, and the scheduling module is further configured to:

When the receiving module receives a reception confirmation message corresponding to the first data sent by the first RNIC, buffering the reception confirmation message corresponding to the first data, and the reception confirmation message corresponding to the first data Including a first PSN, where the first PSN is a serial number of the first data, and a reception confirmation message corresponding to the first data is used to instruct the first RNIC to receive the first data;

Generating a CQE corresponding to the RDMA write persistence request, and the CQE corresponding to the RDMA write persistence request is used to notify the second processor that the first data has been written to the first device;

Receiving an RDMA reception request sent by the second processor;

The receiving module is further configured to receive a first confirmation message sent by the first RNIC, where the first confirmation message includes a second PSN;

The scheduling module is further configured to: when the second PSN is the same as the first PSN, generate a CQE corresponding to the RDMA reception request, and the CQE corresponding to the RDMA reception request is used to notify the second The processor's first data has been stored in a non-volatile memory of the first device.
A data processing method, comprising:

The first remote direct access memory card RNIC receives a remote direct access RDMA write request sent by the second RNIC, the RDMA write request includes first data and a data persistence tag, and the data persistence tag is used to indicate the first The data is data to be persisted, and the RDMA write request is used to request the first data to be written to the first device, the first RNIC is the RNIC of the first device, and the second RNIC is the second RNIC of the device, the first device and the second device communicate based on an RDMA mode;

The first RNIC sends a direct memory access DMA write request to a first processor, the DMA write request includes the first data, and the DMA write request is used to instruct the first processor to send the first data Write to the first device, the first processor is a processor of the first device, and the first RNIC and the first processor communicate based on a DMA method;

The first RNIC instructs the first processor to store the first data to a non-volatile memory of the first device according to the data persistence flag.
The claim according to claim 21, wherein the first RNIC instructs the first processor to store the first data to a non-volatile memory of the first device according to the data persistence flag, comprising:

Adding a DMA read request to a receiving queue corresponding to the second RNIC;

Sending the DMA read request to the first processor, where the DMA read request is used to instruct the first processor to store all data buffered in a peripheral bus link of the first device to the first processor In the non-volatile memory of the first device.
The method according to claim 21, wherein the first RNIC instructs the first processor to store the first data to a non-volatile memory of the first device, comprising:

Generating, by the first RNIC, a work queue entry corresponding to an RDMA reception request according to the first data

After WQE, the WQE is cleared, and the RDMA reception request is used to receive an RDMA transmission request initiated by the second device;

The first RNIC generates a completion queue item CQE corresponding to the RDMA reception request, and the CQE corresponding to the RDMA reception request is used to instruct the first processor to cache in a volatile storage medium of the first processor The first data is stored in a non-volatile memory of the first device.
A remote direct access memory card RNIC is characterized in that it includes:

The first remote direct access memory card RNIC receives a remote direct access RDMA write request sent by the second RNIC, the RDMA write request includes first data, and the RDMA write request is used to request that the first data be written to the first Device, the first RNIC is the RNIC of the first device, the second RNIC is the RNIC of the second device, and the first device and the second device communicate based on the RDMA method;

The first RNIC sends a direct memory access DMA write request to a first processor, the DMA write request includes the first data, and the DMA write request is used to instruct the first processor to send the first data Write to the first device, the first processor is a processor of the first device, and the first RNIC and the first processor communicate based on a DMA method;

The first RNIC instructs the first processor to store the first data to a non-volatile memory of the first device.
The method according to claim 24, wherein the first RNIC instructs the first processor to store the first data in a non-volatile memory of the first device comprises:

Adding, by the first RNIC, a DMA least significant bit read request in a receiving queue corresponding to the second RNIC according to the data persistence flag;

Sending, by the first RNIC, the DMA least significant bit read request to the first processor, where the DMA least significant bit read request is used to instruct the first processor to cache in a peripheral of the first device All data in the bus link is stored in a non-volatile memory of the first device.
A data processing method, comprising:

The first remote direct access memory card RNIC receives a remote direct access RDMA write request sent by the second RNIC, the RDMA write request includes first data, and the RDMA write request is used to request that the first data be written to the first Device, the first RNIC is the RNIC of the first device, the second RNIC is the RNIC of the second device, and the first device and the second device communicate based on the RDMA method;

The first RNIC sends a direct memory access DMA write request to a first processor, the DMA write request includes the first data, and the DMA write request is used to instruct the first processor to send the first data Write to the first device, the first processor is a processor of the first device, and the first RNIC and the first processor communicate based on a DMA method;

The first RNIC instructs the first processor to store the first data to a non-volatile memory of the first device.
The data processing method according to claim 26, wherein the first RNIC instructs the first processor to store the first data to a non-volatile memory of the first device, comprising:

Adding a DMA read request to a receiving queue corresponding to the second RNIC;

Sending the DMA read request to the first processor, where the DMA read request is used to instruct the first processor to store all data buffered in a peripheral bus link of the first device to the first processor Non-volatile memory of the first device.
A first device, characterized in that it comprises a processor, a non-volatile memory, and a remote direct access network card, the remote direct access network card being configured to perform the operations according to any one of claims 1-5 and 21-27. The operation steps of the data processing method described above.
A second device, comprising a processor, a non-volatile memory, and a remote direct access network card, wherein the remote direct access network card is configured to perform data processing according to any one of claims 6 to 10. Steps of the method.
A computer-readable storage medium having instructions stored in the computer-readable storage medium, which when run on a computer, causes the computer to perform the operations of the data processing method according to any one of claims 1-10 and 21-27 step.