CN117873920A

CN117873920A - RDMA memory translation table application method and system

Info

Publication number: CN117873920A
Application number: CN202311746362.0A
Authority: CN
Inventors: 蒋帅
Original assignee: Yusur Technology Co ltd
Current assignee: Yusur Technology Co ltd
Priority date: 2023-12-18
Filing date: 2023-12-18
Publication date: 2024-04-12

Abstract

The invention provides a memory translation table application method and a memory translation table application system of RDMA, wherein the method is applied to a network card connected with DDR, the network card comprises an out-of-order cache module, a DDR control module and a RAM control module, and the method comprises the following steps: receiving a table lookup request, and judging the message type of the table lookup request by the out-of-order buffer module; if the message type is the first packet, acquiring a virtual address of the table lookup request, matching a corresponding physical address from a memory address translation table stored in the DDR based on the DDR control module, and storing the corresponding relation between the acquired virtual address and the physical address into a cache address translation table of a network card RAM; and if the message type is a middle packet or a tail packet, acquiring the virtual address of the table lookup request, and matching the corresponding physical address from a cache address translation table stored in a RAM memory based on the RAM control module. According to the scheme, on one hand, the comprehensiveness of the memory address translation table is guaranteed through DDR, and on the other hand, the matching processing efficiency is improved through the RAM.

Description

RDMA memory translation table application method and system

Technical Field

The invention relates to the technical field of information, in particular to an RDMA memory translation table application method and system.

Background

Remote direct memory access (Remote direct memory access, RDMA) is generated to account for server-side data processing delays in network transmissions. RDMA transfers data directly into a computer's storage area over a network, moves the data from one system quickly to a remote system memory without any impact on the operating system, thus eliminating the overhead of external memory duplication and context switching, and freeing up memory bandwidth and CPU cycles for improved application system performance.

The sending end of the RDMA operation accesses the memory address space of the receiving end through a Virtual Address (VA), but the receiving end cannot directly locate the memory address space of the receiving end through the VA, so a memory address translation table (Memory translation table, MTT) is required to translate the VA into a Physical Address (PA) of the memory of the receiving end. The receiving end can then access the memory space requested by the transmitting end through the PA. A corresponding set of VA and PA is referred to as an entry information of MTT.

At present, the implementation modes of MTT mainly comprise two types: firstly, the method is realized by the memory resource in the chip, and the method has the advantages of fast VA to PA speed and low delay. The method has the defects that the memory resources in the chip are limited, and a large amount of MTT table item information cannot be stored; the implementation method is realized by distributing the space of a double rate synchronous dynamic random access memory (DDR) arranged outside the chip, and has the advantages that the DDR has large storage space and can store a large amount of MTT table item information, but has the defects of slow access rate of the DDR and longer access delay of the DDR than that of directly accessing storage resources inside the chip.

Disclosure of Invention

In view of this, embodiments of the present invention provide a method for applying an RDMA memory translation table to obviate or mitigate one or more of the disadvantages of the related art.

One aspect of the present invention provides a method for applying an RDMA memory translation table, where the method is applied to a network card connected with a DDR, the network card includes an out-of-order cache module, a DDR control module, and a RAM control module, and the steps of the method include:

the network card receives a table look-up request, and the out-of-order buffer module judges the message type of the table look-up request;

if the message type is the first packet, acquiring a virtual address of the table lookup request, matching a corresponding physical address from a memory address translation table stored in the DDR based on the DDR control module, and storing the corresponding relation between the acquired virtual address and the physical address into a cache address translation table of a network card RAM;

and if the message type is a tundish or a tail packet, acquiring a virtual address of the table lookup request, and matching a corresponding physical address from a cache address translation table stored in a RAM memory based on the RAM control module.

By adopting the scheme, firstly, the cache address translation table is stored in the DDR, and when the message type is the first packet, the corresponding physical address is matched from the DDR and is stored in the cache address translation table of the RAM memory of the network card; if the message type is a middle packet or a tail packet, the first packet of the message is processed, so that the physical address is preferentially matched from the cache address translation table.

In some embodiments of the present invention, in the step of storing the obtained correspondence between the virtual address and the physical address in the cache address translation table of the RAM memory of the network card, the RAM control module determines whether the RAM memory has a remaining storage space, if so, the obtained correspondence between the virtual address and the physical address is stored in the cache address translation table of the RAM memory of the network card, and if not, the correspondence between the virtual address and the physical address is not added in the RAM memory.

In some embodiments of the present invention, in the step of obtaining the virtual address of the lookup request, the step of matching the corresponding physical address from the cache address translation table stored in the RAM memory based on the RAM control module, if the corresponding physical address is not matched from the cache address translation table stored in the RAM memory, the corresponding physical address is matched from the memory address translation table stored in the DDR based on the DDR control module.

In some embodiments of the present invention, the message types of the lookup request include a single lookup request and a non-single lookup request, where the message types of the non-single lookup request include a first packet, a middle packet, and a tail packet, and the steps of the method further include, if the message types are the single lookup request, obtaining a virtual address of the single lookup request, and matching a corresponding physical address from a memory address translation table stored in the DDR based on the DDR control module.

As shown in fig. 3, in some embodiments of the present invention, the network card further includes an address calculation module, and the steps of the method include: the address calculation module obtains a corresponding memory storage unit based on the obtained physical address.

In some embodiments of the invention, the steps of the method comprise: and the RAM control module performs memory management on the RAM memory at intervals of preset management time, and the memory management step comprises deleting the corresponding relation between the virtual address and the physical address stored in the RAM memory.

In some embodiments of the present invention, the step of deleting the correspondence between the virtual address and the physical address stored in the RAM memory includes obtaining a last matched time of each correspondence between the virtual address and the physical address stored in the RAM memory, calculating a first time interval between a current time and the last matched time, and deleting the correspondence between the corresponding virtual address and the physical address from the RAM memory if the first time interval is greater than a preset first threshold time.

In some embodiments of the present invention, the step of deleting the correspondence between the virtual address and the physical address stored in the RAM memory further includes obtaining the correspondence between the virtual address and the physical address that are last matched as a tail packet in the RAM memory, obtaining a last matched time of the tail packet, calculating a second time interval between a current time and the last matched time of the tail packet, and deleting the correspondence between the corresponding virtual address and the physical address from the RAM memory if the second time interval is greater than a preset second threshold time.

In some embodiments of the present invention, in the step of deleting the correspondence between the virtual address and the physical address stored in the RAM memory, the second threshold time is smaller than the first threshold time.

The second aspect of the present invention also provides an RDMA memory translation table application system, the system comprising a computer device comprising a processor and a memory, the memory having stored therein computer instructions for executing the computer instructions stored in the memory, the system implementing the steps implemented by the method as described above when the computer instructions are executed by the processor.

The third aspect of the present invention also provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps implemented by the RDMA memory translation table application method described above.

Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention. The objectives and other advantages of the invention may be realized and attained by the structure particularly pointed out in the written description and drawings.

It will be appreciated by those skilled in the art that the objects and advantages that can be achieved with the present invention are not limited to the above-described specific ones, and that the above and other objects that can be achieved with the present invention will be more clearly understood from the following detailed description.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate and together with the description serve to explain the invention.

FIG. 1 is a schematic diagram of one embodiment of a method for applying an RDMA memory translation table according to the present invention;

FIG. 2 is a schematic diagram of an implementation architecture of an RDMA memory translation table application method of the present invention;

FIG. 3 is a schematic diagram of another implementation architecture of the RDMA memory translation table application method of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the following embodiments and the accompanying drawings, in order to make the objects, technical solutions and advantages of the present invention more apparent. The exemplary embodiments of the present invention and the descriptions thereof are used herein to explain the present invention, but are not intended to limit the invention.

It should be noted here that, in order to avoid obscuring the present invention due to unnecessary details, only structures and/or processing steps closely related to the solution according to the present invention are shown in the drawings, while other details not greatly related to the present invention are omitted.

The method specifically comprises the following steps:

as shown in fig. 1 and 2, the present invention proposes an RDMA memory translation table application method, where the method is applied to a network card connected with a DDR, the network card includes an out-of-order cache module, a DDR control module, and a RAM control module, and the steps of the method include:

in a specific implementation process, the network card is a device with a memory capability and a processing capability, and specifically, an intelligent Chip is arranged in the network card, and the intelligent Chip can be a field programmable gate array Chip (Field Programmable Gate Array, FPGA), a System on Chip (SoC) and the like.

In the specific implementation process, in the RDMA technology, the control instruction issued by the CPU to the network card is generally called WR (Work Request). The issued instruction is stored in WQ (Work Queue), and the stored instruction is called WQE (Work Queue Element), and the corresponding hardware automatically goes to the WQ to accept the instruction. Depending on the instruction type deposited, the WQ is divided into two classes SQ (Send Queue) and RQ (Receive Queue), which together make up the min unit QP (Queue Pairs) for communication in RDMA.

Step S100, a network card receives a table look-up request, and the disordered cache module judges the message type of the table look-up request;

in the implementation process, the table look-up request is a table look-up request of a data packet, and the network card receives the data packet and acquires the table look-up request in the data packet.

Step S200, if the message type is the first packet, obtaining the virtual address of the table look-up request, matching the corresponding physical address from a memory address translation table stored in the DDR based on the DDR control module, and storing the corresponding relation between the obtained virtual address and the physical address into a cache address translation table of a network card RAM;

in the implementation process, the sending end of the RDMA operation accesses the memory address space of the receiving end through a Virtual Address (VA), but the receiving end cannot directly locate the memory address space of the receiving end through the VA, so a memory address translation table (Memory translation table, MTT) is required to convert the VA into a Physical Address (PA) of the memory of the receiving end.

Step S300, if the message type is a tundish or a tail packet, obtaining the virtual address of the table lookup request, and matching the corresponding physical address from a cache address translation table stored in a RAM memory based on the RAM control module.

In the implementation process, if the type is a middle packet or a tail packet, judging whether a cached table look-up request of the same QP exists, namely, the first packet of the same QP is processed, if yes, skipping access of the DDR, and directly matching the result through internal storage resources; if there is no matching result, the result is found by DDR.

In particular implementations, in RDMA communications, the first packet is typically used to establish a connection and initiate the communication, and contains the necessary information needed to establish the connection, such as an identifier of a target Queue Pair (Queue Pair), a request reply (Acknowledge Request) flag, and a packet sequence number (Packet Sequence Number). The sender of the first packet needs to copy the data from the user space to the Socket Buffer in the kernel space, add the data packet header for encapsulation, and then send the data packet header to the network after processing a series of network protocols. The receiving party needs to copy the received data packet from the network card Buffer to the Socket Buffer, analyze the data after processing the network protocol, and copy the data to the application program Buffer in the user space.

In RDMA communications, a middle packet is typically used to carry the actual data payload, containing the data that the application is to send or receive. The handling of the middle packets is relatively simple compared to the first packets, as they do not require connection establishment or initialization. The sender copies the data from the user space to the Socket Buffer in the kernel space, and directly sends the data to the target memory address through the RDMA protocol. The receiver can also directly transmit the data from the network card Buffer to the application Buffer in the user space through RDMA protocol without passing through Socket Buffer in the kernel space.

In RDMA communications, the tail packet is typically used to indicate the end of a data transfer or to do some cleanup work. For example, when an RDMA connection is closed, a tail packet needs to be sent to indicate termination of the connection. The tail packet may contain some status information or an indication of the cleaning job. Compared with the first packet and the middle packet, the processing process of the tail packet is simpler.

By adopting the scheme, the RAM control module is used for judging whether the storage space remains in the RAM memory, if the storage space does not remain, the obtained corresponding relation between the virtual address and the physical address is stored in the buffer address translation table of the RAM memory of the network card, and the processing efficiency of the packet in the future and the packet in the tail is ensured for the first packet to be stored in the high-speed RAM memory.

In the implementation process, if the cache address translation table stored in the RAM memory is not matched with the corresponding physical address, the corresponding physical address is matched with the memory address translation table stored in the DDR, and the DDR stores the comprehensive memory address translation table, so that the matching success rate is ensured.

In an implementation, the single lookup request in RDMA (Remote Direct MemoryAccess) communication is a specific operation request for looking up a specific data item in the remote memory.

By adopting the scheme, if the single lookup request is adopted, the corresponding relation between the same virtual address and the physical address is only utilized once, so that the corresponding relation between the virtual address and the physical address in the single lookup request is not required to be stored in the RAM memory, and the utilization effect of the RAM memory is improved.

In some embodiments of the present invention, the network card further includes an address calculation module, and the steps of the method include: the address calculation module obtains a corresponding memory storage unit based on the obtained physical address.

In the implementation process, the main function of the DDR control module is to read and control the DDR, process the read DDR data to complete the bit width conversion, and store the converted data into the FIFO to wait for the address calculation module to take.

In the specific implementation process, the RAM control module manages the RAM memory at intervals of preset management time, manages the content stored in the RAM memory at regular time, deletes the outdated storage item, and ensures the efficient utilization of the RAM memory.

By adopting the scheme, if the corresponding relation between the virtual address and the physical address of each item stored in the RAM memory is not used for a long time, the corresponding data transmission is stopped or interrupted, and the corresponding relation between the virtual address and the physical address is deleted from the RAM memory, so that the efficient utilization of the RAM memory is ensured.

By adopting the scheme, for the correspondence between the virtual address and the physical address which are stored in the RAM memory and are matched into the tail packet for the last time, if the corresponding data transmission is not used for a long time, the corresponding correspondence between the virtual address and the physical address is deleted from the RAM memory, and the efficient utilization of the RAM memory is ensured.

By adopting the scheme, the corresponding relation between the virtual address and the physical address which are matched into the tail packet for the last time shows that the data transmission corresponding to the matching has high probability of ending transmission, so that the second threshold time is smaller than the first threshold time, and the efficient utilization of the RAM memory is further ensured.

The invention provides a method for combining memory resources inside a chip with external DDR to realize MTT. And storing a large amount of MTT (multiple transfer terminal) table entry information in the DDR, and storing the table entry information of the large data packet operation in a memory resource inside the chip. When the operation of the big data packet is finished, the big data packet is released for other table entry information to use. Therefore, for the entry information of the large data packet request, the subsequent VA-to-PA conversion with low delay can be realized at the cost of one-time read-write delay from DDR to memory Resources (RAM) in the chip.

The beneficial effect of this scheme includes:

1. according to the scheme, through DDR, the space for storing the MTT table entry can be lifted from MB to GB, and the lifting is about 1000 times;

2. according to the scheme, the table entries in the DDR are written into the internal memory resource of the chip, so that access to the DDR can be reduced, and the delay is reduced.

The embodiment of the invention also provides an RDMA memory translation table application system, which comprises a computer device, wherein the computer device comprises a processor and a memory, the memory is stored with computer instructions, the processor is used for executing the computer instructions stored in the memory, and the system realizes the steps realized by the method when the computer instructions are executed by the processor.

The embodiments of the present invention also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps implemented by the RDMA memory translation table application method described above. The computer readable storage medium may be a tangible storage medium such as Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disk, a removable memory disk, a CD-ROM, or any other form of storage medium known in the art.

Those of ordinary skill in the art will appreciate that the various illustrative components, systems, and methods described in connection with the embodiments disclosed herein can be implemented as hardware, software, or a combination of both. The particular implementation is hardware or software dependent on the specific application of the solution and the design constraints. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, a plug-in, a function card, or the like. When implemented in software, the elements of the invention are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine readable medium or transmitted over transmission media or communication links by a data signal carried in a carrier wave.

It should be understood that the invention is not limited to the particular arrangements and instrumentality described above and shown in the drawings. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present invention are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after appreciating the spirit of the present invention.

In this disclosure, features that are described and/or illustrated with respect to one embodiment may be used in the same way or in a similar way in one or more other embodiments and/or in combination with or instead of the features of the other embodiments.

The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, and various modifications and variations can be made to the embodiments of the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. The memory translation table application method of the RDMA is characterized by being applied to a network card connected with a DDR, wherein the network card comprises an out-of-order cache module, the DDR control module and a RAM control module, and the method comprises the following steps:

2. The RDMA memory translation table application method according to claim 1, wherein in the step of storing the obtained correspondence between the virtual address and the physical address in the cache address translation table of the network card RAM memory, whether the RAM memory has a remaining memory space is determined by the RAM control module, if yes, the obtained correspondence between the virtual address and the physical address is stored in the cache address translation table of the network card RAM memory, and if no, the correspondence between the virtual address and the physical address is not added in the RAM memory.

3. The RDMA memory translation table application method according to claim 1, wherein in the step of obtaining the virtual address of the lookup request, the corresponding physical address is matched from the cache address translation table stored in the RAM memory based on the RAM control module, if the corresponding physical address is not matched from the cache address translation table stored in the RAM memory, the corresponding physical address is matched from the memory address translation table stored in the DDR based on the DDR control module.

4. The method of claim 1, wherein the message types of the lookup request include a single lookup request and a non-single lookup request, wherein the message types of the non-single lookup request include a first packet, a middle packet, and a tail packet, and the method further comprises the steps of obtaining a virtual address of the single lookup request if the message types are the single lookup request, and matching a corresponding physical address from a memory address translation table stored in the DDR based on the DDR control module.

5. The method for applying a memory translation table of RDMA according to claim 1, wherein said network card further comprises an address calculation module, said method comprising the steps of: the address calculation module obtains a corresponding memory storage unit based on the obtained physical address.

6. The method for applying an RDMA memory translation table according to any one of claims 1 to 5, wherein the steps of the method include: and the RAM control module performs memory management on the RAM memory at intervals of preset management time, and the memory management step comprises deleting the corresponding relation between the virtual address and the physical address stored in the RAM memory.

7. The method according to claim 6, wherein the step of deleting the correspondence between the virtual address and the physical address stored in the RAM memory includes obtaining a last time of matching the correspondence between each of the virtual address and the physical address stored in the RAM memory, calculating a first time interval between a current time and the last time of matching, and deleting the correspondence between the corresponding virtual address and the physical address from the RAM memory if the first time interval is greater than a preset first threshold time.

8. The method according to claim 7, wherein the step of deleting the correspondence between the virtual address and the physical address stored in the RAM memory further comprises obtaining the correspondence between the virtual address and the physical address that were last matched as a tail packet in the RAM memory, obtaining a last matched time of the tail packet, calculating a second time interval between a current time and the last matched time of the tail packet, and deleting the correspondence between the corresponding virtual address and the physical address from the RAM memory if the second time interval is greater than a preset second threshold time.

9. The RDMA memory translation table application method according to claim 8, wherein in the step of deleting the correspondence between the virtual address and the physical address stored in the RAM memory, said second threshold time is smaller than said first threshold time.

10. An RDMA memory translation table application system, comprising a computer device comprising a processor and a memory, the memory having stored therein computer instructions for executing the computer instructions stored in the memory, the system implementing the steps implemented by the method of any of claims 1-9 when the computer instructions are executed by the processor.