CN117675890A

CN117675890A - RDMA dynamic connection transmission service providing method and device

Info

Publication number: CN117675890A
Application number: CN202311678975.5A
Authority: CN
Inventors: 宁佐林; 陈正林
Original assignee: Haiguang Information Technology Co Ltd
Current assignee: Haiguang Information Technology Co Ltd
Priority date: 2023-12-08
Filing date: 2023-12-08
Publication date: 2024-03-08

Abstract

The invention provides a RDMA dynamic connection transmission service providing method and a device, wherein the method comprises the following steps: receiving a dynamic connection request message of RDMA, extracting a source port network address in a packet header and a transmission queue identifier of a dynamic connection requester, performing hash operation, inquiring a corresponding hash table item according to a hash value, and matching dynamic connection context resources; if the matching is successful, generating a context resource identifier corresponding to the dynamic connection context resource according to the hash value and the corresponding hash table item identifier, writing the context information into a corresponding context on-chip cache, and transmitting a dynamic connection request message to an RDMA receiving engine; if the matching fails, the abnormal message is silently discarded, and the data request message is transmitted after the reconstruction of the link is successful. The invention improves the resource utilization rate and the effective bandwidth and enhances the robustness of the system.

Description

RDMA dynamic connection transmission service providing method and device

Technical Field

The invention belongs to the field of network transmission, and particularly relates to a method and a device for providing RDMA dynamic connection transmission service.

Background

Similar to DMA technology within computer systems, RDMA (Remote Direct Memory Access ) technology can support zero copy network technology and kernel bypass technology in high performance computing areas and large data centers to achieve high performance remote data direct access. The InfiniBand protocol (simply IB protocol) is a network protocol that supports RDMA technology. The IB protocol re-interprets a complete set of network layer models, including a transport layer, a network layer, a link layer, and the like, so that devices such as a network card, a switch, and the like, which are common in a network, all support RDMA technology. In RDMA technology, there are three types of reliable connection services commonly used, namely reliable connection RC, extended reliable connection XRC and dynamic connection DC. The DC service is more flexible in connection mode and occupies less resources due to the characteristic that the connection of the DC service can be dynamically established and released. DC is a process of establishing a connection, i.e. establishing a communication pipe, exchanging context information, depending on the method of connection management defined in the IB protocol. The link establishment process does not depend on a connection management method, but informs a destination terminal to establish connection through a hardware packet, and similarly, if the connection is to be released, a link breaking request message is sent. This special packet is called a DC link packet.

The specific flow of the DC link establishment and breaking is shown in fig. 1, and specifically includes:

(1) the software creates DCI of a DC service initiating requester and DCT of a DC service destination receiver according to the configuration, each DCI corresponds to a sending queue of an entity, and the sending queue is the same as a request end of an RC service type and internally contains some basic information. The DCTs are the destination objects of the receive processing DC operations, each corresponding to a shared receive queue.

(2) The software constructs enough receiving work requests WR in a receiving end shared receiving queue, wherein the request WR consists of three important pieces of information, and a ctrl section and a data section contain basic information, address length of data to be moved and the like: the target section is the basis for the sender to analyze the destination end, wherein the data section designates the buffer areas with the same size, so that the requests initiated by a plurality of request ends and a plurality of DCIs can be processed.

(3) When the upper layer application program needs to carry out data movement, a request WR is sent to a sending queue.

(4) And the sender controller compares the current DCI with the information in the current DCI context, if the information is consistent, the WR is continuously processed, and if the information is inconsistent, the connection of the current DCI is disconnected, and the current DCI is re-linked with the target pointed in the WR.

(5) And the receiver controller judges whether a data request or a link establishment request is received, and when the link establishment request is received, the idle DCR, namely the DCS service receiver context, is distributed in a resource pool associated with the DCT and is used for establishing connection with the DCI and storing information. After the allocation is completed, a response packet is returned. If a data request is received, the receiver controller finds the corresponding DCR according to the information in the packet and processes the data.

DC transmission provides a Datagram-like Datagram model that allows a process to connect multiple processes in multiple remote nodes, binding contexts to each other through network data request messages, creating a temporary equivalent RC-transmitted service. This dynamic connection management manner makes it possible for any one of the created DCIs to repeatedly build and break links during its lifetime, and also releases the DCR that is no longer used during connection establishment and switches the enabled new DCR into the active state.

A typical RDMA DC transfer service technology is proposed by melannox, and a ladder diagram of data transfer implemented by the scheme is shown in fig. 2. The DC link establishment request message and the data request message are sent back to back, and the receiving party inquires and distributes DCR through the identification of the source DCI carried in the DC link establishment request message and the source port network address LID. Returning an ACK response after successful allocation; if no resources are available to handle the connection request, a CNAK connection reject answer is returned. If the DC link-breaking request message is received, the response party does not need to return any response, and the sender defaults that the link-breaking is successful.

With the improvement of the chip manufacturing process level, when the lossless network transmission requirement with high processing complexity is met, the traditional DC implementation scheme has some defects, and the defects are specifically shown as follows:

1) Connection context processing is highly complex

At the sending request end, each process typically maintains a set of separate or multiple DC contexts. The hardware network card transmits the context data structures from the host memory to the context cache of the hardware network card as needed. In the scenario that a plurality of nodes use a DC transmission service type to initiate a link establishment request, a receiver of a target node is determined together according to a DCI identifier of a source requester and a requester LID. And DCI marks are 24bit data, LID is 16bit data, mapping is carried out through the two data, the query efficiency is low, and the required storage space utilization rate is low.

2) DC transmission has high independence and difficult management

Since the DC transmission service allows a datagram-like connection model on the connection entity, DCI and DCR are not pair-wise bound as independent connection contexts when implementing the DC transmission service. When DCI is bound to a DCR at a given remote node, it does not mean that there must be a complementary connection in opposite directions at the same time. Therefore, the DC transmission service and the service such as RC have substantial differences in connection management mechanism, so that on the system framework, a sender controller and a receiver controller need to be separately designed for the DC transmission, which affects the complexity of system design and processing stability.

3) Connection packet exception handling latency is large

Compared with RC transmission, the DC transmission has the advantages that connection semantics such as link establishment, link breakage and the like are increased, and accordingly, when abnormal situations such as packet loss, sequence disorder, state errors, insufficient resources and the like are processed, the situation of the connection packets needs to be considered. When the current DC transmission service processes the link establishment request message, if the response party is due to insufficient resources (for example, all DCRs in the resource pool are in use), the link establishment request is refused and the CNAK is replied. For a rejected connection request, no information will be retained on the responder and the subsequent data request message will not match any active DCR and other requests will be silently discarded. After receiving the CNAK, the DCI waits for a configurable period of time and then continues to retry. Because the link establishment request message and the data request message are sent back to back, before the ACK response of the link establishment request message is received, the sender defaults to be able to successfully establish the link, and at this time, the CNAK situation occurs, and the data request message is silently discarded by the receiver. After retransmission, if no resource is available, the data request message is discarded again, and the effective bandwidth is affected.

In addition, the broken link request message needs to wait for the ACK response of all the data request messages to return before being sent. If packet loss occurs in the link-breaking request message, the DCR is not successfully released, and when the same DCI needs to initiate a request to the DCT again, the DCR is allocated but the information is not up to date, which eventually results in that the receiving party receives a valid request and recognizes the valid request as an invalid request, and the receiving party enters an error state, and needs host software to reset the sending and receiving queue, so that normal functions are affected.

Disclosure of Invention

The invention aims to provide a method and a device for providing RDMA dynamic connection transmission service, which aim to realize RDMA DC transmission service and overcome the limitation of DCR context management and the defect of connection packet exception handling.

According to a first aspect of the present invention, there is provided an RDMA dynamic connection transfer service providing method, including:

receiving a dynamic connection request message, extracting a source port network address in a packet header and a transmission queue identifier of a dynamic connection requester, performing hash operation, inquiring a corresponding hash table item in a hash table according to a hash value, and matching dynamic connection context resources;

if the dynamic connection context resource is successfully matched, generating a context resource identifier corresponding to the dynamic connection context resource according to the hash value and the corresponding hash table entry identifier, writing the context information into a context on-chip cache, and transmitting a dynamic connection request message to an RDMA receiving engine;

if the dynamic connection context resource matching fails, determining that the dynamic connection request message is abnormal, discarding the abnormal message in a default mode, and transmitting the data request message after the reconstruction of the link is successful.

Preferably, before the receiving the dynamic connection request message, the method further includes:

judging whether the received message is a dynamic connection request message or not, and if not, directly transmitting the message to the RDMA receiving engine for processing.

Preferably, the querying the hash table for the corresponding hash table entry according to the hash value further includes:

performing hash operation on the source port network address and the transmission queue identifier of the dynamic connection requester, taking the calculated hash value as an address, and inquiring a hash table item corresponding to the address in a hash table;

if the type of the dynamic connection request message is a link establishment request message, when an available hash table entry exists, filling the source port network address and the transmission queue identification of the dynamic connection request party into the hash table entry, determining that the dynamic connection context resource is successfully matched, and otherwise rejecting the link establishment request.

Preferably, the context resource identifier is composed of a dynamic connection transport service fixed offset value, the hash value and a hash table entry identifier.

Preferably, the transmitting the data request message after the successful reestablishment of the link further includes:

when the message is abnormal, a connection refusal response message is returned to the dynamic connection transmission service request party to inform the dynamic connection transmission service request party to retransmit the chain request message,

and after successful link establishment, returning a confirmation response to the dynamic connection transmission service requester, and initiating a data request message by the dynamic connection transmission service requester.

According to a second aspect of the present invention, there is provided an RDMA dynamic connection transfer service providing apparatus, comprising:

the resource matching unit is used for receiving the dynamic connection request message, extracting a source port network address in the packet header and a transmission queue identifier of the dynamic connection requester, carrying out hash operation, inquiring a corresponding hash table item in the hash table according to the hash value, and matching the dynamic connection context resource;

the cache writing unit is used for generating a context resource identifier corresponding to the dynamic connection context resource according to the hash value and the corresponding hash table entry identifier when the dynamic connection context resource is successfully matched, writing the context information into a context on-chip cache, and transmitting a dynamic connection request message to an RDMA receiving engine;

and the abnormal message processing unit is used for determining that the dynamic connection request message is abnormal when the dynamic connection context resource matching fails, discarding the abnormal message in a default way, and transmitting the data request message after the link is successfully reestablished.

The RDMA dynamic connection transmission service providing method and device can solve the defects of the Mellanox in realizing the RDMA dynamic connection transmission service, the mixed scene of DC and RC transmission service and the problem of insufficient resources of a receiver or the abnormal situation of packet loss, and has the following advantages compared with the prior art:

1) The invention carries out HASH operation on LID and DCIN, thereby accelerating the query process of available DCR resources. The mechanism of returning the ACK packet to the DCRn further improves the query efficiency, reduces repeated HASH calculation and can be more directly positioned to the associated DCR context. In addition, the public DCR resource pool is used for management, so that the situation of uneven resource distribution caused by different resource occupation conditions among different DCTs is avoided, the occurrence frequency of CNAK is reduced, the resource utilization rate is improved, and the limited DCR resource can be effectively utilized.

2) The method writes the DCR connection Context generated after processing the DC link establishment request message and the Context of other transmission service types into the Context Cache on-chip Cache for management, uses the generated DCRn as an access identifier of the corresponding Context, and prevents the repetition of the generated identifier and other transmission service types due to the fixed offset value of the DC transmission service in the DCRn identifier. The scheme simplifies the design, so that other modules do not need to perceive specific service types, are directly processed according to normal processing flow, and are convenient to manage and maintain on the basis of realizing functional requirements.

3) The invention optimizes the conditions of CNAK retransmission and DC link-breaking request packet loss. When the retransmission is initiated after waiting for the CNAK, the ACK response packet of the DC link establishment request message is received and then the DC data request message is sent instead of back-to-back transmission of the DC data request message and the DC link establishment request message, so that the situation that the DC data request message is silently discarded when the CNAK is replied again without available resources can be effectively avoided, and the effective bandwidth is improved. In addition, the invention also realizes the configuration of replying ACK by the DC link-breaking request message, when the WR issued to DCI has the application requirement of repeatedly establishing and breaking a link for a certain DCT, the configuration option can be started, thereby avoiding the occurrence of abnormal errors caused by that the DC link-breaking request message does not successfully arrive at a destination end for a certain reason, and enhancing the robustness of the system.

Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention may be realized and attained by the structure and process particularly pointed out in the written description and claims hereof as well as the appended drawings.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow diagram of RDMA DC connection management according to the prior art.

Fig. 2 is a data transmission ladder diagram according to a DC scheme of the prior art.

FIG. 3 is a flow chart of an RDMA dynamic connection transfer service provision method according to the present invention.

Fig. 4 is a diagram of an implementation architecture of RDMA DC transport service types according to the present invention.

Fig. 5 is a ladder diagram of a CNAK retransmission scheme and a broken link packet loss scheme according to the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which are derived by a person skilled in the art from the embodiments according to the invention without creative efforts, fall within the protection scope of the invention.

Aiming at the limitations of DCR context management and the defect of connection packet exception handling in the implementation scheme of Mellanox, the invention provides a method and a device for providing RDMA dynamic connection transmission service. And adopting HASH group management to the DCR resource pool in one node, wherein the idle DCR in the DCR resource pool is not bound to a specific DCT, but is used as a public resource. After the DC connection packet is identified, a corresponding HASH group is calculated according to the DCI mark carried in the packet header information and the source port network address LID, and idle DCR resources are found out from the HASH group to be occupied. When DCR is allocated, the DCT context information of the destination end is read, the created DCR context is combined with the packet header information and written into the context on-chip cache, so that the context is consistent with RC in terms of context management. If the DCT management module generates error codes and transmits the error codes to the downstream module due to insufficient resources, the CNAK packet is returned to the request end. In addition, the abnormal rapid recovery is realized by expanding the processing scheme of the DC connection packet in the abnormal scene. By the scheme, the resource utilization rate is improved, and the processing capacity of the system is enhanced.

Referring to the flowchart of fig. 3, the RDMA dynamic connection transmission service providing method provided by the present invention includes:

step 101: and receiving a dynamic connection request message of RDMA, extracting a source port network address in a packet header and a transmission queue identifier of a dynamic connection requester, performing hash operation, inquiring a corresponding hash table item in a hash table according to the hash value, and matching dynamic connection context resources.

A schematic diagram of an implementation architecture of the present invention is shown in fig. 4. The overall architecture consists of a DCT management module, DCT_MAN, and standard RDMA components, receiving engine RX_ENG and Context Cache on chip.

The DCT_MAN module receives the request message and analyzes the request message according to the packet head operation type field. If not, the DC request message is directly transmitted to the RX_ENG receiving engine for normal processing, and if so, the DCT_MAN is used for processing.

The DCT_MAN module is provided with a HASH Table for HASH group management of the DCR resource pool in the node. When receiving the DC extraction request message, carrying out HASH operation on the source port network address LID in the packet header and the DC request party sending queue identification DCIN, taking the calculated HASH value as an address, and inquiring available DCR resources (HASH Table sub-items) in the HASH Table. Each HASH value corresponds to a number (m shown) of sub-entries for preventing HASH collisions.

Step 102: and if the dynamic connection context resource is successfully matched, generating a context resource identifier corresponding to the dynamic connection context resource according to the hash value and the corresponding hash table entry identifier, and writing the context information into a context on-chip cache.

The DCT_MAN module performs different operations according to whether the type of the DC request message is a link establishment request message, a link breakage request message or a data request message.

If the type of the DC request message is a link establishment request message, an available subitem (field vld is 0) is found from the HASH Table, the LID and DCIN are filled in the subitem, and the field vld is 1. If no available child item is found, the request for chain establishment is denied. If the link establishment is successful, generating a DCR identifier DCRn according to the HASH value and the identifier of the subitem:

dcrn= { DC transport service fixed offset value, HASH value, subitem identification }.

Wherein the DC transport service fixed offset value is used to avoid duplication of the generated identification DCRn and other transport service types.

And feeding back the generated DCRn in the ACK message, reading the DCT Context according to the DCT mark in the packet header, and writing the DCT Context into a storage space corresponding to the DCRn in the Context Cache on-chip Cache of the Context Cache by combining with the effective field information in the packet header, so that the DCRn is consistent with RC in terms of Context management.

In an alternative embodiment, if the type of the DC request message is a broken link request message, a subitem that matches LID and DCIN and has a vld of 1 is found from the HASH Table, and its vld is set to 0, indicating that the subitem is in an available state.

Step 103: if the dynamic connection context resource matching fails, determining that the dynamic connection request message is abnormal, discarding the abnormal message in a default mode, and transmitting the data request message after the reconstruction of the link is successful.

If the type of the DC request message is a data request message, firstly judging whether the data request message carries a DCR identifier DCRn. If the DCRn fed back by the ACK is not carried, whether the corresponding sub-item is valid and matched is inquired according to the HASH values of the LID and the DCIN. If found, generating DCRn in the same way as the link establishment message, otherwise, judging as a DC abnormal request message. On the other hand, if the DCRn fed back by the ACK is carried, the corresponding subitem is indexed according to the DCRn, and if the DCRn is invalid or the information is not matched, the DCRn is judged to be the DC abnormal request message.

If the message is a DC normal request message or a message refused to build a chain, the message is transmitted to an RX_ENG receiving engine for subsequent processing. If the DC exception request message is, the DC exception request message is directly discarded. The invention expands the processing scheme of the DC connection packet under the abnormal scene, eliminates the influence of effective bandwidth reduction caused by silence discard data request message after CNAK retransmission, and eliminates the hidden trouble of sending and receiving into error state caused by packet loss of the broken link request message.

As shown in fig. 5 (a), when a DC link establishment request message and a data request message REQ are normally sent, the DC link establishment request message may be transmitted back to increase the transmission rate, and once the DC link establishment request message fails to find available DCR resources, the request returns a CNAK to the requester, and the requester does not immediately send the data request message REQ, but waits for an ACK packet returned by the successful link establishment, before initiating the data request message REQ, so as to avoid the problem of reduced effective bandwidth caused by the silent discarding of the data request message.

As shown in fig. 5 (b), when the connection is frequently switched and the application scenario where the request DCT has been previously transmitted is switched back, the waiting broken link ACK state of the dashed frame in the figure is increased, and only after the ACK response of the broken link request message returns, the subsequent packet can be transmitted, which is equivalent to adding an isolation between the broken link request message and the link establishment request message of the next destination. Because the broken link request message can be sent only after the ACK of all the data request messages is returned, the ACK of the broken link request message is waited again to return and then the link establishment request message of the next destination terminal, the scheme does not cause larger sending delay, and can effectively avoid abnormal errors caused by that the DC broken link request message does not successfully arrive at the destination terminal for a certain reason, thereby improving the stability of system design.

Therefore, the RDMA dynamic connection transmission service providing method provided by the invention can solve the defects of the Mellanox in realizing the RDMA dynamic connection transmission service, coping with the mixed scene of DC and RC transmission service and the occurrence of the problem of insufficient resources of a receiver or the abnormal situation of packet loss, and has the following advantages compared with the prior art:

Accordingly, the present invention provides in a second aspect an RDMA dynamic connection transfer service providing apparatus comprising:

The above apparatus may be implemented by the RDMA dynamic connection transport service providing method provided by the embodiment of the first aspect, and specific implementation manner may be referred to the description in the embodiment of the first aspect, which is not repeated herein.

It is understood that the hardware structures, parameters, and algorithm types described in the above embodiments are by way of example only. Those skilled in the art may also make and adjust the structural features of the above embodiments as desired without limiting the inventive concept to the specific details of the examples described above.

While the invention has been described in detail with reference to the foregoing embodiments, it will be appreciated by those skilled in the art that variations may be made in the techniques described in the foregoing embodiments, or equivalents may be substituted for elements thereof; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An RDMA dynamic connection transfer service providing method, comprising:

2. The RDMA dynamic connection transfer service providing method according to claim 1, further comprising, prior to said receiving the dynamic connection request message:

3. The RDMA dynamic connection transfer service providing method according to claim 1, wherein said querying the hash table for the corresponding hash table entry according to the hash value further comprises:

4. The RDMA dynamic connection transport service provision method according to claim 1, wherein said context resource identifier consists of a dynamic connection transport service fixed offset value, said hash value and a hash table entry identifier.

5. The RDMA dynamic connection transfer service providing method according to claim 1, wherein said transferring the data request message after the re-establishment of the link is successful, further comprises:

6. An RDMA dynamic connection transfer service providing apparatus, comprising:

7. The RDMA dynamic connection transfer service providing apparatus according to claim 6, wherein said resource matching unit is further configured to:

before the dynamic connection request message is received, judging whether the received message is the dynamic connection request message, and if not, directly transmitting the message to the RDMA receiving engine for processing.

8. The RDMA dynamic connection transfer service providing apparatus according to claim 6, wherein said resource matching unit is further configured to:

9. The RDMA dynamic connection transfer service providing apparatus as recited in claim 6, wherein said context resource identifier consists of a dynamic connection transfer service fixed offset value, said hash value, and a hash table entry identifier.

10. The RDMA dynamic connection transfer service providing apparatus according to claim 6, wherein the exception message processing unit is further configured to: