CN115442318A

CN115442318A - Method, device, storage medium and equipment for realizing reliable multicast of RDMA (remote direct memory Access) network

Info

Publication number: CN115442318A
Application number: CN202110627233.4A
Authority: CN
Inventors: 张君逸; 王巧灵; 张鹏
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2021-06-04
Filing date: 2021-06-04
Publication date: 2022-12-06

Abstract

The application relates to the technical field of communication, and discloses a method, a device, a storage medium and equipment for realizing reliable multicast of an RDMA (remote direct memory Access) network, wherein the method comprises the following steps: the method comprises the steps that a switch firstly receives an Envelope control message and a multicast data message sent by a source end, wherein the Envelope control message load comprises an Envelope message identification label Tag, a network address of a destination end associated with a source end queue pair QP, a QPN of the destination end QP associated with the source end QP, memory area MR information of the destination end and a QPN of the source end, then the Envelope control message is identified according to the Envelope message identification label Tag, the content of the load is extracted and recorded, then the recorded content is used for replacing a corresponding domain in a subsequent multicast data message and/or a corresponding feedback message ACK/NACK, and the multicast data message is forwarded according to a preset rule. Therefore, the replication and forwarding of the multicast message are realized by using the switch under the condition of being compatible with the standard RDMA protocol and not needing to respectively establish RC (remote control) connection between the switch and the source end and the destination end, and further the reliable multicast in the RDMA network is realized.

Description

Method, device, storage medium and equipment for realizing reliable multicast of RDMA (remote direct memory Access) network

Technical Field

The present application relates to the field of communications technologies, and in particular, to a method, an apparatus, a storage medium, and a device for implementing reliable multicast in an RDMA network.

Background

With the rapid development of network technology, the world has entered into the rapidly developing network era, and Multicast (Multicast) is one of the basic transmission modes of the network, and it is important to ensure that each destination can reliably receive each data sent by the source, so as to implement reliable Multicast. The multicast is characterized in that one source end sends the same data to a plurality of destination ends. Multicast includes various implementation manners, for example, a source end sends the same data to N destination ends by using unicast through N times of sending; or the source end sends a copy of data, and the network device (switch or router) copies the packet by N copies and transmits the packet to N destination ends respectively. Obviously, in the second implementation manner, since the source end only transmits one piece of data, the transmission delay is smaller and the occupied bandwidth is smaller.

A basic architecture for implementing multicast, which is commonly used at present, is shown in fig. 1, in which a switch in fig. 1 establishes Remote Direct Memory Access (RDMA) unicast Connection (RC) connections with a multicast Source (MC Source) and each destination (Receiver), respectively. During multicast, the source end firstly sends metadata carrying multicast related information to the switch, and the switch receives and records the metadata. Then, the source end sends the multicast data packet to the switch. The exchanger copies the data message and sends the data message to each destination terminal according to the recorded metadata information. However, in the method for implementing multicast, as the switches and the source end and the destination end are all established with RC connections, the switches need to maintain complex RC connection related state machines. In addition, the switch is used as one end of the RC connection, and also needs to have a packet loss retransmission function, so that a message that has not been confirmed needs to be stored, thereby occupying precious switch cache resources. In addition, because RC connections are established between the switch and the source and destination terminals, the end-to-end communication principle is broken, which may cause a problem of generating false acknowledgement messages (ACKs). For example, when the switch receives the multicast data packet sent by the source end and then needs to reply an ACK, the source end considers that the multicast communication is completed when receiving the ACK. However, if the switch is disconnected from a destination, the multicast communication cannot be completed finally, which causes inconsistency between source end awareness and actual situation, and cannot realize reliable multicast.

Disclosure of Invention

The embodiment of the application provides a method, a device, a storage medium and equipment for realizing reliable multicast in an RDMA (remote direct memory Access) network, which can realize the reliable multicast in the RDMA network by using a switch on the premise of being compatible with a standard RDMA protocol.

In a first aspect, the present application provides a method for implementing reliable multicast in an RDMA network, the method including: the method comprises the steps that a switch receives an Envelope control message and a multicast data message sent by a source end, wherein the Envelope control message is a standard RDMA protocol message, the load of the Envelope control message comprises an Envelope message identification Tag, a network address of a destination end associated with a queue pair QP established by the source end, a QPN of the destination end queue pair associated with the queue pair QP established by the source end, memory area MR information of the destination end and a QPN of the source end, the switch identifies the Envelope control message according to the Envelope message identification Tag, extracts and records the content contained in the load of the Envelope control message, and then the switch replaces the corresponding domain in a multicast data message and/or a feedback message ACK/NACK corresponding to the multicast data message by the content in the recorded Envelope control message, and further can forward the multicast data message according to a preset rule.

Compared with the prior art, the embodiment of the application can realize the copy and the forwarding of the multicast message by using the switch under the condition of being compatible with the standard RDMA protocol and not needing to respectively establish RC (remote control) connection between the switch and the source end and the destination end, thereby realizing the reliable multicast in the RDMA network.

In a possible implementation manner, a switch replaces a corresponding domain in a multicast data packet and/or a feedback packet ACK/NACK corresponding to the multicast data packet with content in a recorded Envelope control packet, including: the exchanger replaces a destination network address, a destination QPN and a memory address domain in the BTH packet header in the multicast data message by the recorded content in the Envelope control message; and/or the switch replaces the source network address of the feedback message ACK/NACK corresponding to the multicast data message and the destination QPN in the BTH packet header by the content in the recorded Envelope control message.

In a possible implementation manner, the forwarding of multicast data packets by a switch according to a preset rule includes:

the switch sends the multicast data message to each output port according to a preset rule and the network address of the destination port establishing a queue pair QP with the source port;

wherein, the preset rule comprises:

the switch only forwards the multicast data message to a destination terminal which does not receive the corresponding acknowledgement message ACK;

the switch only forwards the last arrival in the ACK messages sent by all the destination terminals;

the switch only forwards one packet with the minimum sequence number PSN in the NACK messages sent by all the destination terminals.

In a second aspect, the present application further provides a method for implementing reliable multicast in an RDMA network, where the method includes: firstly, a source end and a destination end respectively establish a queue pair QP according to a standard RDMA protocol, wherein the QP comprises a queue pair identification number QPN; the source end QP is associated with the QP of the first destination end; all destination QP are associated with source QP; the association comprises a QPN and a network address of a designated QP opposite end, then a source end sends an Envelope control message and a multicast data message to the switch by using the QP, so that the switch replaces a corresponding domain in the multicast data message and/or a feedback message ACK/NACK corresponding to the multicast data message by using the content in the Envelope control message, and forwards the multicast data message according to a preset rule;

the load of the Envelope control message comprises an Envelope message identification Tag, a network address of a destination end, a QPN of the destination end, memory area MR information of the destination end and a QPN of a source end.

In a possible implementation manner, the sending, by a source end, an Envelope control packet and a multicast data packet to a switch by using a QP includes: and the source end transmits an Envelope control message and a multicast data message to the switch by using QP in an RDMA WRITE operation or Send/Recv operation mode.

In a third aspect, the present application further provides an apparatus for implementing reliable multicast in an RDMA network, where the apparatus includes:

the receiving unit is used for receiving an Envelope control message and a multicast data message sent by a source end by a switch, wherein the Envelope control message is a standard RDMA protocol message, and the load of the Envelope control message comprises an Envelope message identification Tag, a network address of a destination end associated with the source end and establishing a queue pair QP, a QPN of the destination end queue pair associated with the source end and establishing the queue pair QP, memory area MR information of the destination end and a QPN of the source end;

the extraction unit is used for identifying the Envelope control message by the exchanger according to the Envelope message identification Tag and extracting and recording the content contained in the Envelope control message;

the replacing unit is used for replacing the multicast data message and/or the corresponding domain in the feedback message ACK/NACK corresponding to the multicast data message by the exchanger by using the content in the recorded Envelope control message;

and the forwarding unit is used for forwarding the multicast data message by the switch according to a preset rule.

In one possible implementation, the replacement unit has a processor for: the exchanger replaces a destination network address, a destination QPN and a memory address domain in the BTH packet header in the multicast data message by the recorded content in the Envelope control message;

and/or the switch replaces the source network address of the feedback message ACK/NACK corresponding to the multicast data message and the destination QPN in the BTH packet header by the recorded content in the Envelope control message.

In a possible implementation manner, the forwarding unit is specifically configured to: the switch sends the multicast data message to each output port according to a preset rule and the network address of the destination port establishing a queue pair QP with the source port;

wherein, the preset rule comprises:

the switch only forwards the multicast data message to a destination end which does not receive the corresponding acknowledgement message ACK;

In a fourth aspect, the present application further provides an apparatus for implementing reliable multicast in an RDMA network, where the apparatus includes:

the device comprises a building unit, a queue pair QP generation unit and a queue pair identification unit, wherein the building unit is used for respectively building a queue pair QP by a source end and a destination end according to a standard RDMA protocol, and the QP comprises a queue pair identification number QPN; the source end QP is associated with the QP of the first destination end; all destination QPs are associated with source QPs; the association includes a QPN and a network address specifying a QP peer;

a sending unit, configured to send an Envelope control packet and a multicast data packet to a switch by using a QP at a source end, so that the switch replaces a corresponding domain in the multicast data packet and/or a feedback packet ACK/NACK corresponding to the multicast data packet by using content in the Envelope control packet, and forwards the multicast data packet according to a preset rule;

the load of the Envelope control packet includes an Envelope packet identification Tag, a network address of the destination end, a QPN of the destination end, memory region MR information of the destination end, and a QPN of the source end.

In a possible implementation manner, the sending unit is specifically configured to: and the source end sends an Envelope control message and a multicast data message to the switch by using QP and adopting an RDMA WRITE operation or a Send/Recv operation mode.

In a fifth aspect, the present application further provides an apparatus for implementing reliable multicast in RDMA network, where the apparatus includes: a memory, a processor;

a memory to store instructions; a processor configured to execute the instructions in the memory and perform the method of the first aspect and any one of the possible implementations thereof.

In a sixth aspect, the present application further provides an apparatus for implementing reliable multicast in RDMA network, where the apparatus includes: a memory, a processor;

a memory to store instructions; a processor configured to execute the instructions in the memory and perform the method of the second aspect and any one of its possible implementations.

In a seventh aspect, the present application also provides a computer-readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform any one of the methods described above.

According to the technical scheme, the embodiment of the application has the following advantages:

in order to implement reliable multicast of an RDMA network, a switch first receives an Envelope control packet and a multicast data packet sent by a source end, where the Envelope control packet is a standard RDMA protocol packet, and a load of the Envelope control packet includes an Envelope packet identification Tag, a network address of a destination end associated with the source end and having established a queue pair QP, a QPN of a destination end queue pair associated with the source end and having established a queue pair QP, memory region MR information of the destination end, and a QPN of the source end, and then the switch identifies the Envelope control packet according to the Envelope packet identification Tag, extracts and records contents included in the load of the Envelope control packet, and then replaces a corresponding field in a multicast data packet and/or a feedback packet ACK/NACK corresponding to the multicast data packet by using the recorded contents in the Envelope control packet, and further forwards the multicast data packet according to a preset rule. Therefore, the replication and forwarding of the multicast message can be realized by using the switch under the condition of being compatible with the standard RDMA protocol and not needing to respectively establish RC (remote control) connection between the switch and the source end and the destination end, and further, the reliable multicast in the RDMA network is realized.

Drawings

Fig. 1 is a basic architecture diagram for implementing multicast in a current common manner according to an embodiment of the present application;

fig. 2 is a flowchart of a method for implementing reliable multicast in an RDMA network according to an embodiment of the present application;

fig. 3 is an interaction diagram of packet forwarding by using a switch according to an embodiment of the present application;

fig. 4 is a flowchart of another method for implementing reliable multicast in an RDMA network according to an embodiment of the present application;

fig. 5 is one of exemplary diagrams of a scenario for implementing reliable multicast in an RDMA network according to an embodiment of the present application;

fig. 6 is a second exemplary diagram of a scenario for implementing reliable multicast in an RDMA network according to an embodiment of the present application;

fig. 7 is a block diagram illustrating a structure of a reliable multicast apparatus for implementing RDMA network according to an embodiment of the present application;

fig. 8 is a block diagram illustrating a structure of another apparatus for implementing reliable multicast in an RDMA network according to an embodiment of the present application;

fig. 9 is a schematic structural diagram of an apparatus for implementing reliable multicast in an RDMA network according to an embodiment of the present application;

fig. 10 is a schematic structural diagram of another apparatus for implementing reliable multicast in an RDMA network according to an embodiment of the present application.

Detailed Description

Embodiments of the present application are described below with reference to the accompanying drawings.

In the commonly used method for implementing multicast, a switch generally establishes RDMA unicast connections with a multicast source and destination respectively. During multicast, the source end firstly sends metadata carrying multicast related information to the switch, and the switch receives and records the metadata. Then, the source end sends the multicast data packet to the switch. The exchanger copies and sends the data message to each destination terminal according to the recorded metadata information. However, in the method for implementing multicast, as the switches and the source end and the destination end are all established with RC connections, the switches need to maintain complex RC connection related state machines. In addition, the switch, as one end of the RC connection, also needs to have a packet loss retransmission function, so that the message that has not been confirmed needs to be stored, thereby occupying precious switch cache resources. In addition, because RC connection is established between the switch and the source end and the destination end, the end-to-end communication principle is broken, and the problem that false ACK may be generated is caused. For example, when the switch receives the multicast data packet sent by the source end and then needs to reply an ACK, the source end considers that the multicast communication is completed when receiving the ACK. However, if the switch is disconnected from a destination, the multicast communication cannot be completed finally, which causes inconsistency between source end awareness and actual situation, and cannot realize reliable multicast.

In order to solve the above problems, the present application provides a reliable multicast method for implementing an RDMA network, which can implement reliable multicast in the RDMA network by using a switch on the premise of being compatible with a standard RDMA protocol. Specifically, the switch receives an Envelope control message and a multicast data message sent by a source terminal, wherein the Envelope control message is a standard RDMA protocol message, a load of the Envelope control message includes an Envelope message identification Tag, a network address of a destination terminal associated with a source terminal and a queue pair QP established with the source terminal, a QPN of the destination terminal queue pair associated with the source terminal and a memory area MR information of the destination terminal and a QPN of the source terminal, the switch identifies the Envelope control message according to the Envelope message identification Tag and extracts and records contents included in the load of the Envelope control message, the switch replaces a corresponding domain in a feedback message ACK/NACK corresponding to the multicast data message and/or the multicast data message by using the recorded contents in the Envelope control message, and the multicast data message can be further forwarded according to a preset rule. Therefore, the replication and forwarding of the multicast message can be realized by using the switch under the condition of being compatible with the standard RDMA protocol and not needing to respectively establish RC (remote control) connection between the switch and the source end and the destination end, and further the reliable multicast in the RDMA network is realized.

Referring to fig. 2, a schematic flow chart of a method for implementing reliable multicast in an RDMA network according to this embodiment is shown, where the method includes the following steps:

s201: the method comprises the steps that an Envelope control message and a multicast data message sent by a source end are received by a switch, wherein the Envelope control message is a standard RDMA protocol message, and the load of the Envelope control message comprises an Envelope message identification Tag, a network address of a destination end associated with the source end and establishing a queue pair QP, a QPN of the destination end queue pair associated with the source end and establishing the queue pair QP, memory area MR information of the destination end and a QPN of the source end.

In this embodiment, to implement reliable multicast in an RDMA network, first, a multicast source and a multicast destination need to respectively establish a Queue Pair (QP) according to a standard RDMA protocol, where the QP includes a queue pair identifier QPN. The source end QP is associated with the QP of the first destination end; all destination QPs are associated with source QPs; wherein the association includes a QPN and a network address specifying the QP peer. Specifically, the peer (i.e., the destination) associated with the source QP is a specific virtual QPN (e.g., 0x 800001), the Packet Sequence Number (PSN) is 0, and the network address is the network address of the destination dst 0. And the QPN of the opposite end associated with the destination QP is a specific virtual QPN, the PSN at the start is 0, and the network address is the network address of the source end.

On the basis, the exchanger receives an Envelope control message and a multicast data message which are sent by the source end by using the established QP. The Envelope control packet is a standard RDMA protocol packet, and a load of the Envelope control packet includes an Envelope packet identification Tag, a network address of a destination end, a QPN of the destination end, memory location MR information of the destination end, and a QPN of a source end, for example, one possible format of the Envelope packet is shown in table 1 below:

ETH, dst0's IP, UDP, BTH (QPN =0x 800001), RETH (for putting MR information)
	Envelope Tag
dst1～dstN’s IP
	dst0～dstN’s QPN
dst0～dstN’s MR
	source’s QPN

TABLE 1

Thus, after receiving the Envelope control packet and the corresponding multicast data packet shown in table 1 sent by the source end, the switch may implement reliable RDMA network multicast by executing the subsequent steps S202 to S204 according to the packet forwarding flow shown in fig. 3.

S202: the exchanger identifies the Envelope control message according to the Envelope message identification Tag, and extracts and records the content contained in the load of the Envelope control message.

In this embodiment, after receiving the Envelope control packet and the multicast data packet sent by the source end through step S201, the switch may further identify the multicast packet including the Envelope control packet and the multicast data packet according to the virtual QPN (which is a part of the BTH), identify the Envelope control packet according to the Envelope packet identification Tag, and extract and record content included in a load of the Envelope control packet, that is, the Envelope packet identification Tag, a network address of a destination end associated with the source end and having established a queue pair QP, a QPN of a destination end queue pair associated with the source end and having established a queue pair QP, memory location MR information of the destination end, a QPN of the source end, and the like, and after storing these pieces of information, may be used to execute subsequent step S203.

S203: and the exchanger replaces the corresponding domain in the multicast data message and/or the feedback message ACK/NACK corresponding to the multicast data message by the recorded content in the Envelope control message.

In this embodiment, after the switch extracts and records the content included in the Envelope control packet in step S202, the switch may further replace the corresponding domain in the multicast data packet and/or the feedback packet ACK/NACK corresponding to the multicast data packet with the content in the recorded Envelope control packet. And/or the switch may replace the source network address of the feedback message ACK/NACK and the destination QPN in the BTH header corresponding to the multicast data message with the content in the recorded Envelope control message.

Specific alternatives are shown in table 2 below:

TABLE 2

It should be noted that, if the message egress port and the server are directly connected, that is, in the case of multi-layer networking, the above replacement operation occurs in the last hop.

S204: and the switch forwards the multicast data message according to a preset rule.

In this embodiment, the switch copies the Envelope control packet through step S203, and after replacing the corresponding field in the multicast data packet with the content in the Envelope control packet, the switch may further forward the multicast data packet according to a preset rule, so as to implement reliable multicast in the RDMA network.

An optional implementation manner is that the switch may send the multicast data packet to each egress port according to the network address of the destination port that establishes the queue pair QP with the source port. And forwarding the subsequently received multicast data message, ACK message and NACK message according to a preset rule.

Wherein, the preset rules comprise the following 3 rules:

Specifically, the switch records the ACK PSN status of each port to which the multicast destination is connected. Wherein the ACK PSN status is updated upon receiving the ACK message/NACK message from the port. If ACK PSN = PSN _ ACK recorded by a certain port currently, if an ACK message is received, PSN = m, and m > PSN _ ACK, updating PSN _ ACK = m; if a NACK message is received with PSN = m and m-1>, PSN _ack, then PSN _ ack = m-1 is updated.

When the multicast data message is forwarded, for each switch port, if the port does not receive the ACK corresponding to the message, that is, the PSN > PSN _ ACK of the data message, the message is sent out from the port. Otherwise, the port will discard the message.

When the ACK message is forwarded, the switch only forwards the last ACK message. That is, if the PSN of the arriving ACK packet is m and there is m < PSN _ ACK _ i for the switch port i connected to any multicast destination, the packet is forwarded. Otherwise, the ACK message is discarded.

When the NACK message is forwarded, the switch maintains a NACK message cache, and defines the PSN of the cached NACK message as PSN _ NACK _ buf. On this basis, the PSN of the received NACK message is recorded as m. If the NACK message cache of the switch is empty or m-1-straw psn _nack _ buf, the NACK message is written into the cache, and psn _ NACK _ buf = m-1 is updated, otherwise, the message is discarded. In addition, if psn _ NACK _ buf is less than or equal to psn _ ack _ i for any switch port i connected with the multicast destination end, the cached NACK message is sent out, and the NACK message cache is emptied.

Thus, by adopting virtual QPN, on the premise of being compatible with standard RDMA protocol, the multicast and unicast messages can be effectively distinguished, and the multicast messages from different multicast groups can be distinguished. And the use of the Envelope control message can ensure that the exchanger does not need to store the state information related to the multicast for a long time. The relevant state information can be deleted after the transmission of the subsequent multicast data message is finished. Meanwhile, after the switch replaces the related domain, the multicast related message can be correctly received by the standard RDMA network card of the source end and the destination end. And when the retransmission is sent, the switch only sends the retransmission message to the destination end which does not receive the corresponding ACK, so that the other destination ends are prevented from receiving the redundant retransmission message. Meanwhile, each data of the multicast source end can only receive 1 part of ACK message, and the ACK message represents that all the destination ends successfully receive the corresponding data. In addition, when packet loss is sent, the multicast source end can also retransmit according to the rules of the standard RDMA protocol, thereby ensuring the reliability of multicast.

In summary, in order to implement reliable multicast in an RDMA network, in this embodiment, in order to implement reliable multicast in the RDMA network, a switch first receives an Envelope control packet and a multicast data packet sent by a source end, where the Envelope control packet is a standard RDMA protocol packet, a load of the Envelope control packet includes an Envelope packet identification Tag, a network address of a destination end associated with the source end, a QPN of a destination end queue pair associated with the source end, a memory area MR information of the destination end, and a QPN of the source end, and then, the switch identifies the Envelope control packet according to the Envelope packet identification Tag, and extracts and records a content included in the load of the Envelope control packet, and then, the switch replaces a corresponding domain in a feedback packet ACK/NACK corresponding to a multicast data packet and/or a multicast data packet by using the recorded content in the Envelope control packet, and further may forward the multicast data packet according to a preset rule. Therefore, the replication and forwarding of the multicast message can be realized by using the switch under the condition of being compatible with the standard RDMA protocol and not needing to respectively establish RC (remote control) connection between the switch and the source end and the destination end, and further, the reliable multicast in the RDMA network is realized.

Referring to fig. 4, this figure is a flowchart of another method for implementing reliable multicast in an RDMA network according to an embodiment of the present application. The method is described below.

S401: respectively establishing a queue pair QP by a source end and a destination end according to a standard RDMA protocol, wherein the QP comprises a queue pair identification number QPN; the source end QP is associated with the QP of the first destination end; all destination QP are associated with source QP; the association includes a QPN and a network address specifying the QP peer.

S402: the source end sends an Envelope control message and a multicast data message to the switch by using the QP, so that the switch replaces a corresponding domain in the multicast data message and/or a feedback message ACK/NACK corresponding to the multicast data message by using the content in the Envelope control message, and forwards the multicast data message according to a preset rule.

Specifically, after the source end establishes a queue pair QP with the destination end according to a standard RDMA protocol, the QP may be further used by the source end to Send an Envelope control packet and a multicast data packet to the switch by using an RDMA WRITE operation or a Send/Recv operation mode.

Next, for convenience of understanding, an example will now be described that the method for implementing reliable multicast in an RDMA network provided in the present application is applied to a scenario of multiple copy writing based on a RoCEv2 network in a storage system:

as shown in fig. 5, the left diagram shows the unicast implementation of the multi-copy write of the current storage system. That is, for each data packet, a client (i.e., a source end) initiates N (N =3 in the figure) unicast writes, and sends the data packet to N (N =3 in the figure) servers (i.e., destination ends), respectively. After the method for implementing reliable multicast of the RDMA network provided by the present application is adopted, the client can multicast the data packet to N (N =3 in the figure) servers by one-time writing as shown in the right-side diagram of fig. 5. Therefore, under the conditions of smaller initiating time delay and less occupied bandwidth, reliable multicast is realized, and the specific flow is as follows:

(1) The Client and the corresponding N servers form a multicast group. The Client and Server respectively create QPs according to standard RDMA protocols. When QP association is performed, the QPNs associated with QPs of the Client and the Server are both 0x8xxxxx, and different virtual QPNs are used by different multicast groups. The network address of the opposite end associated with the QP of the Client is the network address of the Server0, and the network address of the opposite end associated with the QP of each Server is the network address of the Client.

(2) And the Client sends an Envelope control message by using the QP, wherein the format of the Envelope control message is shown in the table 1.

(3) And the Client sends the multicast data message by using the QP. And the data message is sent by adopting RDMA WRITE unilateral writing.

(4) And the exchanger processes the Envelope control message, the multicast data message and the ACK/NACK message by executing the steps S201 to S204. After processing, each Server can correctly receive an Envelope control message and a multicast data message sent by a Client; for the Client, each time one data is sent, one ACK packet can be received, and the ACK packet represents that all N servers have correctly received the corresponding packet.

(5) If packet loss occurs, the Client can correctly receive the NACK message processed by the switch, and then packet loss retransmission can be carried out according to the standard RDMA protocol so as to realize reliable multicast.

Therefore, the Client can reliably multicast the data message to all the servers through one-time sending, so that the task completion time delay of multi-copy write operation can be effectively reduced, and the IOPS performance of the system is improved.

In addition, for ease of understanding, the application of the method for reliable multicast over RDMA networks provided by the present application to a scenario of distributed high performance computing HPC is now exemplified:

as shown in fig. 6, similar to the exemplary scenario shown in fig. 5, in this exemplary scenario, one multicast source multicasts the same data to all destinations through one transmission by using the method for implementing reliable multicast over RDMA network provided in this application. The main differences between the two methods include the following two points:

(1) In HPC systems, send/Recv bilateral operations are used instead of RDMA Write unilateral operations. Therefore, the Envelope control message does not need to include the memory address information of the destination.

(2) HPC systems often include a large number of nodes, and thus, there are cases of multi-layer networking. When the network is formed by multiple layers, each switch on the path replaces and forwards the related message by using the method for realizing reliable multicast of the RDMA network provided by the application. It should be noted here that, when the switch performs message replication, if a certain port corresponds to multiple destination ports, for example, a port above switch 0 is connected to dst2 and dst3 in fig. 6, the switch should only send one copy of the replicated data message from the port.

Therefore, the multicast source end can reliably multicast the data to all the destination ends by one-time transmission, and the completion time delay of the multicast operation is reduced, so that the completion time of the multicast operation is reduced to be close to the unicast operation.

To facilitate better implementation of the above-described aspects of the embodiments of the present application, a related apparatus for implementing the above-described aspects is also provided below. Referring to fig. 7, an embodiment of the present application provides an apparatus 700 for implementing reliable multicast in an RDMA network. The apparatus 700 may include: a receiving unit 701, an extracting unit 702, a replacing unit 703 and a forwarding unit 704. The receiving unit 701 is configured to execute S201 in the embodiment shown in fig. 2. The extracting unit 702 is configured to execute S202 in the embodiment shown in fig. 2. The replacing unit 703 is configured to execute S203 in the embodiment shown in fig. 2. The forwarding unit 704 is configured to execute S204 in the embodiment shown in fig. 2. In particular, the method comprises the following steps of,

a receiving unit 701, configured to receive an Envelope control packet and a multicast data packet sent by a source end by a switch, where the Envelope control packet is a standard RDMA protocol packet, and a load of the Envelope control packet includes an Envelope packet identification Tag, a network address of a destination end associated with the source end and having established a queue pair QP, a QPN of the destination end queue pair associated with the source end and having established the queue pair QP, memory area MR information of the destination end, and a QPN of the source end;

an extracting unit 702, configured to identify an Envelope control packet according to an Envelope packet identification Tag, and extract and record contents included in the Envelope control packet by the switch;

a replacing unit 703, configured to replace, by the switch, a corresponding domain in the multicast data packet and/or a feedback packet ACK/NACK corresponding to the multicast data packet by using the content in the recorded Envelope control packet;

a forwarding unit 704, configured to forward the multicast data packet according to a preset rule by the switch.

In an implementation manner of this embodiment, the replacing unit 703 is provided with:

the switch replaces a destination network address in the multicast data message, a destination QPN in the BTH packet header and a memory address domain by using the content in the recorded Envelope control message;

In an implementation manner of this embodiment, the forwarding unit 704 is specifically configured to: the switch sends the multicast data message to each output port according to a preset rule and the network address of the destination port establishing a queue pair QP with the source port;

wherein, the preset rule comprises:

To facilitate better implementation of the above-described aspects of the embodiments of the present application, another related apparatus for implementing the above-described aspects is provided below. Referring to fig. 8, another apparatus 800 for implementing reliable multicast for RDMA network according to an embodiment of the present application is provided. The apparatus 800 may include: a setup unit 801 and a sending unit 802. The establishing unit 801 is configured to execute S401 in the embodiment shown in fig. 4. The sending unit 802 is configured to execute S402 in the embodiment shown in fig. 4. In particular, the method comprises the following steps of,

a establishing unit 801, configured to respectively establish, by a source end and a destination end, a queue pair QP according to a standard RDMA protocol, where the QP includes a queue pair identification number QPN; the source end QP is associated with the QP of the first destination end; all destination QP are associated with source QP; the association includes a QPN and a network address specifying a QP peer;

a sending unit 802, configured to send an Envelope control packet and a multicast data packet to a switch by using a QP, so that the switch replaces a corresponding domain in the multicast data packet and/or a feedback packet ACK/NACK corresponding to the multicast data packet by using the content in the Envelope control packet, and forwards the multicast data packet according to a preset rule;

In an implementation manner of this embodiment, the sending unit 802 is specifically configured to: and the source end transmits an Envelope control message and a multicast data message to the switch by using QP in an RDMA WRITE operation or Send/Recv operation mode.

Referring to fig. 9, an embodiment of the present application provides an apparatus 900 for implementing RDMA network reliable multicast, the apparatus including a memory 901, a processor 902 and a communication interface 903,

a memory 901 for storing instructions;

a processor 902, configured to execute the instructions in the memory 901, and execute the above method for implementing reliable multicast in RDMA network in the embodiment shown in fig. 2;

a communication interface 903 for performing communication.

The memory 901, the processor 902, and the communication interface 903 are connected to each other by a bus 904; the bus 904 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 9, but this does not indicate only one bus or one type of bus.

In a specific embodiment, in order to implement reliable multicast in an RDMA network, the processor 902 is configured to receive an Envelope control packet and a multicast data packet sent by a source end, where the Envelope control packet is a standard RDMA protocol packet, and a load of the Envelope control packet includes an Envelope packet identification Tag, a network address of a destination end that establishes a queue pair QP association with the source end, a QPN of a destination end queue pair that establishes a queue pair QP association with the source end, memory region MR information of the destination end, and a network address of the source end, then the switch identifies the Envelope control packet according to the Envelope packet identification Tag, and extracts and records content included in the load of the Envelope control packet, and then the switch replaces a corresponding domain in the multicast data packet with content in a feedback packet ACK/NACK corresponding to the recorded Envelope control packet and/or multicast data packet, and further may forward the multicast data packet according to a preset rule. Therefore, the replication and forwarding of the multicast message can be realized by using the switch under the condition of being compatible with the standard RDMA protocol and not needing to respectively establish RC (remote control) connection between the switch and the source end and the destination end, and further, the reliable multicast in the RDMA network is realized. For a detailed processing procedure of the processor 902, please refer to the detailed description of S201, S202, S203, and S204 in the embodiment shown in fig. 2, which is not repeated herein.

Referring to fig. 10, another apparatus 1000 for implementing RDMA network reliable multicast is provided in an embodiment of the present application, the apparatus includes a memory 1001, a processor 1002 and a communication interface 1003,

a memory 1001 for storing instructions;

a processor 1002, configured to execute the instructions in the memory 1001 to execute the above method for implementing reliable multicast over RDMA network in the embodiment shown in fig. 4;

a communication interface 1003 for performing communication.

The memory 1001, the processor 1002, and the communication interface 1003 are connected to each other by a bus 1004; the bus 1004 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 10, but this is not intended to represent only one bus or type of bus.

In a specific embodiment, the processor 1002 is configured to, in order to implement reliable multicast in an RDMA network, first, establish a queue pair QP with a destination according to a standard RDMA protocol, where the QP includes a queue pair identifier QPN; the source end QP is associated with the QP of the first destination end; all destination QPs are associated with source QPs; associating QPN and network address of a designated QP opposite end, then sending an Envelope control message and a multicast data message to the switch by the source end by using the QP so that the switch replaces a corresponding domain in the multicast data message and/or a feedback message ACK/NACK corresponding to the multicast data message by using the content in the Envelope control message, and forwarding the multicast data message according to a preset rule; the load of the Envelope control message comprises an Envelope message identification Tag, a network address of a destination end, a QPN of the destination end, MR information of a memory area and a QPN of a source end. For a detailed processing procedure of the processor 1002, please refer to the detailed description of S401 and S402 in the embodiment shown in fig. 4, which is not described herein again.

The memory 901 and the memory 1001 may be a random-access memory (RAM), a flash memory (flash), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register (register), a hard disk, a removable hard disk, a CD-ROM, or any other form of storage medium known to those skilled in the art.

The processor 902 and the processor 1002 may be, for example, a Central Processing Unit (CPU), a general purpose processor, a Digital Signal Processor (DSP), an application-specific integrated circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure of the embodiments of the application. A processor may also be a combination of computing functions, e.g., comprising one or more microprocessors, a DSP and a microprocessor, or the like.

The communication interface 903 and the communication interface 1003 may be interface cards, for example, and may be ethernet (ethernet) interfaces or Asynchronous Transfer Mode (ATM) interfaces.

Embodiments of the present application also provide a computer-readable storage medium, which includes instructions that, when executed on a computer, cause the computer to perform the above method for implementing reliable multicast over RDMA networks.

The terms "first," "second," and the like in the description and in the claims of the present application and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances and are merely descriptive of the various embodiments of the application and how objects of the same nature can be distinguished. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one type of logical functional division, and other divisions may be realized in practice, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

The above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present application.

Claims

1. A method for implementing reliable multicast over an RDMA network, the method comprising:

the method comprises the steps that an exchanger receives an Envelope control message and a multicast data message sent by a source end, wherein the Envelope control message is a standard RDMA protocol message, and the load of the Envelope control message comprises an Envelope message identification Tag, a network address of a destination end associated with a queue pair QP established by the source end, a QPN of the destination end queue pair associated with the queue pair QP established by the source end, memory area MR information of the destination end and a QPN of the source end;

the switch identifies the Envelope control message according to the Envelope message identification Tag, and extracts and records the content contained in the load of the Envelope control message;

the exchanger replaces the corresponding domain in the multicast data message and/or the feedback message ACK/NACK corresponding to the multicast data message by the recorded content in the Envelope control message;

and the switch forwards the multicast data message according to a preset rule.

2. The method according to claim 1, wherein the replacing, by the switch, the corresponding field in the multicast data packet and/or the feedback packet ACK/NACK corresponding to the multicast data packet by using the recorded content in the Envelope control packet includes:

the exchanger replaces a destination network address in the multicast data message, a destination QPN in a BTH packet header and a memory address domain by using the recorded content in the Envelope control message;

3. The method of claim 1, wherein the forwarding, by the switch, the multicast data packet according to a preset rule comprises:

wherein the preset rule comprises:

4. A method for implementing reliable multicast in an RDMA network, the method comprising:

respectively establishing a queue pair QP by a source end and a destination end according to a standard RDMA protocol, wherein the QP comprises a queue pair identification number QPN; the source end QP is associated with the QP of the first destination end; all destination QP are associated with source QP; the association includes a QPN and a network address specifying a QP peer;

the source end sends an Envelope control message and a multicast data message to the switch by using the QP, so that the switch replaces a corresponding domain in the multicast data message and/or a feedback message ACK/NACK corresponding to the multicast data message by using the content in the Envelope control message, and forwards the multicast data message according to a preset rule;

the load of the Envelope control packet includes an Envelope packet identification Tag, a network address of the destination, a QPN of the destination, memory region MR information of the destination, and a QPN of the source.

5. The method according to claim 4, wherein the sending, by the source end, an Envelope control packet and a multicast data packet to the switch by using the QP includes:

and the source end sends an Envelope control message and a multicast data message to the switch by using the QP and adopting an RDMA WRITE operation or a Send/Recv operation mode.

6. An apparatus for implementing RDMA network reliable multicast, the apparatus comprising:

a receiving unit, configured to receive an Envelope control packet and a multicast data packet sent by a source end by a switch, where the Envelope control packet is a standard RDMA protocol packet, and a load of the Envelope control packet includes an Envelope packet identification Tag, a network address of a destination end that establishes a queue pair QP association with the source end, a QPN of a destination end queue pair that establishes a queue pair QP association with the source end, memory area MR information of the destination end, and a QPN of the source end;

an extraction unit, configured to identify the Envelope control packet according to the Envelope packet identification Tag, and extract and record contents included in the Envelope control packet by the switch;

a replacing unit, configured to replace, by the switch, a corresponding domain in the multicast data packet and/or a feedback packet ACK/NACK corresponding to the multicast data packet by using the recorded content in the Envelope control packet;

7. The apparatus of claim 6, wherein the replacement unit has means for:

8. The apparatus according to claim 6, wherein the forwarding unit is specifically configured to:

wherein the preset rule comprises:

9. An apparatus for implementing RDMA network reliable multicasting, the apparatus comprising:

the device comprises a establishing unit, a queue pair identification unit and a queue pair identification unit, wherein the establishing unit is used for respectively establishing a queue pair QP by a source end and a destination end according to a standard RDMA protocol, and the QP comprises a queue pair identification number QPN; the source end QP is associated with the QP of the first destination end; all destination QPs are associated with source QPs; the association includes a QPN and a network address specifying a QP peer;

a sending unit, configured to send an Envelope control packet and a multicast data packet to a switch by using the QP, so that the switch replaces a corresponding domain in the multicast data packet and/or a feedback packet ACK/NACK corresponding to the multicast data packet by using the content in the Envelope control packet, and forwards the multicast data packet according to a preset rule;

10. The apparatus according to claim 9, wherein the sending unit is specifically configured to:

and the source end sends an Envelope control message and a multicast data message to the switch by using the QP in an RDMA WRITE operation or Send/Recv operation mode.

11. An apparatus for implementing RDMA network reliable multicasting, the apparatus comprising a memory, a processor;

the memory to store instructions;

the processor, configured to execute the instructions in the memory, to perform the method of any of claims 1-3.

12. An apparatus for implementing RDMA network reliable multicasting, the apparatus comprising a memory, a processor;

the memory to store instructions;

the processor, configured to execute the instructions in the memory, to perform the method of any of claims 4-5.

13. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method of any of claims 1-5 above.