WO2022151820A1

WO2022151820A1 - Data transmission system, data transmission method, and network device

Info

Publication number: WO2022151820A1
Application number: PCT/CN2021/129667
Authority: WO
Inventors: 卢胜文
Original assignee: 华为技术有限公司
Priority date: 2021-01-14
Filing date: 2021-11-10
Publication date: 2022-07-21
Also published as: CN114765631A

Abstract

The present application discloses a data transmission system, a data transmission method and a network device, which are used for improving transmission efficiency. Said system is applied to remote direct memory access (RDMA) data transmission. Said system comprises: N pieces of data of N VMs running on a first host need to be transmitted to a second host by means of RDMA, wherein the N VMs are VMs that can normally extract data, and a first network device in the first host may encapsulate the N pieces of data and write addresses of the N pieces of data according to an RDMA protocol, and then send a message generated by means of encapsulation to a second network device on the second host. The second network device may decapsulate the message, extract the N pieces of data and the write addresses of the N pieces of data therefrom and store the N pieces of data at locations indicated by the write addresses.

Description

Data transmission system, data transmission method and network device

This application claims the priority of the Chinese patent application filed on January 4, 2021 with the application number 202110049814.4 and the invention titled "Data Transmission System, Data Transmission Method and Network Equipment", the entire contents of which are incorporated by reference in in this application.

technical field

The present application relates to the field of network communication technologies, and in particular, to a data transmission system, a data transmission method, and a network device.

Background technique

The Remote Direct Memory Access (RDMA) protocol is a protocol that transfers data directly from one system to another in memory over a network without operating system intervention. The RDMA protocol encapsulates the data to be transmitted into one or more RDMA packets, and sends the one or more RDMA packets from the sender to the receiver.

RDMA transfer is to transfer data from one system to the memory of another system through a send queue (SQ), wherein the data in the SQ only includes the data of one virtual machine (VM).

When a service involves data of multiple VMs, the data of multiple VMs cannot be transferred to the memory of another system at one time, which affects the transmission efficiency.

SUMMARY OF THE INVENTION

The present application provides a data transmission system, a data transmission method and a network device for improving transmission efficiency.

A first aspect of the present application provides a data transmission system, the system includes: the data transmission system includes a first network device and a second network device, the first network device is set on the first host, and the second network device is set on the second network device On the host, there are N virtual machines running on the first host; the first network device is used to obtain N data, the N data comes from the N VMs, and the N data and the N data are converted according to the remote direct memory access RDMA protocol. The write address of the data is encapsulated into a packet, and the packet is sent to the second network device, where N is an integer greater than 1; the second network device is used to receive the packet and decapsulate the packet to obtain N data and N data write addresses are stored on the second host according to the N data write addresses.

In the above-mentioned first aspect, N data of N VMs running on the first host need to be transferred to the second host through RDMA, wherein the N VMs are VMs that can normally extract data, and the first network in the first host The device may encapsulate the above-mentioned N pieces of data and the write addresses of the N pieces of data according to the RDMA protocol, and then send the encapsulated packet to the second network device on the second host. The second network device may decapsulate the packet, extract the N data and the write address of the N data, and store the N data in the location indicated by the write address. The first network device can directly send the data of the multiple virtual machines to the second network device to improve transmission efficiency.

In a possible implementation manner, the first network device is configured to acquire the identifiers and memory addresses of the N VMs; and acquire N pieces of data according to the identifiers and memory addresses of the N VMs.

In the above possible implementation manner, the first network device can directly obtain N pieces of data according to the VM identifiers and VM memory addresses of the N pieces of data that it wants to RDMA to transmit to the second network device, and the write address received by the second network device includes VM. ID and VM memory address, the second network device can directly use the write address to store N pieces of data without going through a chip logical address (CLA), which can reduce the processing flow.

In a possible implementation manner, an abnormal VM exists on the first host, and data cannot be obtained according to the identifier and memory address of the abnormal VM; the first network device is configured to obtain data according to the identifier and memory address of the abnormal VM and some of the N VMs. The identifier and memory address of the VM are obtained, and M pieces of data are obtained, where M is a positive integer less than N; the M pieces of data are encapsulated into an exception message according to the RDMA protocol.

In the above-mentioned possible implementation manner, if there is an abnormal VM in the N VMs, for example, the VM fails, shuts down or restarts, etc., the first network device cannot obtain data according to the identification and memory address of the abnormal VM, that is, according to the identification and memory of the abnormal VM. An exception message generated by encapsulating the address, the identifiers of some of the VMs in the N VMs, and the memory address cannot be sent to the second network device through a queue pair (queue pair, QP) message.

In a possible implementation manner, the first network device is configured to generate a packet sequence, where the packet sequence includes an abnormal packet and at least one packet.

In the above possible implementation manner, when an abnormal VM exists in the N VMs, the first network device needs to generate a message sequence from the abnormal message and at least one of the above messages, so that the first network device can generate a message sequence according to the message of the message sequence. The sequence number sends the abnormal message and the message in sequence, so as to improve the practicability of the solution.

In a possible implementation manner, the first network device is configured to modify the packet sequence, wherein modifying the packet sequence includes deleting abnormal packets, and adding padding packets to the packet sequence.

In the above possible implementation manner, there is an abnormal message in the message sequence, and the abnormal message cannot be sent to the second network device, causing the second network device to continuously request message retransmission, and multiple retransmission failures will cause the first network device. The link with the second network device is disconnected, and the RDMA transmission fails. The present application can delete abnormal packets in the packet sequence, and then supplement the packet sequence with padding packets, which are invalid packets, so as to keep the data length unchanged and avoid retransmission due to inconsistent data lengths.

In a possible implementation manner, the second network device is configured to receive the modified packet sequence, determine the padding packet in the modified packet sequence, and delete the padding packet.

In the above-mentioned possible implementation manner, after receiving the above-mentioned modified packet sequence, the second network device can determine the invalid packet by checking, that is, determine the padding packet in the modified packet sequence, and then delete the modified packet. The padding message is used to improve the reliability of message transmission.

A second aspect of the present application provides a data transmission method, the method includes: a first network device acquires N pieces of data, the first network device is set on a first host, and N virtual machines VM run on the first host, and N The data comes from N VMs, where N is an integer greater than 1; the first network device encapsulates the N data and the write address of the N data into one message according to the remote direct memory access RDMA protocol; the first network device Send the message to the second network device.

In a possible implementation manner, an abnormal VM runs on the first host, and the abnormal VM cannot obtain data according to the identifier and memory address of the abnormal VM. The method further includes: the first network device according to the identifier and memory address of the abnormal VM and the The identifiers and memory addresses of some VMs in the N VMs are obtained, and M pieces of data are obtained, wherein M is a positive integer less than N; the first network device encapsulates the M pieces of data into an exception message according to the RDMA protocol.

In a possible implementation manner, the method further includes: the first network device generates a packet sequence, where the packet sequence includes an abnormal packet and at least one packet.

In a possible implementation manner, the method further includes: the first network device modifies the packet sequence, wherein modifying the packet sequence includes deleting abnormal packets, and adding padding packets to the packet sequence.

A third aspect of the present application provides a data transmission method, the method includes: a second network device receives a message from a first network device, the first network device is set on the first host, and the second network device is set on a second network device On the host, there are N virtual machines running on the first host, and the message is a message generated by encapsulating the N data and the write address of the N data according to the remote direct memory access RDMA protocol, and the N data comes from the N data. VM, where N is an integer greater than 1; the second network device decapsulates the packet to obtain N data and N data write addresses; the second network device decapsulates the N data in the first N pieces of data are stored on the second host.

In a possible implementation manner, the method further includes: the second network device receives a modified packet sequence, where the modified packet sequence includes a padding packet; and the second network device determines a packet in the modified packet sequence. Padding packets, and deleting padding packets.

A fourth aspect of the present application provides a network device, comprising: an acquisition unit configured to acquire N pieces of data, the network device is set on a first host, N virtual machines VM are running on the first host, and the N pieces of data come from N VMs, where N is an integer greater than 1; the encapsulation unit is used to encapsulate the N data and the write addresses of the N data into a message according to the remote direct memory access RDMA protocol; the sending unit is used to encapsulate the message The message is sent to the second network device.

The network device is configured to execute the method of the second aspect or any one of the implementation manners of the second aspect.

A fifth aspect of the present application provides a network device, including: a receiving unit configured to receive a message from a first network device, the first network device is set on the first host, the network device is set on the second host, and the first network device is set on the second host. There are N virtual machine VMs running on a host, and the message is a message generated by encapsulating N data and N data write addresses according to the remote direct memory access RDMA protocol. The N data comes from N VMs, where, N is an integer greater than 1; the decapsulation unit is used to decapsulate the message to obtain the N data and the write address of the N data; the storage unit is used to store the N data in the second according to the write address of the N data N pieces of data are stored on the host.

The network device is configured to execute the method of the third aspect or any one of the implementation manners of the third aspect.

A sixth aspect of the present application provides a network device, including: a processor, a memory, and a communication interface, where the processor is configured to execute instructions stored in the memory, so that the network device executes the second aspect or any one of the second aspects The method provided by the optional manner, the communication interface is used for receiving or sending an indication. For the specific details of the network device provided in the sixth aspect, reference may be made to the second aspect or any optional manner of the second aspect, which will not be repeated here.

A seventh aspect of the present application provides a network device, including: a processor, a memory, and a communication interface, where the processor is configured to execute instructions stored in the memory, so that the network device executes the third aspect or any one of the third aspects The method provided by the optional manner, the communication interface is used for receiving or sending an indication. For the specific details of the network device provided in the seventh aspect, reference may be made to the third aspect or any optional manner of the third aspect, which will not be repeated here.

An eighth aspect of the present application provides a computer-readable storage medium, where a program is stored in the computer-readable storage medium. When the computer executes the program, the computer executes the second aspect or any optional manner of the second aspect. method.

A ninth aspect of the present application provides a computer-readable storage medium, where a program is stored in the computer-readable storage medium, and when the computer executes the program, the computer executes the third aspect or any optional manner provided by the third aspect. method.

A tenth aspect of the present application provides a computer program product. When the computer program product is executed on a computer, the computer executes the method provided in the second aspect or any optional manner of the second aspect.

An eleventh aspect of the present application provides a computer program product. When the computer program product is executed on a computer, the computer executes the method provided in the third aspect or any optional manner of the third aspect.

Description of drawings

FIG. 1 is a system frame diagram of a data transmission method provided by an embodiment of the present application;

2 is a schematic structural diagram of a data transmission system provided by an embodiment of the present application;

FIG. 3 is an embodiment of a data transmission method provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of a CLA address provided in an embodiment of the present application;

FIG. 5 is another embodiment of a data transmission method provided by an embodiment of the present application;

6 is a schematic diagram of a combined strip provided in an embodiment of the present application;

7 is a schematic diagram of a padding message provided by an embodiment of the present application;

8 is another schematic diagram of filling a filling message provided by an embodiment of the present application;

FIG. 9 is a schematic structural diagram of a network device provided by an embodiment of the present application;

FIG. 10 is a schematic structural diagram of a network device provided by an embodiment of the present application;

FIG. 11 is a schematic structural diagram of a network device provided by an embodiment of the application;

FIG. 12 is a schematic structural diagram of a network device provided by an embodiment of the present application.

Detailed ways

Embodiments of the present application provide a data transmission system, a data transmission method, and a network device, which are used to reduce processing flow and memory occupation. The embodiments of the present application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Those of ordinary skill in the art know that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

The terms "first", "second" and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It is to be understood that data so used may be interchanged under appropriate circumstances so that the embodiments described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

The data transmission method of the embodiment of the present application is mainly applicable to the application scenario of RDMA transmission. When an application program initiates an RDMA read/write request, the system does not perform a data copy action, which reduces the amount of time spent in the kernel space and in the processing of network communication. The number of user space context switches. Without any kernel memory participation, RDMA requests are sent from the application running in user space to the local network card, and then sent to the remote network card through the network. Therefore, RDMA transmission does not require the participation of the operating system and will not increase the system load. FIG. 1 is a system frame diagram of a data transmission method according to an embodiment of the present application. The figure shows a transmission scenario of RDMA transmission: the first application fetches data from the memory to generate an RDMA message, and sends the RDMA message to the It is sent to the local network card through the cache, and then transmitted to the remote network card through the network. The remote network card caches the received RDMA message, and the second application program takes out the data from the cache and writes it into the memory. Likewise, the process of reading the data in the memory of the second application by the first application is similar to the description of the above-mentioned writing process, and details are not repeated here. In addition, the network card includes an RDMA-capable network interface card (RDMA network interface card, RNIC) or a host channel adapter (host channel adapter, HCA).

RDMA transfer is to transfer data from one system to the memory of another system through a send queue (SQ), wherein the data in the SQ only includes the data of a virtual machine (VM), when a service When data of multiple VMs is involved, the data of multiple VMs cannot be transferred to the memory of another system at one time, which affects the transfer efficiency.

In order to solve the above problem, an embodiment of the present application provides a data transmission system, the structure of which can be seen in FIG. 2 , the data transmission system includes a first network device 21 and a second network device 22 , and the first network device 21 is arranged on the first network device 21 . On the host 211, the second network device 22 is set on the second host 221, and N virtual machines 2111 run on the first host 211.

The first network device 21 is used to obtain N pieces of data, and the N pieces of data come from N pieces of VM 2111. According to the remote direct memory access RDMA protocol, the N pieces of data and the write addresses of the N pieces of data are encapsulated into a message, and the report The message is sent to the second network device 22, where N is an integer greater than 1.

The second network device 22 is configured to receive the message, decapsulate the message to obtain N data and N data write addresses, and store N data on the second host 221 according to the N data write addresses data.

Specifically, N data of N VMs 2111 running on the first host 211 need to be transferred to the second host 221 through RDMA, wherein the N VMs 2111 are VMs that can normally extract data, and the first host 211 A network device 21 may encapsulate the above-mentioned N pieces of data and the write addresses of the N pieces of data according to the RDMA protocol, and then send the packet generated by encapsulation to the second network device 22 on the second host 221 . The second network device 22 may decapsulate the packet, extract the N pieces of data and the write addresses of the N pieces of data, and store the N pieces of data in the positions indicated by the write addresses. The first network device 21 can directly send data of multiple virtual machines to the second network device 22, which improves transmission efficiency.

Optionally, the first network device 21 is used to acquire the identifiers and memory addresses of the N VMs 2111; and acquire N pieces of data according to the identifiers and memory addresses of the N VMs.

Specifically, the first network device 21 can directly acquire N pieces of data according to the VM identifiers and VM memory addresses of the N pieces of data that it wants to RDMA to transmit to the second network device 22, and the write address received by the second network device 22 includes the VM identifier. and the VM memory address, the second network device 22 can directly use the write address to store N pieces of data without going through a chip logical address (chip logical address, CLA), which can reduce the processing flow.

Optionally, there is an abnormal VM running on the first host 211, and data cannot be obtained according to the identification and memory address of the abnormal VM; the first network device 21 is used for the identification and memory address of the abnormal VM and some VMs in the N VMs 2111. The identifier and memory address are obtained, and M pieces of data are obtained, where M is a positive integer less than N; the M pieces of data are encapsulated into an exception message according to the RDMA protocol.

Specifically, if there is an abnormal VM in the N VMs 2111, for example, the VM fails, shuts down or restarts, etc., the first network device 21 cannot obtain data according to the identification and memory address of the abnormal VM, that is, according to the identification and memory address of the abnormal VM and The abnormal packets generated by the identification and memory address encapsulation of some VMs in the N VMs cannot be sent to the second network device 22 through a queue pair (queue pair, QP) message.

Optionally, the first network device 21 is configured to generate a packet sequence, where the packet sequence includes an abnormal packet and at least one packet.

Specifically, when an abnormal VM exists in the N VMs, the first network device 21 needs to generate a message sequence from the abnormal message and at least one of the foregoing messages, so that the first network device 21 can generate a message sequence according to the message sequence number of the message sequence. The above-mentioned abnormal message and the above-mentioned message are sent in sequence, so as to improve the implementability of the solution.

Optionally, the first network device 21 is configured to modify the packet sequence, where modifying the packet sequence includes deleting abnormal packets and adding padding packets to the packet sequence.

Specifically, there is an abnormal message in the message sequence, and the abnormal message cannot be sent to the second network device 22, causing the second network device 22 to continuously request message retransmission, and multiple retransmission failures will cause the first network device 21 and The link between the second network devices 22 is disconnected, and the RDMA transmission fails. The present application can delete abnormal packets in the packet sequence, and then supplement the packet sequence with padding packets, which are invalid packets, so as to keep the data length unchanged and avoid retransmission due to inconsistent data lengths.

Optionally, the second network device 22 is configured to receive the modified packet sequence, determine the padding packet in the modified packet sequence, and delete the padding packet.

Specifically, after receiving the above-mentioned modified packet sequence, the second network device 22 can determine the invalid packet by checking, that is, determine the padding packet in the modified packet sequence, and then delete the padding packet. message to improve the reliability of message transmission.

In the technical solutions of the embodiments of the present application, the first network device may directly send data of multiple VMs to the second network device, thereby improving transmission efficiency.

The VMs running on the first host where the first network device is set may all be VMs that can normally extract data, and may also include abnormal VMs that cannot extract data, which will be described separately below.

1. The VMs running on the first host are all VMs that can extract data normally.

Please refer to an embodiment of the data transmission method shown in FIG. 3:

301. The first network device acquires N pieces of data.

In this embodiment, N (N is an integer greater than 1) pieces of data are data that the first network device needs to transmit and write to the second network device through RDMA, and the N pieces of data store the N data of the first host of the first network device. Among the VMs, the first network device may obtain the N pieces of data from the N VMs according to the identifiers and memory addresses of the N VMs where the N pieces of data are located. In this embodiment, the N VMs are all VMs that can normally extract data.

302. The first network device encapsulates the N pieces of data and the write addresses of the N pieces of data into one packet according to the RDMA protocol.

In this embodiment, the above-mentioned encapsulated message may be an RDMA write message, and the RDMA write message includes the following three types: RDMA write First, RDMA write Middle, and RDMA write Last, and the RDMA extended transport header (RDMAextended transport header, RETH) is a field in the RDMA message format, which carries the destination address of the message data, and the base transport header (BTH) is a field in the RDMA message format, including PSN. Therefore, the RDMA write First message includes the packet sequence number (PSN) and the write address, while the BTH field exists in the RDMA write Middle and RDMA write Last messages.

In this embodiment, the first network device may complete the creation of the QP for RDMA in advance, which may be created either on a virtual machine manager (Hypervisor) of the host, or on another central processing unit (central processing unit, CPU) or On the device, at the same time, the RDMA hardware device (RNIC or HCA or other hardware device with RDMA capability) itself can directly access the memory space of the VM. When the RDMA transmission hardware is limited to the CLA, the first network device creates a CLA for RDMA hardware access in advance. )/R_Key (used for reading the memory of the remote device, L_key and R_Key can also use the same value) The corresponding CLA is unique. Then perform RDMA memory registration, map the memory of the VM to be read into the CLA, and all RDMA data operation commands can be performed based on the CLA. The registered registry includes L_key, R_key, CLA start address, VM identifier, VM memory address and length, wherein there are multiple VM identifiers, and each VM identifier in the multiple VM identifiers corresponds to a VM memory address respectively and length. The first network device may query the registry based on the CLA and length of the target data to obtain the VM identifier, VM memory address and length corresponding to the CLA and the length, and then extract the target data from the position corresponding to the VM identifier, the VM memory address and the length. to encapsulate. In this embodiment, the CLA address of the target data can be referred to as shown in FIG. 4 . One CLA address includes addresses of data of multiple VMs. Exemplarily, VM1 includes data sg11 , sg12 and sg13 , and VM2 includes data sg21 , sg22 and sg23 , VM3 includes data sg31, sg32 and sg33, and the address of a CLA may include addresses adr11, adr21 and adr31, where address adr11 may indicate data sg11 in VM1, address adr21 may indicate data sg21 in VM2, and address adr31 may indicate VM3 Data in sg31.

Optionally, the registry may only include L_key, R_key, VM identifier, VM memory address and length, and the first network device may directly use the VM identifier, VM memory address, and length of the target data to extract the target data from the corresponding VM. to encapsulate. In this embodiment, security verification needs to be performed through the registry for the VM identifier, the VM memory address and the length, to avoid when the indicated read-write position exceeds the memory area indicated by the VM identifier, the VM memory address and the length in the registry, The case of reading and writing VMs in other memory areas other than the registry.

The VM identifier in this embodiment may be a physical function (Physical Function, PF)/virtual function (Virtual Function, VF) based on a single-root I/O virtualization (SR-IOV) device. Identification (ID), based on Scalable I/O virtualization (Scalable-IOV) devices can be assignable device interfaces (ADI), or other identification processes that can identify different VMs or address domains Address space identifier (Process Address Space identifier, PASID).

Optionally, the storage may be the memory space of the application program in the second network device, and the target data may be data stored in the memory space of the application program in the first network device.

The maximum transmission unit (maximum transmission unit, MTU) is the maximum data packet size that can be transmitted in each RDMA transmission in the RDMA protocol, and the number of the above-mentioned first packets can be determined according to the data size of the target data and the MTU.

Optionally, in this embodiment, when the first network device transmits data to the second network device through RDMA transmission, the first network device may directly write data to the second network device RDMA, or the second network device may write data to the first network device. The network device RDMA reads data. When the second network device reads data from the first network device RDMA, before step 301 , the second network device also needs to send a data read request to the first network device to trigger step 301 .

303. The first network device sends the packet to the second network device.

In this embodiment, when the above-mentioned packet is successfully encapsulated, the first network device may directly send the packet to the second network device according to the PSN sequence of the packet. Correspondingly, the second network device receives the message.

304. The second network device decapsulates the packet to obtain N pieces of data and N pieces of data write addresses.

After receiving the above-mentioned packet, the second network device may directly decapsulate the packet, for example, disassemble the protocol packet, process the information in the packet header, and extract the N data in the payload and the write address of the N data.

305. The second network device stores N pieces of data on the second host according to the write addresses of the N pieces of data.

After obtaining the above write address, the second network device may store N pieces of data in the VM indicated by the write address. For example, the write address is the CLA address and length, and the second network device may query the registration according to the CLA address and length. The table is stored by matching the corresponding VM ID, VM memory address and length, or the write address is the VM ID, VM memory address and length, and the second network device directly stores N pieces of data according to the VM ID, VM memory address and length .

In this embodiment of the present application, the first network device encapsulates N data of N VMs running on the first host into a packet, and sends the packet to the second network device. The first network device can directly The data of each VM is sent to the second network device without performing multiple RDMA transmissions, which can improve transmission efficiency.

2. The VMs running on the first host include abnormal VMs that cannot extract data.

Please refer to FIG. 5. As shown in FIG. 5, another embodiment of the data transmission method provided by the embodiment of the present application includes:

501. The first network device acquires N pieces of data.

502. The first network device encapsulates the N pieces of data and the write addresses of the N pieces of data into one packet according to the RDMA protocol.

For steps 501 and 502, reference may be made to the relevant descriptions of steps 301 and 302 in the data transmission method shown in FIG. 3, and details are not repeated here.

503. The first network device acquires M pieces of data according to the identifier and memory address of the abnormal VM and the identifiers and memory addresses of some VMs in the N VMs.

In this embodiment, the abnormal VM is a VM from which data cannot be extracted normally. The abnormal VM may be a VM failure, shutdown or restart, etc. The data transmitted by the first network device through RDMA involves the data in the abnormal VM, and the RDMA command initiated by the first network device For example, if direct memory access (DMA) fails, data cannot be obtained according to the ID and memory address of the abnormal VM, and M data can only be obtained according to the ID and memory address of some VMs in N VMs.

504. The first network device encapsulates the M pieces of data into an exception packet according to the RDMA protocol.

When the first network device encapsulates the data of the abnormal VM and the data of some VMs in the N VMs according to the RDMA protocol, since the data of the abnormal VM cannot be obtained, the encapsulated message is the abnormal message, and the first network device can directly return After the command is completed, configure the indication information for the abnormal message. The indication information can indicate that the abnormal message is an error message. For example, the indication can be configured in the completion queue element (complete queue element, CQE) of the completion queue (complete queue, CQ). information.

In this embodiment, the sequence of steps 501 to 502 and steps 503 to 504 is not limited.

505. The first network device generates a packet sequence, where the packet sequence includes an abnormal packet and at least one packet.

When the first network device RDMA transmits data to the second network device, the data is sequentially transmitted according to the PSNs of the transmission packets. In this embodiment, the first device sorts the normally encapsulated packets and the abnormal packets to form a packet sequence, which is a normal packet. Configure PSN for encapsulated packets and abnormal packets.

506. The first network device modifies the packet sequence, where the modification of the packet sequence includes deleting an abnormal packet and adding a padding packet to the packet sequence.

In this embodiment, the first network device may process the factors in the packet sequence that affect the related state received by the second network device. Exemplarily, the first network device sends the packet sequence to the second device through a QP message, and the first network device sends the packet sequence to the second device through a QP message. A network device can modify the PSN in the message sequence, skip the error message and send the next message, so as to avoid the second network device finding that the PSN is discontinuous and thinking that the middle message is lost, so it tries to retransmit repeatedly. After several unsuccessful attempts, it is considered that the QP is faulty or the QP is disconnected. Exemplarily, as shown in the schematic diagram of the combined strips shown in FIG. 6, with VM0, VM1, VM2, VM3, VM4 (not shown in the figure) and VM5 (not shown in the figure), VM0 is represented by a thin solid line box. The data D000, D001 and D020 of VM1, the data D110 and D111 of VM1, the data D210, D220 and D221 of VM2, the data D300, D301 and D310 of VM3, the data of VM4 and VM5 are represented by solid boxes. That is, the data strips from VM0 to VM5 can be combined to obtain MSG0 to MSG5 messages as shown in FIG. 6 , and the first network device can send MSG0 to the second network device according to the sequence in the send queue (SQ) in the figure. Message to MSG5.

When VM2 fails, there is a problem with the data of VM2, so that MSG2 and MSG3 cannot directly obtain data from VM2, which causes MSG2 and MSG3 to fail to send, and the connection will be broken after multiple retransmissions, resulting in other normal messages. If it cannot be sent, in this embodiment of the present application, the subsequent data of MSG4 and MSG5 can be moved forward to replace the data of MSG2 and MSG3, as shown in Figure 7, that is, the data of MSG4 is now numbered as MSG2', and the data of MSG5 is currently numbered. is MSG3'. Then, the padding message is supplemented as the data of the new MSG4 and MSG5 to obtain a new SQ' to ensure that the data length is correct, so that the RDMA message can be sent to the second network device normally. Specifically, the padding data in the padding packet may be other VM data or blank data.

The above-mentioned way of supplementing the message needs to reorder the PSN of the message. Optionally, the embodiment of the present application may also be a way of supplementing the message that does not need to reorder the PSN of the message, as shown in FIG. 8 , The first network device directly replaces the target VM data with data of other VMs or blank data to obtain a new SQ'.

507. The first network device sends the modified packet sequence to the second network device.

The first network device sequentially sends the message and the padding message to the second network device according to the PSN of the modified message sequence. Correspondingly, the second network device sequentially receives the message and the padding message in the modified message sequence according to the PSN sequence. Fill the message.

508. The second network device determines a padding packet in the modified packet sequence, and deletes the padding packet.

After the second network device receives the modified packet sequence, it can pass the data integrity field (DIF) check. For example, the data in the padding packet comes from other VMs that do not belong to the N VMs, then It can be verified as an invalid packet that does not conform to the VM type, or the padding packet includes blank data, which can be directly verified as an invalid packet. The upper-layer data processing program will remove the invalid packet and save only the modified packet. the above message in the sequence.

509. The second network device decapsulates the packet sequence after deleting the padding packet, so as to obtain N data and N data write addresses.

In this embodiment, the packet sequence after deleting the padding packet only includes the packet in step 502, and the packet can be decapsulated, such as disassembling the protocol packet, processing the information in the packet header, and taking out the N in the payload. write address of data and N data.

510. The second network device stores N pieces of data on the second host according to the write addresses of the N pieces of data.

After obtaining the above write address, the second network device may store N pieces of data in the VM indicated by the corresponding write address, for example, may query the registry to match the corresponding VM identifier and VM memory according to the CLA address and length. The address and length are stored, or directly stored according to the VM ID, VM memory address and length.

In this embodiment of the present application, the first network device encapsulates N data of N VMs running on the first host into a packet, and sends the packet to the second network device. The first network device can directly When the data of each VM is sent to the second network device, it is not necessary to perform multiple RDMA transmissions, which can improve transmission efficiency.

Further, the first network device modifies the message sequence including the abnormal message, deletes the abnormal message and adds a padding message, and sends the modified message sequence to the second network device to prevent the second network device from discovering the sequence. If the number is discontinuous, it is considered that the message in the middle is lost, which leads to repeated attempts to retransmit. After several unsuccessful attempts, it is considered that the QP is faulty or the QP chain is disconnected. In this way, the transmission efficiency can be improved.

The data transmission method is described above, and the following describes the network device according to the embodiments of the present application with reference to the accompanying drawings.

FIG. 9 is a schematic diagram of an embodiment of a network device 90 in an embodiment of the present application.

As shown in FIG. 9 , an embodiment of the present application provides a network device, where the network device includes:

The obtaining unit 901 is used to obtain N pieces of data, the network device is set on the first host, and N virtual machines VMs are running on the first host, and the N pieces of data come from the N pieces of VM, where N is an integer greater than 1;

an encapsulation unit 902, configured to encapsulate the N data and the write addresses of the N data into a message according to the remote direct memory access RDMA protocol;

The sending unit 903 is configured to send the message to the second network device.

Optionally, the obtaining unit 901 is specifically configured to: obtain identifiers and memory addresses of N VMs; and obtain N pieces of data according to the identifiers and memory addresses of N VMs.

Optionally, the obtaining unit 901 is further configured to: obtain M pieces of data according to the identifier and memory address of the abnormal VM and the identifiers and memory addresses of some VMs in the N VMs, the abnormal VM runs on the first host, and the abnormal VM The identity and memory address of the VM cannot obtain data, where M is a positive integer less than N;

The encapsulation unit 902 is further configured to encapsulate the M pieces of data into an exception packet according to the RDMA protocol.

Optionally, the network device 90 further includes a generating unit 904, and the generating unit 904 is specifically configured to: generate a packet sequence, where the packet sequence includes an abnormal packet and at least one packet.

Optionally, the network device 90 further includes a modification unit 905, and the modification unit 905 is specifically configured to: modify the packet sequence, wherein modifying the packet sequence includes deleting abnormal packets and adding padding packets to the packet sequence.

In this embodiment, the network device may perform the operations performed by the first network device in the foregoing embodiments shown in FIG. 3 and FIG. 5 , and details are not repeated here.

FIG. 10 is a schematic diagram of another embodiment of the network device 100 in the embodiment of the present application.

As shown in FIG. 10 , an embodiment of the present application provides a network device, where the network device includes:

The receiving unit 1001 is configured to receive a message from a first network device, the first network device is set on a first host, the network device is set on a second host, and N virtual machines VM are running on the first host, and the message A message generated by encapsulating N data and N data write addresses according to the remote direct memory access RDMA protocol, where N data comes from N VMs, where N is an integer greater than 1;

A decapsulation unit 1002, configured to decapsulate the message to obtain N data and N data write addresses;

The storage unit 1003 is configured to store N pieces of data on the second host according to the write addresses of the N pieces of data.

Optionally, the receiving unit 1001 is further configured to: receive a modified message sequence, where the modified message sequence includes a padding message;

The network device 100 further includes a deletion unit 1004, and the deletion unit 1004 is specifically configured to: determine the padding message in the modified message sequence, and delete the padding message.

In this embodiment, the network device may perform the operations performed by the second network device in the foregoing embodiments shown in FIG. 3 and FIG. 5 , and details are not described herein again.

As shown in FIG. 11 , a schematic diagram of a possible logical structure of the network device 110 provided by the embodiment of the present application is shown. The network device 110 includes: a processor 1101 , a communication interface 1102 , a storage system 1103 and a bus 1104 . The processor 1101 , the communication interface 1102 , and the storage system 1103 are connected to each other through a bus 1104 . In this embodiment of the present application, the processor 1101 is configured to control and manage the actions of the network device 110. For example, the processor 1101 is configured to execute the steps performed by the first network device in the method embodiments of FIG. 3 and FIG. 5 . The communication interface 1102 is used to support the communication of the network device 110 . The storage system 1103 is used to store program codes and data of the network device 110 .

The processor 1101 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logical blocks, modules and circuits described in connection with this disclosure. The processor 1101 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and the like. The bus 1104 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. For ease of presentation, only one thick line is used in FIG. 11, but it does not mean that there is only one bus or one type of bus.

The sending unit 903 in the network device 90 is equivalent to the communication interface 1102 in the network device 110 , and the acquiring unit 901 , the encapsulating unit 902 , the generating unit 904 and the modifying unit 905 in the network device 90 are equivalent to the processor 1101 in the network device 110 .

The network device 110 in this embodiment may correspond to the first network device in the foregoing method embodiments in FIG. 3 and FIG. 5 , and the communication interface 1102 in the network device 110 may implement the first network device in the foregoing method embodiments in FIG. 3 and FIG. 5 . For the sake of brevity, the functions possessed by the network device and/or the various steps implemented are not repeated here.

As shown in FIG. 12 , a schematic diagram of a possible logical structure of the network device 120 provided by the embodiment of the present application is shown. The network device 120 includes: a processor 1201 , a communication interface 1202 , a storage system 1203 and a bus 1204 . The processor 1201 , the communication interface 1202 , and the storage system 1203 are connected to each other through a bus 1204 . In this embodiment of the present application, the processor 1201 is configured to control and manage the actions of the network device 120. For example, the processor 1201 is configured to execute the steps performed by the second network device in the method embodiments of FIG. 3 and FIG. 5 . The communication interface 1202 is used to support the communication of the network device 120 . The storage system 1203 is used to store program codes and data of the network device 120 .

The processor 1201 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logical blocks, modules and circuits described in connection with this disclosure. The processor 1201 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and the like. The bus 1204 may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one thick line is shown in FIG. 12, but it does not mean that there is only one bus or one type of bus.

The receiving unit 1001 in the network device 100 is equivalent to the communication interface 1202 in the network device 120 , and the decapsulating unit 1002 , the storage unit 1003 and the deleting unit 1004 in the network device 100 may be equivalent to the processor 1201 .

The network device 120 in this embodiment may correspond to the second network device in the foregoing method embodiments in FIGS. 3 and 5 , and the processor 1201 and the communication interface 1202 in the network device 120 may implement the foregoing method embodiments in FIGS. 3 and 5 . For the sake of brevity, the functions and/or various steps performed by the second network device in , are not repeated here.

In another embodiment of the present application, a computer-readable storage medium is also provided, where computer-executable instructions are stored in the computer-readable storage medium. When the processor of the device executes the computer-executable instructions, the device executes the above-mentioned FIG. 3 and Steps of the data transmission method performed by the first network device in FIG. 5 .

In another embodiment of the present application, a computer-readable storage medium is also provided, where computer-executable instructions are stored in the computer-readable storage medium. When the processor of the device executes the computer-executable instructions, the device executes the above-mentioned FIG. 3 and Steps of the data transmission method performed by the second network device in FIG. 5 .

In another embodiment of the present application, a computer program product is also provided, the computer program product includes computer-executable instructions, and the computer-executable instructions are stored in a computer-readable storage medium; when a processor of a device executes the computer-executable instructions , the device executes the steps of the data transmission method executed by the first network device in the above-mentioned FIG. 3 and FIG. 5 .

In another embodiment of the present application, a computer program product is also provided, the computer program product includes computer-executable instructions, and the computer-executable instructions are stored in a computer-readable storage medium; when a processor of a device executes the computer-executable instructions , the device executes the steps of the data transmission method executed by the second network device in the above-mentioned FIG. 3 and FIG. 5 .

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above may refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, and the computer software products are stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), magnetic disk or optical disk and other media that can store program codes .

Claims

A data transmission system, characterized in that the data transmission system includes a first network device and a second network device, the first network device is set on a first host, and the second network device is set on a second host On the first host, there are N virtual machines running on the first host;

The first network device is configured to acquire N pieces of data, the N pieces of data are from the N pieces of VM, and write the N pieces of data and the write addresses of the N pieces of data according to the remote direct memory access (RDMA) protocol Encapsulate into a packet, and send the packet to the second network device, where N is an integer greater than 1;

The second network device is configured to receive the packet, and decapsulate the packet to obtain the N pieces of data and the write addresses of the N pieces of data, according to the write address of the N pieces of data The N pieces of data are stored on the second host at the incoming address.
The data transmission system according to claim 1, wherein,

The first network device is configured to acquire the identifiers and memory addresses of the N VMs; and acquire the N pieces of data according to the identifiers and memory addresses of the N VMs.
The data transmission system according to claim 2, wherein an abnormal VM runs on the first host, and data cannot be obtained according to the identifier and memory address of the abnormal VM;

The first network device is configured to obtain M pieces of data according to the identifier and memory address of the abnormal VM and the identifiers and memory addresses of some VMs in the N VMs, where M is a positive integer less than N; The RDMA protocol encapsulates the M pieces of data into an exception message.
The data transmission system according to claim 3, wherein,

The first network device is configured to generate a packet sequence, where the packet sequence includes the abnormal packet and at least one of the packets.
The data transmission system according to claim 4, wherein,

The first network device is configured to modify the message sequence, wherein modifying the message sequence includes deleting the abnormal message and adding a padding message to the message sequence.
The data transmission system according to claim 5, wherein,

The second network device is configured to receive the modified packet sequence, determine the padding packet in the modified packet sequence, and delete the padding packet.
A data transmission method, comprising:

The first network device obtains N pieces of data, the first network device is set on a first host, and N virtual machines VMs run on the first host, and the N pieces of data come from the N VMs, wherein, N is an integer greater than 1;

The first network device encapsulates the N data and the write address of the N data into a message according to the remote direct memory access RDMA protocol;

The first network device sends the message to the second network device.
The data transmission method according to claim 7, wherein the acquisition of N pieces of data by the first network device comprises:

The first network device obtains the identifiers and memory addresses of the N VMs;

The first network device acquires the N pieces of data according to the identifiers and memory addresses of the N VMs.
The data transmission method according to claim 7, wherein an abnormal VM runs on the first host, and the abnormal VM cannot acquire data according to the identifier and memory address of the abnormal VM, and the method further comprises:

The first network device obtains M pieces of data according to the identifier and memory address of the abnormal VM and the identifiers and memory addresses of some VMs in the N VMs, where M is a positive integer less than N;

The first network device encapsulates the M pieces of data into an exception packet according to the RDMA protocol.
The data transmission method according to claim 9, wherein the method further comprises:

The first network device generates a message sequence, and the message sequence includes the abnormal message and at least one of the messages.
The data transmission method according to claim 10, wherein the method further comprises:

The first network device modifies the message sequence, wherein modifying the message sequence includes deleting the abnormal message and adding a padding message to the message sequence.
A data transmission method, comprising:

The second network device receives the message from the first network device, the first network device is set on the first host, the second network device is set on the second host, and N running on the first host The virtual machine VM, the message is a message generated by encapsulating N pieces of data and the write addresses of the N pieces of data according to the remote direct memory access RDMA protocol, and the N pieces of data are from the N VMs, wherein , N is an integer greater than 1;

The second network device decapsulates the message to obtain the N pieces of data and the write addresses of the N pieces of data;

The second network device stores the N pieces of data on the second host according to the write addresses of the N pieces of data.
The data transmission method according to claim 12, wherein the method further comprises:

receiving, by the second network device, a modified message sequence, where the modified message sequence includes a padding message;

The second network device determines the padding packet in the modified packet sequence, and deletes the padding packet.
A network device, characterized in that it includes:

an acquisition unit, configured to acquire N pieces of data, the network device is set on a first host, and N virtual machines VMs run on the first host, and the N pieces of data come from the N VMs, where N is an integer greater than 1;

an encapsulation unit, configured to encapsulate the N pieces of data and the write addresses of the N pieces of data into a message according to the remote direct memory access RDMA protocol;

A sending unit, configured to send the message to the second network device.
The network device according to claim 14, wherein the obtaining unit is specifically configured to:

Obtain the identifiers and memory addresses of the N VMs;

The N pieces of data are acquired according to the identifiers and memory addresses of the N VMs.
The network device according to claim 14, wherein the obtaining unit is further configured to:

Acquire M pieces of data according to the identifier and memory address of the abnormal VM and the identifiers and memory addresses of some VMs in the N VMs, the abnormal VM runs on the first host, and the abnormal VM is based on the abnormal VM The identifier and memory address of , cannot obtain data, where M is a positive integer less than N;

The packaging unit is also used for:

The M pieces of data are encapsulated into an exception message according to the RDMA protocol.
The network device according to claim 16, wherein the network device further comprises a generating unit, and the generating unit is specifically configured to:

A message sequence is generated, the message sequence including the abnormal message and at least one of the messages.
The network device according to claim 17, wherein the network device further comprises a modification unit, and the modification unit is specifically configured to:

Modifying the message sequence, wherein modifying the message sequence includes deleting the abnormal message, and adding a padding message to the message sequence.
A network device, characterized in that it includes:

a receiving unit, configured to receive a message from a first network device, the first network device is set on a first host, the network device is set on a second host, and N virtual machines run on the first host machine VM, the message is a message generated by encapsulating N pieces of data and the write addresses of the N pieces of data according to the remote direct memory access RDMA protocol, and the N pieces of data are from the N VMs, wherein, N is an integer greater than 1;

a decapsulation unit, configured to decapsulate the message to obtain the N pieces of data and the write addresses of the N pieces of data;

A storage unit, configured to store the N pieces of data on the second host according to the write addresses of the N pieces of data.
The network device according to claim 19, wherein the receiving unit is further configured to:

receiving a modified sequence of messages, the modified sequence of messages including padding messages;

The network device further includes a deletion unit, and the deletion unit is specifically used for:

The padding message in the modified message sequence is determined, and the padding message is deleted.
A network device, comprising: a processor and a memory,

The processor is configured to execute instructions stored in the memory, so that the network device performs the method of any one of claims 7 to 11.
A network device, comprising: a processor and a memory,

The processor is configured to execute the instructions stored in the memory, so that the network device executes the method of any one of claims 12 to 13.