CN116226027B

CN116226027B - Data transmission system, method, device, communication equipment and storage medium

Info

Publication number: CN116226027B
Application number: CN202310488688.1A
Authority: CN
Inventors: 刘钧锴; 王彦伟; 阚宏伟
Original assignee: Suzhou Inspur Intelligent Technology Co Ltd
Current assignee: Suzhou Inspur Intelligent Technology Co Ltd
Priority date: 2023-05-04
Filing date: 2023-05-04
Publication date: 2023-08-15
Anticipated expiration: 2043-05-04
Also published as: CN116226027A

Abstract

The embodiment of the application provides a data transmission system, a method, a device, communication equipment and a storage medium, which are applied to the technical field of communication and comprise the following steps: a host channel adapter module, a first target channel adapter module, and a second target channel adapter module; the host channel adapter module is in communication connection with the first target channel adapter module and the second target channel adapter module; the first target channel adapter module is in communication connection with the second target channel adapter module; the host channel adapter module is used for sending a remote direct data access operation data packet to the first target channel adapter module, and remotely controlling the first target channel adapter module to transmit data to the second target channel adapter module according to the remote direct data access operation data packet. According to the embodiment of the application, the host channel adapter module remotely controls the data transmission among the target channel adapter modules, so that the direct data transmission among the target channel adapters is realized without adding equipment.

Description

Data transmission system, method, device, communication equipment and storage medium

Technical Field

The present application relates to the field of communications technologies, and in particular, to a data transmission system, a method, an apparatus, a communication device, and a storage medium.

Background

With the rapid development of cloud computing, artificial intelligence, big data and other technologies, the amount of data to be processed increases in geometric progression, and the requirements on network delay and bandwidth of data transmission are also increasing, so remote direct data access (Remote Direct Memory Access, RDMA) is becoming the mainstream data transmission protocol of data centers.

Based on the data transmission of the remote direct data access protocol, a Queue Pair (QP) connection needs to be established between nodes before transmission, and each node may have multiple queues and module resources, so that the QP module number used by the node needs to be notified when the connection is established. Remote direct data access divides computing nodes into two types, host channel adapter (Host Channel Adapter, host channel adapter module) and target channel adapter (Target Channel Adapter, TCA), wherein the host channel adapter module is a server with an operating system, etc., that can initiate remote direct data access operations (send data to the far end or read far end data); the TCA is a network hard disk without a host, an accelerator card, etc., and can only respond to remote direct data access operations.

In the prior art, data transmission is performed between two target channel adapter modules, namely, data is read out from one target channel adapter module through a host channel adapter module and then is sent to the other target channel adapter module, so that delay is larger in the data transmission process, and the network utilization rate is lower.

Disclosure of Invention

The embodiment of the application aims to provide a data transmission system, a method, a device, communication equipment and a storage medium, which are used for solving the technical problems of larger delay and lower network utilization rate in the data transmission process through a host channel adapter module in the prior art. The specific technical scheme is as follows:

in a first aspect of the present application, there is provided a data transmission system, comprising: a host channel adapter module, a first target channel adapter module, and a second target channel adapter module;

the host channel adapter module is in communication connection with the first target channel adapter module and the second target channel adapter module;

the first target channel adapter module is communicatively coupled to the second target channel adapter module;

The host channel adapter module is configured to send a remote direct data access operation data packet to the first target channel adapter module, and remotely control the first target channel adapter module to perform data transmission to the second target channel adapter module according to the remote direct data access operation data packet.

Optionally, the first target channel adapter module and the second target channel adapter module communicatively connect comprises: the host channel adapter module sends a first remote direct data access data packet and a second remote direct data access data packet to the first target channel adapter module and the second target channel adapter module respectively, so that the first target channel adapter module and the second target channel adapter module establish connection according to the first remote direct data access data packet and the second remote direct data access data packet.

Optionally, the first target channel adapter module includes a remote work queue event engine module, a remote completion queue event engine module, a first queue pair module, and a second queue pair module;

the remote work queue event engine module is used for forming work queue event data recorded by an internal register in the first queue pair module into a work queue event, sending the work queue event to the second queue pair module and triggering remote direct data access operation;

The remote completion queue event engine module is configured to send a work queue event to the first queue pair module by using completion queue event information returned by the second queue pair module and a physical address corresponding to the host channel adapter module recorded by an internal register in the first queue pair module, so that the first queue pair module returns the completion queue event information to the host channel adapter module.

Optionally, the first target channel adapter module further includes a first parsing module;

the first parsing module is configured to receive a remote direct data access data packet sent by the host channel adapter module, parse the remote direct data access data packet, and convert the parsed remote direct data access data packet into a data bus write operation.

Optionally, the second target channel adapter module includes a third queue pair module and a fourth queue pair module;

the third queue pair module is in communication connection with a sixth queue pair module in the host channel adapter module;

the fourth queue pair module is communicatively coupled to a second queue pair module in the first target channel adapter module.

Optionally, the second target channel adapter module further includes a second parsing module;

the second analyzing module is configured to receive the remote direct data access data packet sent by the host channel adapter module, analyze the remote direct data access data packet, and identify the first target channel adapter module according to the analyzed remote direct data access data packet.

Optionally, the host channel adapter module includes a fifth queue pair module and a sixth queue pair module;

the fifth queue pair module is in communication connection with a third queue pair module in the second target channel adapter module;

the sixth queue pair module is communicatively coupled to the first queue pair module in the second target channel adapter module.

Optionally, the host channel adapter module is further configured to send a first remote direct data access packet to a first parsing module in the first target channel adapter module, where the first remote direct data access packet includes a physical address corresponding to the second target channel adapter module, and send a second remote direct data access packet to a second parsing module in the second target channel adapter module, where the second remote direct data access packet includes a physical address corresponding to the first target channel adapter module, so that the first target channel adapter module and the second target channel adapter module establish a connection according to the remote direct data access packet.

Optionally, a remote completion queue event register and a remote work queue event register are preset in a host memory corresponding to the host channel adapter module;

the remote completion queue event register is used for storing completion queue event information returned by the target channel adapter;

the remote work queue event register is used for storing remote direct data access operation information according to a preset remote write register format.

Optionally, the remote direct data access operation information includes work queue event information and a physical address of a remote completion queue event register;

the work queue event corresponds to an internal register of the remote work queue event engine module in the first target channel adapter module;

the physical address of the remote completion queue event register corresponds to an internal register of the remote work queue event engine module in the first target channel adapter module.

In still another aspect of the present application, there is further provided a data transmission method applied to a first target channel adapter module, the data transmission method including:

receiving a remote direct data access operation data packet sent by a host channel adapter module under the condition that the first target channel adapter module and the second target channel adapter module are detected to establish connection, wherein the remote direct data access operation data packet is used for remotely controlling the first target channel adapter module to transmit data to the second target channel adapter module;

And sending the target storage data corresponding to the first target channel adapter module to a second target channel adapter module according to the remote direct data access operation data packet so as to enable the second target channel adapter module to acquire the target storage data.

Optionally, the remote direct data access operation data packet includes physical addresses of remote work queue event register data and remote completion queue event register, and the sending, according to the remote direct data access operation data packet, target storage data corresponding to the first target channel adapter module to the second target channel adapter module, so that the second target channel adapter module obtains the target storage data includes:

analyzing the remote direct data access operation data packet, writing a first work queue event in the remote work queue event register data into an internal register of a remote work queue event engine module according to the analyzed remote direct data access operation data packet, and writing a physical address of the remote completion queue event register into the internal register of the remote completion queue event engine module;

The remote work queue event engine module sending the first work queue event to a second queue pair module;

the second queue pair module obtains target storage data corresponding to the first target channel adapter module according to the first work queue event, generates a target remote direct data access operation data packet according to the target storage data, and sends the target remote direct data access operation data packet to the second target channel adapter module so that the second target channel adapter module obtains the target remote direct data access operation data packet.

Optionally, the work queue event includes a work queue event identifier, a storage physical address of a first target channel adapter module, a destination storage address of a second target channel adapter module, and a data byte length, the second queue pair module obtains target storage data corresponding to the first target channel adapter module according to the work queue event, generates a target remote direct data access operation data packet according to the target storage data, and sends the target remote direct data access operation data packet to the second target channel adapter module, including:

The second queue pair module obtains target storage data corresponding to the first target channel adapter module according to the storage physical address of the first target channel adapter module, generates a target remote direct data access operation data packet according to the target storage data, and sends the target remote direct data access operation data packet to the second target channel adapter module according to the target storage address of the second target channel adapter module.

Optionally, after the step of the second queue pair module obtaining the target storage data corresponding to the first target channel adapter module according to the first work queue event, generating a target remote direct data access operation data packet according to the target storage data, and sending the target remote direct data access operation data packet to a second target channel adapter module, the method includes:

and receiving a data transmission completion signal returned by the second target channel adapter module, and determining that the second target channel adapter module receives the target remote direct data access operation data packet according to the data transmission completion signal.

Optionally, the receiving a data transmission completion signal returned by the second target channel adapter module, and determining, according to the data transmission completion signal, that the second target channel adapter module receives the target remote direct data access operation data packet includes:

And receiving a confirmation character data packet returned by a fourth queue pair module in the second target channel adapter module based on a remote direct data access protocol, and determining that the second target channel adapter module receives the target remote direct data access operation data packet according to the confirmation character data packet.

Optionally, after the step of receiving, in the second target channel adapter module, an acknowledge character data packet returned by the fourth queue pair module based on the remote direct data access protocol, and determining, according to the acknowledge character data packet, that the second target channel adapter module receives the target remote direct data access operation data packet, the method includes:

generating a completion queue event according to the confirmation character data packet;

generating a second work queue event according to the physical addresses of the completion queue event and the remote completion queue event register, and sending the second work queue event to the host channel adapter module so that the host channel adapter module stores the completion queue event information in the remote completion queue event register according to the second work queue event.

Optionally, before the step of receiving the remote direct data access packet sent by the host channel adapter module, the method includes:

Receiving a first remote direct data access data packet sent by the host channel adapter module, wherein the first remote direct data access data packet comprises a physical address corresponding to the second target channel adapter module;

and identifying the second target channel adapter module according to the physical address corresponding to the second target channel adapter module.

Optionally, the receiving the remote direct data access packet sent by the host channel adapter module includes:

and receiving a remote direct data access data packet sent by the host channel adapter module under the condition that the completion of the preset configuration in the host channel adapter module is detected, and a trigger register corresponding to the module in a fifth queue in the host channel adapter module is in a trigger state.

In still another aspect of the present application, there is further provided a data transmission method applied to a second target channel adapter module, where a first target channel adapter module is communicatively connected to the second target channel adapter module, the data transmission method including:

and receiving target storage data sent by the first target channel adapter module, wherein the target storage data is transmitted by the first target channel adapter module according to the remote direct data access operation data packet by the host channel adapter module.

In still another aspect of the present application, there is further provided a data transmission method applied to a host channel adapter module, where the host channel adapter module is communicatively connected to the first target channel adapter module and the second target channel adapter module, the data transmission method includes:

and sending a remote direct data access operation data packet to a first target channel adapter module so that the first target channel adapter module performs data transmission to a second target channel adapter module according to the remote direct data access operation data packet.

In still another aspect of the present application, there is also provided a data transmission apparatus applied to a first target channel adapter module, the data transmission apparatus including:

the first receiving module is used for receiving a remote direct data access operation data packet sent by the host channel adapter module under the condition that the first target channel adapter module and the second target channel adapter module are detected to be connected, wherein the remote direct data access operation data packet is used for remotely controlling the first target channel adapter module to transmit data to the second target channel adapter module;

And the first sending module is used for sending the target storage data corresponding to the first target channel adapter module to the second target channel adapter module according to the remote direct data access operation data packet so as to enable the second target channel adapter module to acquire the target storage data.

In still another aspect of the present application, there is further provided a data transmission apparatus applied to a second target channel adapter module, the first target channel adapter module being communicatively connected to the second target channel adapter module, the data transmission method including:

and the second receiving module is used for receiving the target storage data sent by the first target channel adapter module, wherein the target storage data is transmitted by the first target channel adapter module according to the remote direct data access operation data packet by the host channel adapter module.

In still another aspect of the present application, there is further provided a data transmission apparatus applied to a host channel adapter module, where the host channel adapter module is communicatively connected to the first target channel adapter module and the second target channel adapter module, and the data transmission method includes:

And the second sending module is used for sending a remote direct data access operation data packet to the first target channel adapter module so that the first target channel adapter module can conduct data transmission to the second target channel adapter module according to the remote direct data access operation data packet.

In yet another aspect of the present application, there is also provided a communication device including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus;

a memory for storing a computer program;

and the processor is used for realizing any one of the data transmission methods when executing the program stored in the memory.

In yet another aspect of the present application, there is also provided a computer readable storage medium having instructions stored therein, which when run on a computer, cause the computer to perform any of the data transmission methods described above.

In yet another aspect of the application, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform any of the data transmission methods described above.

The data transmission system provided by the embodiment of the application comprises: a host channel adapter module, a first target channel adapter module, and a second target channel adapter module; the host channel adapter module is in communication connection with the first target channel adapter module and the second target channel adapter module; the first target channel adapter module is communicatively coupled to the second target channel adapter module; the host channel adapter module is used for sending a remote direct data access operation data packet to the first target channel adapter module, and remotely controlling the first target channel adapter module to carry out data transmission to the second target channel adapter module according to the remote direct data access operation data packet, namely the embodiment of the application remotely controls the data transmission among the target channel adapter modules through the host channel adapter module, so that the data can be directly transmitted among the target channel adapter devices under the condition that the devices are not increased, the direct remote direct data access transmission of the data among the target channel adapters is realized, the network utilization rate is improved, the data transmission delay is reduced, the implementation of an FPGA (field programmable gate array) is easy, and the distributed computation delay of the FPGA cloud platform can be greatly reduced.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.

Fig. 1 shows a schematic diagram of a data transmission system according to an embodiment of the present application;

fig. 2 is a schematic diagram of a related art data transmission system according to an embodiment of the present application;

fig. 3 is a schematic diagram of another data transmission system according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a target channel adapter module according to an embodiment of the present application;

fig. 5 shows a step flowchart of a data transmission method according to an embodiment of the present application;

fig. 6 is a flowchart illustrating steps of another data transmission method according to an embodiment of the present application;

fig. 7 is a flowchart illustrating steps of another data transmission method according to an embodiment of the present application;

fig. 8 is a flowchart illustrating steps of another data transmission method according to an embodiment of the present application;

fig. 9 shows a block diagram of a data transmission device according to an embodiment of the present application;

fig. 10 is a block diagram of another data transmission device according to an embodiment of the present application;

Fig. 11 is a block diagram of another data transmission apparatus according to an embodiment of the present application;

fig. 12 is a block diagram of a communication device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the following detailed description of the embodiments of the present application will be given with reference to the accompanying drawings. However, those of ordinary skill in the art will understand that in various embodiments of the present application, numerous technical details have been set forth in order to provide a better understanding of the present application. However, the claimed application may be practiced without these specific details and with various changes and modifications based on the following embodiments. The following embodiments are divided for convenience of description, and should not be construed as limiting the specific implementation of the present application, and the embodiments can be mutually combined and referred to without contradiction.

It should be noted that, based on the data transmission of the remote direct data access protocol, a queue pair connection needs to be established between the nodes before the transmission, each node may have a plurality of queue pair module resources, and when the connection is established, the queue pair module number used by the opposite node needs to be notified. The remote direct data access divides the computing nodes into two types of host channel adapter modules and target channel adapters, wherein the host channel adapter modules are servers with operating systems and the like, and can initiate remote direct data access operation; while the target channel adapter is a network hard disk, an accelerator card, etc. without a host, only can respond to remote direct data access operation, referring to fig. 2, fig. 2 shows a schematic diagram of a data transmission system in a related art provided by an embodiment of the present application, in the related art, data transmission is performed between two target channel adapter modules, in which, data is read from one of the target channel adapter modules by the host channel adapter module and then sent to the other target channel adapter module, specifically, the host channel adapter module is connected to the first target channel adapter module by a queue pair 1 and thus obtains data, and the host channel adapter module is connected to the second target channel adapter module by a queue pair 2 and a queue pair 1 in the second target channel adapter module, so that data in the first target channel adapter module is transmitted to the second target channel adapter module, thereby causing a host to allocate resources such as a hard disk or an accelerator card or delay when in use is larger in the data transmission process.

Referring to fig. 1 and 3, fig. 1 shows a schematic diagram of a data transmission system provided by an embodiment of the present application, and fig. 3 shows a schematic diagram of another data transmission system provided by an embodiment of the present application, where the data transmission system includes a host channel adapter module, a first target channel adapter module, and a second target channel adapter module.

It should be noted that, in the embodiment of the present application, the host channel adapter module is communicatively connected to the first target channel adapter module and the second target channel adapter module, and the first target channel adapter module is communicatively connected to the second target channel adapter module.

Specifically, the first target channel adapter module and the second target channel adapter module are in communication connection through a queue pair, and when it is required to be described, in the remote direct data access protocol, the WQ is a queue for storing a work request, the WQE is an element in the WQ queue, the WQE can be considered as a "task description", the work request is issued to hardware by software, and the description includes a task that the hardware hoped to do by the software and detailed information about the task. For example, a task is "send 10 bytes of data at address 0x12345678 to the opposite node", and after the hardware receives the task, it will fetch the data in the memory by DMA, assemble the packet, and send it.

Any communication process has two transceiving ends, so the queue pair is a combination of a sending work queue and a receiving work queue, which may be respectively referred to as SQ (Send Queue) and RQ (Receive Queue), and therefore, in the embodiment of the present application, when the host channel adapter module and the target channel adapter module communicate or transmit data, there may be a plurality of corresponding queue pair modules, for example, the first queue pair module in fig. 1 is one of the first target channel adapter modules, and may be connected to the fifth queue pair module in the host channel adapter module, and when the host channel adapter module needs to perform a remote direct data access operation, the remote direct data access packet may be sent to the first queue pair module through the fifth queue pair module.

Therefore, in the embodiment of the present application, in order to directly perform data transmission between the target channel adapter and the target channel adapter, it is necessary to first establish a connection between the first target channel adapter module and the second target channel adapter module.

Further, the first target channel adapter module and the second target channel adapter module communicatively coupled includes: the host channel adapter module sends a first remote direct data access data packet and a second remote direct data access data packet to the first target channel adapter module and the second target channel adapter module respectively, so that the first target channel adapter module and the second target channel adapter module establish connection according to the first remote direct data access data packet and the second remote direct data access data packet.

Further, the host channel adapter module is further configured to send a first remote direct data access packet to a first parsing module in the first target channel adapter module, where the first remote direct data access packet includes a physical address corresponding to the second target channel adapter module, and send a second remote direct data access packet to a second parsing module in the second target channel adapter module, where the second remote direct data access packet includes a physical address corresponding to the first target channel adapter module, so that the first target channel adapter module and the second target channel adapter module establish a connection according to the remote direct data access packet.

It should be noted that, in the embodiment of the present application, the parsing module may be denoted as a reg_operation module, so the first parsing module may be denoted as a first reg_operation module, and the second parsing module may be denoted as a second reg_operation module.

It should be noted that, the host channel adapter module may perform remote transmission control based on a remote direct data access protocol, so the host channel adapter module may send remote direct data access data packets to multiple target channel adapter modules (for example, a first target channel adapter module and a second target channel adapter module), where in this embodiment of the present application, the data packets may include physical addresses corresponding to the respective target channel adapter modules or identification information corresponding to the respective target channel adapter modules, for example, the host channel adapter module may send a first remote direct data access data packet to a first parsing module in the first target channel adapter module, where the first remote direct data access data packet may include physical addresses corresponding to the second target channel adapter module, or identification information of the second target channel adapter module, and in this way, after the first target channel adapter module parses the data packets, the second target channel adapter module may obtain identification information of the second target channel adapter module, and the second target channel adapter module may receive the second remote direct data access data packet, where the second target channel adapter module may be parsed by the first target channel adapter module and the first target channel adapter module.

Specifically, a register of an idle queue pair module in the target channel adapter can be configured by a remote register writing mode of the host channel adapter module, so that the first target channel adapter module and the second target channel adapter module are connected by the queue pair module. After the two target channel adapters establish queue pair connection, the host channel adapter module can initiate remote direct data access operation to control the data of the first target channel adapter module to be written into the preset address of the second target channel adapter module.

Further, the host channel adapter module is configured to send a remote direct data access operation data packet to the first target channel adapter module, and remotely control the first target channel adapter module to perform data transmission to the second target channel adapter module according to the remote direct data access operation data packet.

The remote direct data access operation data packet can comprise a load and a header, the load of the data packet can be the write address and write data of an internal register of the target channel adapter module, and the data format can be predefined as a preset remote write register format; the destination storage address field of the remote direct data access packet basic transport layer header (Base Transport Head, BTH) may be the address of the resolution module reg_operation inside the target channel adapter module. After receiving the data packet, the reg_operation module of the target channel adapter module extracts the write address and write data of the register in the load, converts the write address and the write data into data bus write operation, and further performs write operation on all registers connected with the data bus inside the target channel adapter.

Further, the first target channel adapter module includes an rwqe_engine module, an rcqe_engine module, a first queue pair module, and a second queue pair module; the rwqe_engine module is used for forming WQE of work queue event WQE data recorded by an internal register in the first queue pair module, sending the WQE to the second queue pair module and triggering remote direct data access operation; the rcqe_engine module is configured to send the completion queue event CQE information returned by the second queue pair module and the physical address corresponding to the host channel adapter module recorded by the internal register in the first queue pair module to the first queue pair module, so that the first queue pair module returns the CQE information to the host channel adapter module.

Further, the first target channel adapter module further comprises a first parsing module; the first parsing module is configured to receive a remote direct data access data packet sent by the host channel adapter module, parse the remote direct data access data packet, and convert the parsed remote direct data access data packet into a data bus write operation.

Specifically, two modules, namely a rwqe_engine module and an rcqe_engine module, are added in the first target channel adapter module, wherein the rwqe_engine module can send a remote WQE data composition WQE recorded by an internal register to the second queue pair module to trigger remote direct data access operation. The rcqe_engine module can store the CQE information returned by the second queue module and the physical address of the remote CQE returned by the host channel adapter module recorded by the internal register, and form WQE to be sent to the queue pair module, so that the CQE information generated by the first target channel adapter module is fed back to the host channel adapter module in the form of a remote direct data access data packet.

In addition, a first parsing module is further added in the first target channel adapter module, and the first parsing module is a parsing module, and is configured to receive a remote direct data access data packet sent by the host channel adapter module, parse the remote direct data access data packet, and convert the parsed remote direct data access data packet into a data bus write operation.

Further, the host channel adapter module comprises a fifth queue pair module and a sixth queue pair module; the fifth queue pair module is in communication connection with a third queue pair module in the second target channel adapter module; the sixth queue pair module is communicatively coupled to the first queue pair module in the second target channel adapter module. The host channel adapter module is configured to send a remote direct data access operation data packet to the first target channel adapter module, and remotely control the first target channel adapter module to perform data transmission to the second target channel adapter module according to the remote direct data access operation data packet.

It should be noted that, in the embodiment of the present application, the host channel adapter module is mainly used to remotely control the first target channel adapter module, so that the first target channel adapter module transmits the stored data to the second target channel adapter module.

Further, a remote completion queue event register and a remote work queue event register are preset in a host memory corresponding to the host channel adapter module; the remote completion queue event register is used for storing CQE information returned by the target channel adapter; the remote work queue event register is used for storing remote direct data access operation information according to a preset remote write register format.

The remote direct data access operation information comprises WQE information and a physical address of a remote completion queue event register; the WQE corresponds to an internal register of the rwqe_engine module in the first target channel adapter module; the physical address of the remote completion queue event register corresponds to an internal register of the rcqe_engine module in the first target channel adapter module.

It should be noted that, in the embodiment of the present application, the host channel adapter module is connected to a host, where the host may be a server, and before the target channel adapter module performs data transmission, a remote completion queue event register and a remote work queue event register are preset in a host memory of the host channel adapter module, where the remote completion queue event register is used to store CQE information returned by the target channel adapter; the remote work queue event register is used to store information required to initiate a remote direct data access operation in a remote write register format. The information includes WQE information of remote direct data access and physical address of remote completion queue event register, which correspond to rwqe_Engine module internal register and rcqe_Engine module internal register in target channel adapter respectively.

Further, the second target channel adapter module comprises a third queue pair module and a fourth queue pair module; the third queue pair module is in communication connection with a sixth queue pair module in the host channel adapter module; the fourth queue pair module is communicatively coupled to a second queue pair module in the first target channel adapter module. The second target channel adapter module further comprises a second analysis module; the second analyzing module is configured to receive the remote direct data access data packet sent by the host channel adapter module, analyze the remote direct data access data packet, and identify the first target channel adapter module according to the analyzed remote direct data access data packet.

It should be noted that, in the embodiment of the present application, there may be a plurality of second target channel adapter modules, where the second target channel adapter modules are mainly used to receive the data stored in the first target channel adapter modules.

Therefore, the second target channel adapter module includes a fourth queue pair module that needs to establish a queue pair connection with the first target channel adapter module, and a third queue pair module that is connected with the host channel adapter module, and the second parsing module is configured to receive the remote direct data access packet sent by the host channel adapter module, parse the remote direct data access packet, and identify the first target channel adapter module according to the parsed remote direct data access packet.

The data transmission system provided by the embodiment of the application comprises: a host channel adapter module, a first target channel adapter module, and a second target channel adapter module; the host channel adapter module is in communication connection with the first target channel adapter module and the second target channel adapter module; the first target channel adapter module is communicatively coupled to the second target channel adapter module; the host channel adapter module is used for sending a remote direct data access operation data packet to the first target channel adapter module, and remotely controlling the first target channel adapter module to carry out data transmission to the second target channel adapter module according to the remote direct data access operation data packet, namely the host channel adapter module is used for remotely controlling the data transmission among the target channel adapter modules, so that the data can be directly transmitted among the target channel adapter devices without increasing devices, the data transmission delay is greatly reduced, and the network bandwidth utilization rate is improved.

Referring to fig. 4, fig. 4 shows a schematic diagram of a target channel adapter module provided by the embodiment of the present application, in the embodiment of the present application, any one server may be used as a host, the host is connected to the host through a PCI-E slot to form a host channel adapter module, two accelerator cards that are not connected to the host and independently supply power are used as a first target channel adapter module and a second target channel adapter module, the accelerator card is shown in fig. 4, the main chip may be an FPGA, and the external devices include an sdam, a PCI-E slot, and a 100G optical module. By the method and the device, the host can control the data in the sdam of the first target channel adapter module through the host channel adapter module, and the data is directly moved to the sdam of the second target channel adapter module through the Ethernet without being transferred through the host channel adapter module.

Therefore, the embodiment of the application can realize the direct remote direct data access transmission of the data among the target channel adapters without increasing a server, improves the network utilization rate, reduces the data transmission delay, is easy for the FPGA to realize, and can greatly reduce the delay of distributed computation of the FPGA cloud platform.

Referring to fig. 5, a step flowchart of a data transmission method provided by an embodiment of the present application is shown, where the method may include:

and step 101, receiving a remote direct data access operation data packet sent by the host channel adapter module under the condition that the first target channel adapter module and the second target channel adapter module are detected to establish connection, wherein the remote direct data access operation data packet is used for remotely controlling the first target channel adapter module to transmit data to the second target channel adapter module.

Step 102, sending the target storage data corresponding to the first target channel adapter module to a second target channel adapter module according to the remote direct data access operation data packet, so that the second target channel adapter module obtains the target storage data.

Further, step 102 includes: analyzing the remote direct data access operation data packet, writing a first work queue event in the remote work queue event register data into an internal register of a remote work queue event engine module according to the analyzed remote direct data access operation data packet, and writing a physical address of the remote completion queue event register into the internal register of the remote completion queue event engine module; the remote work queue event engine module sending the first work queue event to a second queue pair module; the second queue pair module obtains target storage data corresponding to the first target channel adapter module according to the first work queue event, generates a target remote direct data access operation data packet according to the target storage data, and sends the target remote direct data access operation data packet to the second target channel adapter module so that the second target channel adapter module obtains the target remote direct data access operation data packet.

It should be noted that, in the foregoing steps 101-102, when it is detected that the first target channel adapter module and the second target channel adapter module establish a connection, a remote direct data access operation data packet sent by the host channel adapter module is received, where the remote direct data access operation data packet includes remote work queue event register data and a physical address of a remote completion queue event register;

it should be noted that, in the embodiment of the present application, the first target channel adapter module is applied to a first target channel adapter module, where the first target channel adapter module includes a rwqe_engine module, an rcqe_engine module, a first queue pair module, a second queue pair module, and a first parsing module; the rwqe_engine module is used for forming WQE of work queue event WQE data recorded by an internal register in the first queue pair module, sending the WQE to the second queue pair module and triggering remote direct data access operation; the rcqe_engine module is configured to send the completion queue event CQE information returned by the second queue pair module and the physical address corresponding to the host channel adapter module recorded by the internal register in the first queue pair module to the first queue pair module, so that the first queue pair module returns the CQE information to the host channel adapter module. The first parsing module is configured to receive a remote direct data access data packet sent by the host channel adapter module, parse the remote direct data access data packet, and convert the parsed remote direct data access data packet into a data bus write operation.

The first target channel adapter module may perform data transmission after establishing a queue pair connection with the second target channel adapter module, where the first target channel adapter module may receive a remote direct data access operation data packet sent by the host channel adapter module, where the remote direct data access operation data packet includes remote work queue event register data and a physical address of a remote completion queue event register.

Before data transmission, a remote completion queue event register and a remote work queue event register are preset in a host memory corresponding to a host connected with the host channel adapter module, wherein the remote completion queue event register is used for storing CQE information returned by the target channel adapter; the remote work queue event register is used to store information required to initiate a remote direct data access operation in a remote write register format. The information includes WQE information of remote direct data access and physical address of remote completion queue event register, which correspond to rwqe_Engine module internal register and rcqe_Engine module internal register in target channel adapter respectively.

Thus, the remote direct data access operation data packet sent by the host channel adapter module to the first target channel adapter module includes the physical addresses of the remote work queue event register data and the remote completion queue event register.

Further, the receiving the remote direct data access packet sent by the host channel adapter module includes: and receiving a remote direct data access data packet sent by the host channel adapter module under the condition that the completion of the preset configuration in the host channel adapter module is detected, and a trigger register corresponding to the module in a fifth queue in the host channel adapter module is in a trigger state.

Specifically, the local storage address in the WQE in the host memory is set to be the physical address of the remote work queue event register, the destination address is set to be the reg_operation module address of the first target channel adapter module, and the data length is the size of the remote work queue event register. When the trigger register of the fifth queue pair module of the host channel adapter module is triggered in the host, the fifth queue pair module can automatically acquire WQE in the host memory, automatically acquire remote work queue event register data according to the WQE, and form a remote direct data access operation data packet to be sent to the first target channel adapter module.

It should be noted that, in the embodiment of the present application, after receiving the remote direct data access operation data packet, the data packet is parsed, and the parsed remote direct data access operation data packet writes the first WQE in the remote work queue event register data into the internal register of the rwqe_engine module, and writes the physical address of the remote completion queue event register into the internal register of the rcqe_engine module.

Specifically, the reg_operation module of the first target channel adapter module receives the data packet and analyzes the load, and writes WQE information of remote direct data access in the event register data of the remote work queue into an internal register of the rwqe_engine module according to the register address; the physical address of the remote completion queue event register is written to an rcqe_engine module internal register.

It should be noted that, in the embodiment of the present application, the rwqe_engine module sends the WQE of the remote direct data access to the WQE port of the queue pair module according to the register configuration, and triggers the remote direct data access to write data.

Further, the WQE includes a WQE identifier, a storage physical address of a first target channel adapter module, a destination storage address of a second target channel adapter module, and a data byte length, the second queue pair module obtains target storage data corresponding to the first target channel adapter module according to the WQE, generates a target remote direct data access operation data packet according to the target storage data, and sends the target remote direct data access operation data packet to the second target channel adapter module, including: the second queue pair module obtains target storage data corresponding to the first target channel adapter module according to the storage physical address of the first target channel adapter module, generates a target remote direct data access operation data packet according to the target storage data, and sends the target remote direct data access operation data packet to the second target channel adapter module according to the target storage address of the second target channel adapter module.

It should be noted that, in the embodiment of the present application, the WQE includes information such as a WQE identifier (wqe_id), a local storage physical address (local_addr) of data (first target channel adapter module), a destination storage address (remote_addr) of data sent to a remote end (second target channel adapter module), and a data byte length. And the second queue pair module automatically acquires data stored locally (the first target channel adapter module) according to the WQE information, and forms a target remote direct data access operation data packet to be sent to the second target channel adapter module.

Further, after the step of the second queue pair module obtaining the target storage data corresponding to the first target channel adapter module according to the first WQE, generating a target remote direct data access operation data packet according to the target storage data, and sending the target remote direct data access operation data packet to a second target channel adapter module, the method includes: and receiving a data transmission completion signal returned by the second target channel adapter module, and determining that the second target channel adapter module receives the target remote direct data access operation data packet according to the data transmission completion signal.

Further, the receiving the data transmission completion signal returned by the second target channel adapter module, and determining, according to the data transmission completion signal, that the second target channel adapter module receives the target remote direct data access operation data packet includes: and receiving an acknowledgement character (Acknowledge character, ACK) data packet returned by a fourth queue pair module in the second target channel adapter module based on a remote direct data access protocol, and determining that the second target channel adapter module receives the target remote direct data access operation data packet according to the ACK data packet.

Further, after the step of receiving, by the fourth queue pair module in the second target channel adapter module, an ACK packet returned by the fourth queue pair module based on the remote direct data access protocol, and determining, according to the ACK packet, that the second target channel adapter module receives the target remote direct data access operation packet, the method includes: generating CQE according to the ACK data packet; generating a second WQE according to the physical address of the CQE and the remote completion queue event register, and sending the second WQE to the host channel adapter module so that the host channel adapter module stores the CQE information in the remote completion queue event register according to the second WQE.

It should be noted that, in the embodiment of the present application, the fourth queue pair module of the second target channel adapter module stores the data into the preset storage space of the second target channel adapter module according to the remote_addr (data is sent to the destination storage address of the second target channel adapter module) information included in the remote direct data access data packet, and returns the ACK data packet to the first target channel adapter module according to the remote direct data access protocol. And after the second queue pair module of the first target channel adapter module receives the ACK data packet, judging that the data transmission is completed, and generating CQE information to the rcqe_engine module. The rcqe_engine module composes the CQE information and the physical address of the remote completion queue event register recorded in the register into WQE, and sends the WQE information and the physical address of the remote completion queue event register to a first queue pair module connected with the host channel adapter module, and the first queue pair module composes the CQE information into a remote direct data access data packet according to the WQE and sends the remote direct data access data packet to the host channel adapter module.

In the embodiment of the application, a remote direct data access operation data packet sent by a host channel adapter module is received under the condition that the connection between the first target channel adapter module and the second target channel adapter module is detected, wherein the remote direct data access operation data packet comprises remote work queue event register data and a physical address of a remote completion queue event register; analyzing the remote direct data access operation data packet, writing a first WQE in the remote work queue event register data into an internal register of an rwqe_engine module according to the analyzed remote direct data access operation data packet, and writing a physical address of the remote completion queue event register into the internal register of the rcqe_engine module; the rwqe_engine module sends the first WQE to a second queue pair module; the second queue pair module obtains target storage data corresponding to the first target channel adapter module according to the first WQE, generates a target remote direct data access operation data packet according to the target storage data, and sends the remote direct data access operation data packet to a second target channel adapter module so that the second target channel adapter module obtains the remote direct data access operation data packet. In the embodiment of the application, under the condition that the first target channel adapter module and the second target channel adapter module are connected, the first target channel adapter module directly transmits the data stored in the first target channel adapter module to the second target channel adapter module through the remote control of the host channel adapter module, and under the condition that a server is not added, the direct remote direct data access transmission of the data among the target channel adapters is realized, the network utilization rate is improved, the data transmission delay is reduced, the realization of an FPGA is easy, and the distributed computation delay of an FPGA cloud platform can be greatly reduced.

Referring to fig. 6, a flowchart illustrating steps of another data transmission method according to an embodiment of the present application is shown, where the method may include:

step 001, receiving a first remote direct data access data packet sent by the host channel adapter module, where the first remote direct data access data packet includes a physical address corresponding to the second target channel adapter module;

step 002, identifying a second target channel adapter module according to the physical address corresponding to the second target channel adapter module;

it should be noted that, in the foregoing steps 001-002, before the data transmission between the target channel adapters is performed, a queue pair connection between the first target channel adapter module and the second target channel adapter module needs to be established, so that, at this time, the first target channel adapter module may receive a first remote direct data access data packet sent by the host channel adapter module, where the first remote direct data access data packet includes a physical address corresponding to the second target channel adapter module, and identifies the second target channel adapter module according to the physical address corresponding to the second target channel adapter module.

Likewise, the second target channel adapter module also receives the second remote direct data access packet, thereby identifying the first target channel adapter module.

Step 101, receiving a remote direct data access operation data packet sent by a host channel adapter module under the condition that the first target channel adapter module and the second target channel adapter module are detected to establish connection, wherein the remote direct data access operation data packet is used for remotely controlling the first target channel adapter module to transmit data to the second target channel adapter module;

It should be noted that, in the embodiment of the present application, the steps 101-102 are discussed with reference to the foregoing, and are not repeated here.

According to the embodiment of the application, under the condition that the first target channel adapter module and the second target channel adapter module are connected, the first target channel adapter module directly transmits the data stored in the first target channel adapter module to the second target channel adapter module through the remote control of the host channel adapter module, and under the condition that a server is not added, the direct remote direct data access transmission of the data among the target channel adapters is realized, the network utilization rate is improved, the data transmission delay is reduced, the realization of an FPGA is easy, and the distributed computation delay of an FPGA cloud platform can be greatly reduced.

Referring to fig. 7, a flowchart illustrating steps of another data transmission method according to an embodiment of the present application is shown, where the method may include:

step 201, receiving target storage data sent by the first target channel adapter module, where the target storage data is transmitted by the first target channel adapter module according to the host channel adapter module remotely controlled by the host channel adapter module according to a remote direct data access operation data packet.

It should be noted that, in the embodiment of the present application, the second target channel adapter module is applied, and the first target channel adapter module is communicatively connected to the second target channel adapter module.

Referring to fig. 8, a flowchart illustrating steps of another data transmission method according to an embodiment of the present application is shown, where the method may include:

step 301, a remote direct data access operation data packet is sent to a first target channel adapter module, so that the first target channel adapter module performs data transmission to the second target channel adapter module according to the remote direct data access operation data packet.

It should be noted that, in the embodiment of the present application, the application is applied to a host channel adapter module, where the host channel adapter module is communicatively connected to the first target channel adapter module and the second target channel adapter module.

Referring to fig. 9, fig. 9 shows a data transmission apparatus provided by an embodiment of the present application, where the apparatus may include:

a first receiving module 401, configured to receive a remote direct data access operation data packet sent by a host channel adapter module when it is detected that the first target channel adapter module and the second target channel adapter module establish a connection, where the remote direct data access operation data packet is used to remotely control the first target channel adapter module to perform data transmission to the second target channel adapter module;

and the first sending module 402 is configured to send, according to the remote direct data access operation data packet, target storage data corresponding to the first target channel adapter module to the second target channel adapter module, so that the second target channel adapter module obtains the target storage data.

Referring to fig. 10, fig. 10 shows another data transmission apparatus provided in an embodiment of the present application, where the apparatus may include:

and a second receiving module 501, configured to receive target storage data sent by the first target channel adapter module, where the target storage data is transmitted by the first target channel adapter module according to the host channel adapter module and remotely controlled by the host channel adapter module according to a remote direct data access operation data packet.

Referring to fig. 11, fig. 11 shows another data transmission apparatus provided in an embodiment of the present application, where the apparatus may include:

And the second sending module 601 is configured to send a remote direct data access operation data packet to the first target channel adapter module, so that the first target channel adapter module performs data transmission to the second target channel adapter module according to the remote direct data access operation data packet.

The embodiment of the present application also provides a communication device, as shown in fig. 12, including a processor 901, a communication interface 902, a memory 903, and a communication bus 904, where the processor 901, the communication interface 902, and the memory 903 perform communication with each other through the communication bus 904,

A memory 903 for storing a computer program;

the processor 901, when executing the program stored in the memory 903, may implement the following steps:

The communication bus mentioned by the above terminal may be a peripheral component interconnect standard (Peripheral Component Interconnect, abbreviated as PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, abbreviated as EISA) bus, etc. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.

The communication interface is used for communication between the terminal and other devices.

The memory may include random access memory (Random Access Memory, RAM) or non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.

The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but also digital signal processors (Digital Signal Processing, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), field-programmable gate arrays (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

In yet another embodiment of the present application, a computer readable storage medium having instructions stored therein, which when run on a computer, cause the computer to perform the data transmission of any of the above embodiments is also provided.

In yet another embodiment of the present application, there is also provided a computer program product containing instructions that, when run on a computer, cause the computer to perform the data transmission of any of the above embodiments.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or third database to another website, computer, server, or third database by a wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device including one or more servers, third databases, etc. that can be integrated with the available medium. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.

It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.

The foregoing description is only of the preferred embodiments of the present application and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A data transmission system, the data transmission system comprising: a host channel adapter module, a first target channel adapter module, and a second target channel adapter module;

the host channel adapter module is used for sending a remote direct data access operation data packet to the first target channel adapter module, and remotely controlling the first target channel adapter module to carry out data transmission to the second target channel adapter module according to the remote direct data access operation data packet;

the first target channel adapter module comprises a remote work queue event engine module, a remote completion queue event engine module, a first queue pair module, a second queue pair module and a first analysis module, and is used for establishing communication, triggering data access and converting write operation;

The second target channel adapter module comprises a third queue pair module, a fourth queue pair module and a second analysis module, and is used for establishing communication and triggering data access;

the host channel adapter module sends the remote direct data access operation data packet to a plurality of target channel adapter modules respectively, the remote direct data access operation data packet comprises physical addresses corresponding to the target channel adapter modules or identification information corresponding to the target channel adapter modules, and the host channel adapter module is in communication connection with the first target channel adapter module and the second target channel adapter module respectively through the physical addresses and the identification information.

2. The data transmission system of claim 1, wherein the first target channel adapter module and the second target channel adapter module communicatively connect comprises: the host channel adapter module sends a first remote direct data access data packet and a second remote direct data access data packet to the first target channel adapter module and the second target channel adapter module respectively, so that the first target channel adapter module and the second target channel adapter module establish connection according to the first remote direct data access data packet and the second remote direct data access data packet.

3. The data transmission system of claim 1, wherein the first target channel adapter module comprises a remote work queue event engine module, a remote completion queue event engine module, a first queue pair module, and a second queue pair module;

4. The data transmission system of claim 1, wherein the first target channel adapter module further comprises a first parsing module;

the first analysis module is used for receiving the remote direct data access operation data packet sent by the host channel adapter module, analyzing the remote direct data access operation data packet, and converting the analyzed remote direct data access operation data packet into a data bus write operation.

5. The data transmission system of claim 1, wherein the second target channel adapter module comprises a third queue pair module and a fourth queue pair module;

6. The data transmission system of claim 1, wherein the second target channel adapter module further comprises a second parsing module;

the second analyzing module is configured to receive a second remote direct data access packet sent by the host channel adapter module, analyze the second remote direct data access packet, and identify the first target channel adapter module according to the analyzed second remote direct data access packet.

7. The data transmission system of claim 1, wherein the host channel adapter module comprises a fifth queue pair module and a sixth queue pair module;

8. The data transmission system of claim 1, wherein the host channel adapter module is further configured to send a first remote direct data access packet to a first parsing module of the first target channel adapter modules, the first remote direct data access packet including a physical address corresponding to the second target channel adapter module, and a second remote direct data access packet to a second parsing module of the second target channel adapter modules, the second remote direct data access packet including a physical address corresponding to the first target channel adapter module, such that the first target channel adapter module and the second target channel adapter module establish a connection according to the remote direct data access packet.

9. The data transmission system according to claim 1, wherein a remote completion queue event register and a remote work queue event register are preset in a host memory corresponding to the host channel adapter module;

10. The data transfer system of claim 9, wherein the remote direct data access operation information includes work queue event information and a physical address of a remote completion queue event register;

the work queue event corresponds to an internal register of a remote work queue event engine module in the first target channel adapter module;

11. A data transmission method, applied to the first target channel adapter module in the data transmission system of any one of claims 1 to 10, comprising:

12. The method of claim 11, wherein the remote direct data access operation data packet includes remote work queue event register data and physical addresses of remote completion queue event registers, and wherein the sending the target storage data corresponding to the first target channel adapter module to a second target channel adapter module according to the remote direct data access operation data packet to cause the second target channel adapter module to acquire the target storage data comprises:

13. The method according to claim 12, wherein the work queue event includes a work queue event identifier, a storage physical address of a first target channel adapter module, a destination storage address of a second target channel adapter module, and a data byte length, the second queue pair module obtains target storage data corresponding to the first target channel adapter module according to the work queue event, and generates a target remote direct data access operation data packet according to the target storage data, and transmitting the target remote direct data access operation data packet to the second target channel adapter module includes:

14. The method of claim 12, wherein after the step of the second queue pair module obtaining the target storage data corresponding to the first target channel adapter module according to the first work queue event, and generating a target remote direct data access operation data packet according to the target storage data, and sending the target remote direct data access operation data packet to a second target channel adapter module, the method comprises:

15. The method according to claim 14, wherein receiving a data transmission completion signal returned by the second target channel adapter module, and determining that the second target channel adapter module receives the target remote direct data access operation data packet according to the data transmission completion signal comprises:

16. The data transmission method according to claim 15, wherein after said step of receiving an acknowledge character data packet returned by a fourth queue pair module in a second target channel adapter module based on a remote direct data access protocol, and determining that said second target channel adapter module receives said target remote direct data access operation data packet based on said acknowledge character data packet, said method comprises:

17. The method of claim 11, wherein prior to the step of receiving a remote direct data access packet sent by the host channel adapter module, the method comprises:

18. The method of claim 11, wherein receiving the remote direct data access packet sent by the host channel adapter module comprises:

19. A data transmission method, applied to the second target channel adapter module in the data transmission system of any one of claims 1 to 10, wherein the first target channel adapter module is communicatively connected to the second target channel adapter module, the data transmission method comprising:

And receiving target storage data sent by the first target channel adapter module, wherein the target storage data is transmitted by the first target channel adapter module according to a remote direct data access operation data packet by the host channel adapter module.

20. A data transmission method, applied to the host channel adapter module in any one of the data transmission systems of claims 1 to 10, the host channel adapter module being communicatively connected to a first target channel adapter module and a second target channel adapter module, the data transmission method comprising:

21. A data transmission device, characterized by a first target channel adapter module for use in the data transmission system of any of claims 1-10, said data transmission device comprising:

22. A data transmission device, characterized by a second target channel adapter module for use in any of the data transmission systems of claims 1-10, the first target channel adapter module being communicatively coupled to the second target channel adapter module, the data transmission device comprising:

23. A data transmission device, characterized by a host channel adapter module applied in the data transmission system of any one of claims 1-10, the host channel adapter module being communicatively connected with a first target channel adapter module and a second target channel adapter module, the data transmission device comprising:

24. A communication device, comprising: a transceiver, a memory, a processor, and a program stored on the memory and executable on the processor;

the processor is configured to read a program in the memory to implement the data transmission method according to any one of claims 11 to 17, or the data transmission method according to claim 19, or the data transmission method according to claim 20.

25. A readable storage medium storing a program, wherein the program, when executed by a processor, implements the data transmission method according to any one of claims 11 to 17, or the data transmission method according to claim 19, or the data transmission method according to claim 20.