CN112398817B - Data sending method and device - Google Patents

Data sending method and device Download PDF

Info

Publication number
CN112398817B
CN112398817B CN202011203280.8A CN202011203280A CN112398817B CN 112398817 B CN112398817 B CN 112398817B CN 202011203280 A CN202011203280 A CN 202011203280A CN 112398817 B CN112398817 B CN 112398817B
Authority
CN
China
Prior art keywords
virtual
port
virtual machine
address
identifier
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011203280.8A
Other languages
Chinese (zh)
Other versions
CN112398817A (en
Inventor
付斌章
苏金钊
谭焜
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to CN202011203280.8A priority Critical patent/CN112398817B/en
Publication of CN112398817A publication Critical patent/CN112398817A/en
Application granted granted Critical
Publication of CN112398817B publication Critical patent/CN112398817B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L61/00—Network arrangements, protocols or services for addressing or naming
    • H04L61/09—Mapping addresses
    • H04L61/10—Mapping addresses of different types
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/08—Protocols specially adapted for terminal emulation, e.g. Telnet
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/30—Definitions, standards or architectural aspects of layered protocol stacks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/45587—Isolation or security of virtual machine instances

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Software Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Computing Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The embodiment of the invention provides a data sending method and equipment, and relates to the technical field of communication. The invention can realize the isolation of the virtual network and the physical network of the virtual machine, ensure the safety of the virtual machine and be compatible with the RoCE protocol.

Description

Data sending method and device
Technical Field
The present application relates to the field of communications technologies, and in particular, to a method and a device for data transmission.
Background
At present, a Remote Direct Access over Ethernet (RoCE) Protocol based on Converged Ethernet is widely applied to cloud computing, and the RoCE Protocol is adopted, so that on one hand, the problems of large network delay and high load of a Central Processing Unit (CPU) when a traditional Transmission Control Protocol/Internet Protocol (TCP/IP) Protocol transmits and receives data can be solved, and on the other hand, the Remote Direct Access over Ethernet (RoCE) Protocol can be compatible, and the requirements of low investment and low operation cost can be met. A plurality of virtual machines are generally arranged in a server in cloud computing, and meanwhile, a Host Channel Adapter (HCA) supporting a RoCE protocol is installed, the HCA card includes a plurality of physical ports, and for each physical port, the physical port may be abstracted into an ethernet interface supporting an ethernet protocol and an RDMA interface supporting a Remote Direct Memory Access (RDMA) protocol, and an administrator may configure a physical IP address corresponding to the virtual machine on the ethernet interface.
In the prior art, when the virtual machines in different servers need to communicate data, hardware virtualization functions, such as SR-IOV, need to be started on the HCA card. In this way, the virtual machine can establish connection with the other side according to the physical IP address configured by the administrator, establish a Queue Pair (QP) on the corresponding RDMA interface, and set the source address of the QP Queue as the physical IP address of the virtual machine itself and the destination address as the physical IP address of the other side. In the subsequent data transmission process, the HCA card directly reads the data in the memory corresponding to the application program, packages the data according to the source address and the destination address in the QP queue to obtain a data message, and transmits the data message through a physical network.
Therefore, in the prior art, network virtualization of the RoCE protocol cannot be realized through software, that is, the prior art cannot provide a data transmission method which can realize isolation of a virtual network of a virtual machine from a physical network, ensure safety of the virtual machine, and be compatible with the RoCE protocol.
Disclosure of Invention
The embodiment of the invention provides a data sending method and equipment, which can realize virtual network data isolation and reduce the security risk of a virtual machine. The technical scheme is as follows:
in a first aspect, a method for data transmission is provided, where an apparatus applied in the method includes a first virtual machine and a host, and the method includes: the host obtains a first instruction and a second instruction from the first virtual machine, wherein the first instruction is used for indicating that a source address of a queue pair QP queue is a virtual address of the first virtual machine, the second instruction is used for indicating that a destination address of the QP queue is a virtual address of the second virtual machine, and the QP queue is used for communication between the first virtual machine and the second virtual machine; the host deletes the first instruction and modifies the second instruction into a third instruction, wherein the third instruction is used for indicating that a destination address in the QP queue is a physical address of the second virtual machine; and the host sets the QP queue according to the third instruction so as to send data to the second virtual machine through the set QP queue.
In the embodiment of the invention, when a host receives a first instruction and a second instruction sent by a first virtual machine, the first instruction is used for indicating the host to set a source address in a QP queue as a virtual address of the first virtual machine, the second instruction is used for indicating the host to set a destination address in the QP queue as a virtual address of a second virtual machine, the host deletes the first instruction, modifies the second instruction into a third instruction, and the third instruction is used for indicating the host to set the destination address in the QP queue as a physical address of the second virtual machine; and then setting the attribute of the QP queue according to the modified instruction so as to send data to the second virtual machine based on the set QP queue. Based on the invention, the virtual network and the physical network of the virtual machine can be isolated, the safety of the virtual machine is ensured, and the RoCE protocol can be compatible.
In one possible implementation manner, the first virtual machine sends the first instruction and the second instruction to the host through a first port, where the first port is a port of a virtual remote direct access RDMA network card of the first virtual machine, and the first port corresponds to a global identity GID of the first virtual machine in an RDMA network. This global GID can uniquely identify the first virtual machine, thereby enabling the host to look up the relevant information.
In a possible implementation manner, the modifying the second instruction into a third instruction includes: the host determines a physical address of the second virtual machine according to a first mapping table stored in advance and a virtual address of the second virtual machine in the second instruction, wherein the first mapping table comprises a corresponding relation between the virtual address and the physical address; and the host modifies the virtual address of the second virtual machine in the second instruction into the physical address of the second virtual machine to obtain a third instruction.
In one possible implementation manner, the first virtual machine further includes a port of a virtual ethernet network card, and the method further includes: the first virtual machine acquires the identifier of the virtual RDMA network card and the identifier of the first port; the first virtual machine determines the identifier of a corresponding second port according to the identifier of the virtual RDMA network card, the identifier of the first port and a second mapping table prestored in the host, wherein the second mapping table comprises the corresponding relation among the identifier of the virtual RDMA network card, the identifier of the port of the virtual RDMA network card and the identifier of the port of the virtual Ethernet network card, and the second port is the port of the virtual Ethernet network card of the first virtual machine; the first virtual machine determines a first GID of the first port according to a preset GID generation algorithm and a virtual address corresponding to the identifier of the second port; the first virtual machine sets an address of the first port to the first GID.
In one possible implementation manner, the determining, by the first virtual machine, an identifier of a corresponding second port according to the identifier of the virtual RDMA network card, the identifier of the first port, and a second mapping table pre-stored in the host includes: the first virtual machine sends query request information to the host, wherein the query request information carries the identifier of the virtual RDMA network card and the identifier of the first port; the host determines the identifier of the corresponding second port according to the query request information and a pre-stored second mapping table; and the host sends query response information to the first virtual machine, wherein the query response information carries the identifier of the second port.
In one possible implementation, the method further includes: when the first virtual machine detects information carrying the identifier of the second port and used for changing the Ethernet address, determining the identifier of the corresponding virtual RDMA network card and the identifier of the first port according to the identifier of the second port and a prestored third mapping table, wherein the third mapping table comprises the corresponding relation among the identifier of the port of the virtual Ethernet network card, the identifier of the virtual RDMA network card and the identifier of the port of the virtual RDMA network card; and the first virtual machine updates the address of the first port according to the event type carried in the information for changing the Ethernet address.
In a possible implementation manner, the updating, by the first virtual machine, the address of the first port according to the event type carried in the ethernet address change information includes: when the event type is the network card starting, the first virtual machine acquires a virtual address corresponding to the second port identifier; and the first virtual machine determines a second GID of the first port according to a virtual address corresponding to the second port identifier and a preset GID generation algorithm, and sets the address of the first port as the second GID.
In a possible implementation manner, the updating, by the first virtual machine, the address of the first port according to the event type carried in the ethernet address change information includes: and when the event type is network card closing, the first virtual machine deletes the address of the first port.
In a second aspect, an apparatus is provided, which includes a host configured to obtain a first instruction and a second instruction from a first virtual machine, where the first instruction is configured to indicate that a source address of a queue pair QP queue is a virtual address of the first virtual machine, the second instruction is configured to indicate that a destination address of the QP queue is a virtual address of a second virtual machine, and the QP queue is a QP queue used by the first virtual machine to communicate with the second virtual machine; deleting the first instruction, and modifying the second instruction into a third instruction, wherein the third instruction is used for indicating that a destination address in the QP queue is a physical address of the second virtual machine; and the host sets the QP queue according to the third instruction so as to send data to the second virtual machine through the set QP queue.
In the embodiment of the invention, when a host receives a first instruction and a second instruction sent by a first virtual machine, the first instruction is used for indicating the host to set a source address in a QP queue as a virtual address of the first virtual machine, the second instruction is used for indicating the host to set a destination address in the QP queue as a virtual address of a second virtual machine, the host deletes the first instruction and modifies the second instruction into a third instruction, and the third instruction is used for indicating the host to set the destination address in the QP queue as a physical address of the second virtual machine; and then setting the attribute of the QP queue according to the modified third instruction so as to send data to the second virtual machine based on the set QP queue. Based on the invention, the virtual network and the physical network of the virtual machine can be isolated, the safety of the virtual machine is ensured, and the RoCE protocol can be compatible.
In one possible implementation, the apparatus further includes a first virtual machine, where the first virtual machine is configured to send the first instruction and the second instruction to the host through a first port, where the first port is a port of a virtual remote direct access RDMA network card of the first virtual machine, and the first port corresponds to a global identity GID of the first virtual machine in an RDMA network.
In a possible implementation manner, the host is specifically configured to: determining a physical address of the second virtual machine according to a first mapping table stored in advance and a virtual address of the second virtual machine in the second instruction, wherein the first mapping table comprises a corresponding relation between the virtual address and the physical address; and modifying the virtual address of the second virtual machine in the second instruction into the physical address of the second virtual machine to obtain a third instruction.
In a possible implementation manner, the first virtual machine further includes a port of a virtual ethernet network card, and the first virtual machine is further configured to: acquiring an identifier of the virtual RDMA network card and an identifier of the first port; determining a corresponding identifier of a second port according to the identifier of the virtual RDMA network card, the identifier of the first port and a second mapping table prestored in the host, wherein the second mapping table comprises a corresponding relation among the identifier of the virtual RDMA network card, the identifier of the port of the virtual RDMA network card and the identifier of the port of the virtual Ethernet network card, and the second port is the port of the virtual Ethernet network card of the first virtual machine; determining a first GID of the first port according to a preset GID generation algorithm and a virtual address corresponding to the identifier of the second port; setting an address of the first port to the first GID.
In a possible implementation manner, the first virtual machine is specifically configured to: sending query request information to the host, wherein the query request information carries the identifier of the virtual RDMA network card and the identifier of the first port; the host is specifically configured to: determining the identifier of a corresponding second port according to the query request information and a pre-stored second mapping table; and sending query response information to the first virtual machine, wherein the query response information carries the identifier of the second port.
In a possible implementation manner, the first virtual machine is further configured to: the first virtual machine is further configured to: when the first virtual machine detects information carrying the identifier of the second port and used for changing the Ethernet address, determining the identifier of the corresponding virtual RDMA network card and the identifier of the first port according to the identifier of the second port and a prestored third mapping table, wherein the third mapping table comprises the corresponding relation among the identifier of the port of the virtual Ethernet network card, the identifier of the virtual RDMA network card and the identifier of the port of the virtual RDMA network card; and the first virtual machine updates the address of the first port according to the event type carried in the information for changing the Ethernet address.
In a possible implementation manner, the first virtual machine is specifically configured to: when the event type is the starting of the network card, acquiring a virtual address corresponding to the second port identifier; and determining a second GID of the first port according to the virtual address and a preset GID generation algorithm, and setting the address of the first port as the second GID.
In a possible implementation manner, the first virtual machine is specifically configured to: and when the event type is the network card closing, deleting the address of the first port.
In a third aspect, an apparatus is provided, the apparatus comprising: the apparatus comprises: a memory, a processing circuit and a host channel adapter HCA card; wherein the memory is used for storing codes; the processing circuit is used for operating a first virtual machine and a host according to the codes stored in the memory; the processing circuit is further configured to implement the method of the first aspect by the first virtual machine and a host; the HCA card is used for sending data to the second virtual machine.
In a fourth aspect, there is provided a computer-readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the method of the first aspect.
In a fifth aspect, there is provided a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of the first aspect.
In the embodiment of the invention, when a host receives a first instruction and a second instruction sent by a first virtual machine, the first instruction is used for indicating the host to set a source address in a QP queue as a virtual address of the first virtual machine, the second instruction is used for indicating the host to set a destination address in the QP queue as a virtual address of a second virtual machine, the host deletes the first instruction and modifies the second instruction into a third instruction, and the third instruction is used for indicating the host to set the destination address in the QP queue as a physical address of the second virtual machine; and then setting the attribute of the QP queue according to the modified third instruction so that the HCA card sends data to the second virtual machine based on the set QP queue. Based on the invention, the virtual network and the physical network of the virtual machine can be isolated, the safety of the virtual machine is ensured, and the RoCE protocol can be compatible.
Drawings
FIG. 1 is a flow chart of data transmission in a TCP/IP network according to an embodiment of the present invention;
FIG. 2 is a flow diagram of data transfer in an RDMA network according to an embodiment of the present invention;
fig. 3 is an architecture diagram of a RoCE protocol provided in an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of an apparatus according to an embodiment of the present invention;
FIG. 5 is a schematic structural diagram of an apparatus according to an embodiment of the present invention;
fig. 6 is a flowchart of a method for associating a port of a virtual ethernet network card with a port of a virtual RDMA network card according to an embodiment of the present invention;
FIG. 7 is a flowchart illustrating an example of a method for associating a port of a virtual Ethernet network card with a port of a virtual RDMA network card according to the present invention;
fig. 8 is a flowchart of a QP attribute setting information management method according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of an apparatus according to an embodiment of the present invention.
Detailed Description
Fig. 1 is a flow chart of data transmission in a TCP/IP network according to an embodiment of the present invention, as shown in fig. 1, in a conventional TCP/IP network, first, a first server needs to copy data from a buffer of an application to a buffer of a TCP/IP protocol stack; then, packaging the data into a data packet according to a TCP/IP protocol, and copying the data packet to a buffer area driven by the network card; and finally, transmitting the data packet to a second server through the network card. Correspondingly, in the conventional TCP/IP network, the second server copies the data packet in the buffer area driven by the network card to the buffer area of the TCP/IP protocol stack, decapsulates the data packet according to the TCP/IP protocol to obtain data, and copies the data from the buffer area of the TCP/IP protocol stack to the buffer area of the application program.
This causes two problems, i.e., a large number of data copies and a large communication delay. On the other hand, the protocol stack needs a Central Processing Unit (CPU) to operate both for encapsulating and decapsulating data, which results in a high load on the CPU. For example, in a search application, each query results in several query requests to the back-end server, and these query requests have deadline requirements. Only query responses returned to the search application before the expiration time will have their results considered, while query responses that exceed the expiration time will be discarded by the search application. Thus, when the delay in a TCP/IP network is large, the proportion of the query response that is dropped will increase significantly. On the one hand, it will result in a reduced search quality of the search application; on the other hand, the large number of invalid query responses transmitted in the TCP/IP network further increases network delay. As another example, a higher CPU load may result in a reduction in the computing power that the server may provide. In cloud computing, a CPU is a core scheduling resource, and a high CPU load directly causes a decrease in the number of tenants that can be served, thereby reducing the profitability of the overall system.
In order to solve the problems of large delay and high CPU load in a TCP/IP network, the RDMA protocol is applied, and compared with the TCP/IP protocol, the RDMA protocol has the advantages of low delay and small CPU load. Fig. 2 is a flowchart of data transmission in an RDMA network according to an embodiment of the present invention, as shown in fig. 2, in the RDMA network, first, an application registers a memory in an operating system of a server before performing data transmission, and sends an address of the registered memory to an HCA card of the server, and then, the HCA card of the server can directly read and write the registered memory of the application. In this way, the transmission and reception of data no longer requires multiple copies between buffers, and the delay can be reduced considerably. Meanwhile, the HCA card also realizes the functions of an RDMA protocol stack, namely the HCA card completes the encapsulation and decapsulation processing of the data packet. Therefore, CPU resources are released, the load of the CPU is reduced, and the server can have more CPU resources allocated to the tenants for use.
The RDMA protocol is implemented based on an Infiniband (IB) network architecture. However, in the cloud computing scenario, the TCP/IP protocol remains the most dominant network protocol. Therefore, in order to be compatible with the TCP/IP protocol, the RoCE protocol is proposed in the industry. Currently, the RoCE technology mainly includes two versions, RoCEv1 and RoCEv 2. Fig. 3 is an architecture diagram of the RoCE protocol according to an embodiment of the present invention, and as shown in fig. 3, a link layer protocol of the TCP/IP protocol in the RoCEv1 version replaces a link layer protocol of the IB protocol. In the RoCEv2 version, the IB Protocol is replaced with a User Datagram Protocol (UDP) in the network layer Protocol of the TCP/IP Protocol and the transport layer Protocol. Specifically, the network layer protocol of the TCP/IP protocol is used for route learning and transmission, and the UDP protocol of the TCP/IP protocol can be used for equivalent routing (ECMP). Currently, the RoCEv2 version is commonly adopted in cloud computing networks.
In the prior art, when the virtual machines in different servers need to perform data communication, a hardware virtualization function, such as SR-IOV, needs to be started on the HCA card. In this way, the virtual machine can establish connection with the other side according to the physical IP address configured by the administrator, establish a Queue Pair (QP) on the corresponding RDMA interface, and set the source address of the QP Queue as the physical IP address of the virtual machine itself and the destination address as the physical IP address of the other side. In the subsequent data transmission process, the HCA card directly reads the data in the registered memory corresponding to the application program, packages the data according to the source address and the destination address in the QP queue to obtain a data message, and transmits the data message through a physical network.
In a conventional TCP/IP protocol, an administrator generally sets a virtual IP address in a virtual machine in order to isolate a virtual network of the virtual machine from a physical network and ensure the security of the virtual machine. However, based on the prior art, after the virtual machine sets the virtual IP address, when the virtual machine establishes a connection with the peer virtual machine, the virtual machine sets the source address in the QP queue as the virtual IP address of the virtual machine, and sets the destination address as the virtual IP address of the peer virtual machine, and when the subsequent HCA card encapsulates the data of the virtual machine according to the QP queue, the source address of the data packet is the virtual IP address of the virtual machine, and the destination address is the virtual IP address of the peer virtual machine. Since there is no routing table entry related to the virtual IP address in the network device of the physical network, the data packet cannot be transmitted.
The embodiment of the invention provides a method for sending data in a cloud computing scene, which is applied to equipment. In this scenario, as shown in fig. 4, a Virtual Machine 410, a host 420, and an HCA card 430 supporting a RoCE or other similar protocols may be disposed in the device, where the Virtual Machine 410 is connected to the host 420 and the HCA card 430, the host 420 is connected to the HCA card 430, and the host 420 may be a Virtual Machine with a management function or multiple implementation manners such as a Virtual Machine Monitor (VMM) or a Hypervisor, which is not limited in the present invention. Specifically, as shown in fig. 5, the virtual machine 410 includes a virtual ethernet network card 411, a virtual RDMA network card 412, a virtual ethernet management module 413, and a virtual RDMA management module 414. The host 420 includes a virtual RDMA control module 421 and may also include a virtual ethernet control module 422 and a controller 423.
The HCA card 430 includes a plurality of physical ports, and the administrator may virtualize the HCA card 430, and for each physical port on the HCA card 430, the administrator may virtualize the physical port into one or more groups of virtual RoCE network cards, where each group of virtual RoCE network cards may include one virtual ethernet card 411 and one virtual RDMA network card 412. The invention is described by taking an example that one port is virtualized into a group of virtual RoCE network cards, and other situations are similar to the above. The virtual ethernet card 411 includes a plurality of ports of the virtual ethernet card, and the virtual RDMA card 412 includes a plurality of ports of the virtual RDMA card.
The virtual machine 410 is connected to a virtual ethernet network card 411 and a virtual RDMA network card 412, respectively, the virtual ethernet network card 411 is connected to a port of the corresponding HCA card 430 through a virtual ethernet control module 422, the virtual RDMA network card 412 is connected to a port of the corresponding HCA card 430 through a virtual RDMA control module 421, and meanwhile, the virtual RDMA control module 421 is connected to the controller 423.
A virtual RDMA management module 414 for registering and managing the virtual RDMA network card 412, a virtual ethernet management module 413 for registering and managing the virtual ethernet network card 411, a virtual ethernet control module 422 for controlling the ethernet protocol packet sent by the virtual machine 410, a virtual RDMA control module 421 for controlling the QP attribute setting information sent by the virtual machine 410, and a controller 423 for storing the correspondence among the identifier of the virtual RDMA network card, the identifier of the port of the virtual RDMA network card, and the identifier of the port of the virtual ethernet network card, and the correspondence between the virtual address and the physical address of the peer virtual machine. The virtual address of the virtual machine may be a virtual IP address, or a virtual Media Access Control (MAC) address, and similarly, the physical address may be a physical IP address, or a physical MAC address.
In the prior art, from the perspective of an operating system, an HCA network card is an HCA device, and an HCA device includes an ethernet device and an RDMA device. For an HCA device, the HCA driver first creates a base device in the operating system. And abstracts an ethernet device and an RDMA device on the base device. In the RDMA communication process, an IP address is set on the Ethernet equipment for establishing connection, and the RDMA equipment is used for reading and writing data. Since an application specifies only an IP address when performing an RDMA operation, the operating system must provide a method for determining an RDMA device by an IP address.
In the HCA card, because the ethernet device and the RDMA device are both mounted on the same basic device, the HCA driver can automatically determine the GID corresponding to the RDMA device according to the IP address of the ethernet device and a pre-stored GID generation algorithm, and set the address of the RDMA device as the GID. Meanwhile, when the IP address of the ethernet device changes, the HCA driver may automatically update the GID of the RDMA device according to the IP address of the ethernet device. In this way, the application program can determine the corresponding RDMA device according to its own IP address and the pre-stored GID generation algorithm, thereby implementing data transmission. However, for the virtual ethernet network card and the virtual RDMA network card, both are independent devices in the operating system, without any association. Therefore, for convenience of description, the embodiment of the present invention first introduces a method for providing a port of a virtual ethernet network card and an association method of a port of a virtual RDMA network card corresponding to a virtual machine, as shown in fig. 5 and fig. 6, the specific steps may be as follows:
in step 601, the first virtual machine 410 obtains the identifier of the preset virtual RDMA network card 412 and the identifier of the first port.
In an implementation, the administrator may pre-store the identification of the virtual RDMA network card 412 in the virtual RDMA management module 414 of the virtual machine in order to manage the virtual RDMA network card 412 through the virtual RDMA management module 414. When the virtual RDMA management module 414 detects that a certain virtual RDMA network card 412 is started, the virtual RDMA management module 414 may obtain a pre-stored identifier of the virtual RDMA network card 412, and then send a registration message to an RDMA network card driver module (such as an ib-core module), where the registration message carries the identifier of the virtual RDMA network card 412, so as to register the virtual RDMA network card 412. Meanwhile, the registration message may also carry a link layer type, where the link layer type may be RDMA, and after receiving the registration message, the RDMA network card driver module may stop managing the GIDs of the ports of the virtual RDMA network cards on the virtual RDMA network card 412. For example, the virtual RDMA management module 414 calls the ib _ register _ device () interface to register the virtual RDMA network card 412 with the ib _ core module. Wherein, the virtual RDMA management module 414 specifies the parameter link layer type in the IB _ register _ device () interface as "IB" so that the IB _ core stops managing the gib _ cache of the first port of the virtual RDMA network card 412.
After the virtual RDMA management module 414 completes the registration of the virtual RDMA network card 412, the virtual RDMA management module 414 may scan in the bus, and query the identification of the first port on the virtual RDMA network card 412, where the identification of the first port may be a port number of the first port.
In step 602, the first virtual machine 410 determines the identifier of the corresponding second port according to the identifier of the virtual RDMA network card 412, the identifier of the first port, and a second mapping table pre-stored in the host 420.
The second mapping table includes a correspondence relationship among an identifier of the virtual RDMA network card 412, an identifier of a port of the virtual RDMA network card, and an identifier of a port of the virtual ethernet network card, and the second port is the port of the virtual ethernet network card of the first virtual machine.
In an implementation, the host 420 may be provided with a virtual RDMA control module 421, and the virtual RDMA control module 421 may be stored with a second mapping table in advance. The second mapping table may be preset by an administrator, and when the administrator virtualizes the physical ports of the HCA into the virtual ethernet network card 411 and the virtual RDMA network card 412, the administrator may store the correspondence between the identifier of the virtual RDMA network card 412, the identifiers of the ports of the virtual RDMA network cards on the virtual RDMA network card 412, and the identifiers of the ports of the corresponding virtual ethernet network cards in the second mapping table. The identifier of the port of the virtual ethernet network card may be a Media Access Control (MAC) address of the port of the virtual ethernet network card.
After acquiring the identifier of the virtual RDMA network card 412 and the identifier of the first port, the virtual RDMA management module 414 may query the identifier of the second port corresponding to the first port through the virtual RDMA control module 421. The specific steps can be as follows:
in step one, the first virtual machine 410 sends query request information to the host 420.
The query request may carry an identifier of the virtual RDMA network card 412 and an identifier of the first port.
In step two, the host 420 determines the identifier of the corresponding second port according to the query request information and a second mapping table stored in advance.
In implementation, after receiving the query request information, the virtual RDMA control module 421 analyzes the query request information to obtain the identifier of the virtual RDMA network card 412 and the identifier of the first port, which are carried in the query request information, and then queries the identifier of the corresponding second port (i.e., the identifier of the second port corresponding to the first port) in the second mapping table according to the identifier of the virtual RDMA network card 412 and the identifier of the first port.
Step three, the host 420 sends the query response information to the first virtual machine 410.
In implementation, after querying the identifier of the second port corresponding to the first port, the virtual RDMA control module 421 may send query response information to the virtual RDMA management module 414, where the query response information carries the identifier of the second port.
It should be noted that, in order to improve the query efficiency, the administrator may only cache the entry of the second mapping table that has been queried in the virtual RDMA control module 421, and store the complete second mapping table in the controller 423 of the host 420. The virtual RDMA control module 421 receives the query request information (for convenience of distinction, hereinafter referred to as first query request information) sent by the virtual RDMA management module 414, obtains the identifier of the virtual RDMA network card 412 and the identifier of the first port, and then locally queries whether the identifier of the second port corresponding to the first port exists. If yes, send a query response message to the virtual RDMA management module 414, where the query response message carries the identifier of the second port (for convenience of differentiation, hereinafter referred to as a first query response message); otherwise, the virtual RDMA control module 421 sends query request information (hereinafter referred to as second query request information for convenience of distinction) to the controller 423. The controller 423 receives the second query request information sent by the virtual RDMA control module 421, obtains the identifier of the virtual RDMA network card 412 and the identifier of the first port, then queries whether the identifier of the second port corresponding to the first port exists in a second mapping table stored in advance, and if so, sends a query response information to the virtual RDMA control module 421, where the query response information carries the identifier of the second port (for convenience of distinguishing, hereinafter referred to as second query response information); otherwise, query failure information is returned to the virtual RDMA control module 421. When the virtual RDMA control module 421 receives the second query response message, the identifier of the second port is obtained, and the first query response message is sent to the virtual RDMA management module 414. Meanwhile, the virtual RDMA management module 414 stores the correspondence between the identifier of the virtual RDMA network card 412, the identifier of the first port, and the identifier of the second port locally, so as to provide a fast query in the following. When the virtual RDMA control module 421 receives the query failure information, the query failure information is returned to the virtual RDMA management module 414, and after receiving the query failure information, the virtual RDMA management module 414 may notify the administrator in a display or a short message manner, so that the administrator may modify the error in time.
Step 603, the first virtual machine 410 determines the first GID of the first port according to a preset GID generation algorithm and a virtual IP address corresponding to the identifier of the second port.
In implementation, the virtual RDMA management module 414 stores a GID generation algorithm in advance. The GID generation algorithm can adopt an industry standard GID conversion method. For example, if the address length of the GID is 128 bits, if the IPv4 address is converted into the GID, since the address length of the IPv4 is 32 bits, the IPv4 address can be used as the lower 32 bits of the GID, and the upper 96 bits of the GID are all complemented by 0; if the IPv6 address is converted into the GID, since the address length of the IPv6 is 128 bits, the IPv6 address can be directly used as the GID.
After receiving the query response message, the virtual RDMA management module 414 parses the query response message to obtain the identifier of the second port corresponding to the first port. The virtual RDMA management module 414 may scan in the bus according to the identifier of the second port, query the corresponding port of the virtual ethernet network card and the virtual IP address pre-configured on the port of the virtual ethernet network card, and obtain the first GID corresponding to the first port based on the virtual IP address and the pre-stored GID generation algorithm.
In step 604, the first virtual machine 410 sets the address of the first port to the first GID.
In an implementation, after the virtual RDMA management module 414 determines the first GID of the first port, the address of the first port may be set to the first GID.
In this embodiment of the present invention, when the administrator modifies the virtual IP address of the second port, the virtual RDMA management module 414 may further perform corresponding configuration on the GID of the first port, and the specific steps may be as follows:
step one, when the first virtual machine 410 detects the ethernet address change information carrying the identifier of the second port, the identifier of the corresponding virtual RDMA network card 412 and the identifier of the first port are determined according to the identifier of the second port and a pre-stored third mapping table.
The third mapping table includes a correspondence relationship among the identifier of the port of the virtual ethernet network card 411, the identifier of the virtual RDMA network card 412, and the identifier of the port of the virtual RDMA network card 412.
In an implementation, a third mapping table may be stored in the virtual RDMA management module 414, and the third mapping table may be set by the virtual RDMA management module 414. The virtual RDMA management module 414 receives the query response information sent by the virtual RDMA control module 421, analyzes the query response information, and obtains the identifier of the second port, and the virtual RDMA management module 414 may store the correspondence between the identifier of the virtual RDMA network card 412, the identifier of the first port, and the identifier of the second port in the third mapping table.
For real-time monitoring of IP address change of the second port, the virtual RDMA management module 414 may register an ethernet address change event with the operating system of the virtual machine, so that when the administrator modifies the virtual IP address on the second port, the virtual RDMA management module 414 may configure the GID of the first port accordingly. For example, virtual RDMA management module 414 calls the register _ inetaddr _ notifier () interface to register a callback function for an ethernet address change event in the operating system.
When the operating system of the virtual machine detects a management instruction corresponding to a virtual IP address, the ethernet address change event is triggered, where the management instruction may be a delete instruction or a configuration instruction.
When the operating system detects a delete instruction for deleting the virtual IP address of the second port by the administrator, the operating system may determine that the type of the ethernet address change event is network card shutdown. When the operating system detects a configuration instruction for configuring the virtual IP address of the port of the virtual ethernet network card by the administrator, the operating system may determine that the type of the ethernet address change event is network card startup. The operating system may send ethernet address change event information to the virtual RDMA management module 414, where the ethernet address change event information carries an identifier of the determined type of the ethernet address change event.
The os sends ethernet address change information to the RDMA management module 414, where the ethernet address change information carries the identifier of the port of the virtual ethernet network card and the type of the ethernet address change event.
The virtual RDMA management module 414 receives the ethernet address change information, parses the ethernet address change information to obtain the identifier of the port of the virtual ethernet network card, and then queries whether the identifier of the virtual RDMA network card 412 and the identifier of the port of the virtual RDMA network card corresponding to the identifier of the port of the virtual ethernet network card exist in the third mapping table. If yes, executing step two, otherwise, not executing other processing.
Step two, the first virtual machine 410 configures the GID of the first port according to the event type carried in the ethernet address change information.
In implementation, after obtaining the event type carried in the ethernet address change information, the virtual RDMA management module 414 may correspondingly update the GID of the first port according to the event type.
Optionally, the virtual RDMA management module 414, according to the type of the ethernet address change event, specifically divides the configuration update of the GID of the first port into the following two cases:
in the first case: when the event type is network card shutdown, the first virtual machine 410 deletes the address of the first port.
In the second case: when the event type is network card starting, the first virtual machine 410 acquires a virtual IP address corresponding to the second port identifier, and the first virtual machine 410 determines the second GID of the first port according to the virtual IP address and a preset GID generation algorithm, and sets the first port address as the second GID.
In implementation, after receiving the ethernet address change event information, the virtual RDMA management module 414 obtains an identifier of an event type, and if the identifier is used to indicate that the network card is closed, deletes the address of the first port corresponding to the port of the virtual ethernet network card.
If the identifier is used to indicate that the network card is started, the virtual RDMA management module 414 may query, in the bus, the virtual IP address configured on the second port according to the identifier of the second port, and then determine, according to the virtual IP address and a pre-stored GID generation algorithm, the second GID of the first port corresponding to the port of the virtual ethernet network card. After the virtual RDMA management module 414 determines the second GID for the first port, the address of the first port may be set to the second GID.
The present invention further provides an example of a method for associating a port of a virtual ethernet network card with a port of a virtual RDMA network card, as shown in fig. 5 and 7, the specific processing procedures are as follows:
in step 701, after the virtual RDMA network card 412 is started, the virtual RDMA management module 414 sends a registration message to the RDMA network card driver module to register the virtual RDMA network card 412, where the registration message carries the identifier of the virtual RDMA network card 412.
In step 702, the RDMA network card driver module registers the virtual RDMA network card 412 and stops managing the GIDs of the ports of the virtual RDMA network cards on the virtual RDMA network card 412.
In step 703, the virtual RDMA management module 414 obtains the identifier of the virtual RDMA network card 412 and the identifier of the first port.
In step 704, the virtual RDMA management module 414 sends a query request message to the virtual RDMA control module 421, where the query request message carries the identifier of the virtual RDMA network card 412 and the identifier of the first port.
Step 705, the virtual RDMA control module 421 returns a query response message to the virtual RDMA management module 414, where the query response message carries the identifier of the second port.
In step 706, the virtual RDMA management module 414 queries whether the second port is configured with a virtual IP address, if so, then step 707 and 708 are executed, otherwise, step 709 is executed.
In step 707, the virtual RDMA management module 414 determines the first GID of the first port according to the preset GID generation algorithm and the virtual IP address corresponding to the identifier of the second port.
At step 708, virtual RDMA management module 414 sets the address of the first port to the first GID.
In step 709, virtual RDMA management module 414 registers the Ethernet Address Change event with the operating System.
Step 710, when the virtual RDMA management module 414 detects the ethernet address change information, determining the identifier of the corresponding virtual RDMA network card 412 and the identifier of the first port according to the identifier of the second port and a third mapping table stored in advance.
In step 711, the virtual RDMA management module 414 obtains the event type carried in the ethernet address change information, if the event type is network card start, step 712 and 713 are executed, and if the event type is network card close, step 714 is executed.
In step 712, the virtual RDMA management module 414 obtains the virtual IP address corresponding to the second port identifier, and determines the second GID of the first port according to the virtual IP address and the preset GID generation algorithm.
In step 713, virtual RDMA management module 414 sets the first port address to the second GID.
In step 714, virtual RDMA management module 414 deletes the address of the first port.
The specific processing procedures from step 701 to step 714 can refer to steps 601 to 604, which are not described in detail herein.
In this embodiment of the present invention, after the virtual RDMA management module 414 determines the GID of the first port of the virtual RDMA network card 412, the virtual RDMA control module 421 may manage the QP attribute setting information sent by the virtual machine, as shown in fig. 5 and 8, the steps may be as follows:
in step 801, the first virtual machine 410 determines the global identity GID of the RDMA network corresponding to the first virtual machine 410 according to the virtual address of the first virtual machine 410.
In implementation, when an application in the first virtual machine 410 needs to perform data transmission with an application in the second virtual machine, the application in the first virtual machine 410 may determine the GID of the corresponding first port according to the local virtual IP address and the pre-stored GID generation algorithm.
Step 802, the first virtual machine 410 sends QP attribute setting information to the host 420 through a first port corresponding to the GID, where the first port is a port of the virtual RDMA network card of the first virtual machine, the QP attribute setting information includes a first instruction and a second instruction, the first instruction is used to instruct the host 420 to set a source address in a QP queue as a virtual address of the first virtual machine 410, the second instruction is used to instruct the host 420 to set a destination address in the QP queue as a virtual address of the second virtual machine, and the QP queue is a QP queue used for communication between the first virtual machine and the second virtual machine in the QP queue corresponding to the HCA card.
In implementation, the application in the first virtual machine 410 may send QP attribute setting information to the virtual RDMA control module 421 through the first port corresponding to the GID. Wherein the QP attribute setting information includes a first instruction and a second instruction, wherein the first instruction may be used to indicate that the source address is set to the virtual IP address of the first virtual machine 410, and the second instruction may be used to indicate that the destination address is set to the virtual IP address of the second virtual machine. The virtual IP address of the second virtual machine may be set in advance by an administrator in the application of the first virtual machine 410. For example, the first virtual machine 410 sends QP attribute setting information to the host 420 through an ib _ modify _ QP (QP _ attr) interface, where QP _ attr specifies a source address as the virtual IP address of the first virtual machine 410 and a destination address as the virtual IP address of the second virtual machine.
In step 803, the host 420 deletes the first instruction, and modifies the second instruction into a third instruction, where the third instruction is used to instruct the host 420 to set the destination address in the QP queue as the physical address of the second virtual machine.
In implementation, if the source address is the virtual IP address of the first virtual machine 410 and the destination address is the virtual IP address of the second virtual machine in the attributes of the QP queue, when the subsequent HCA card 430 performs protocol encapsulation on data sent by the application program of the first virtual machine 410 to the application program of the second virtual machine, the virtual IP address of the first virtual machine 410 is used as the source address and the virtual IP address of the second virtual machine is used as the destination address, and then the data is transmitted through the physical network between the first virtual machine 410 and the second virtual machine. Since the physical network does not have the routing entry of the virtual IP address corresponding to the first virtual machine 410 and the second virtual machine, but only has the routing entry of the physical IP address corresponding to the first virtual machine 410 and the second virtual machine, after receiving the QP attribute setting information, the virtual RDMA control module 421 needs to delete the first instruction, and at the same time, modify the virtual IP address corresponding to the second virtual machine in the second instruction to the physical IP address corresponding to the second virtual machine, so as to obtain the third instruction. The third instruction is to instruct to set the destination address to a physical IP address of the second virtual machine.
Optionally, the second instruction is modified into a third instruction, and the steps may be as follows:
in step one, the host 420 determines a physical IP address of the second virtual machine according to the pre-stored virtual IP address of the second virtual machine in the first mapping table and the second instruction.
The first mapping table comprises a corresponding relation between a virtual IP address and a physical IP address.
In implementation, the virtual RDMA control module 421 may store a first mapping table in advance, and the first mapping table may be set by an administrator in advance.
After receiving the QP attribute setting information, the virtual RDMA control module 421 parses the QP attribute setting information to obtain the virtual IP address of the second virtual machine in the second instruction, and then queries the physical IP address corresponding to the virtual IP address of the second virtual machine in the first mapping table.
It should be noted that, in order to improve the query efficiency, the administrator may only cache the entry of the first mapping table that has been queried in the virtual RDMA control module 421, and store the complete first mapping table in the controller 423 of the host 420. The virtual RDMA control module 421 receives the QP attribute setting information, obtains the virtual IP address of the second virtual machine in the second instruction, and then locally queries whether a physical IP address corresponding to the virtual IP address of the second virtual machine exists. If yes, executing the step two; otherwise, the virtual RDMA control module 421 sends an address query request message to the controller 423, where the address query request message carries the virtual IP address of the second virtual machine. The controller 423 receives the address query request information sent by the virtual RDMA control module 421 to obtain the virtual IP address of the second virtual machine, and then the controller 423 queries whether a physical IP address corresponding to the virtual IP address of the second virtual machine exists in a pre-stored first mapping table, if so, sends address query response information to the virtual RDMA control module 421, where the address query response information carries the physical IP address; otherwise, address query failure information is returned to the virtual RDMA control module 421. When the virtual RDMA control module 421 receives the address query response message, the physical IP address is obtained, then step two is executed, and the corresponding relationship between the virtual IP address of the second virtual machine and the physical IP address is stored locally, so as to provide fast query in the following. When the virtual RDMA control module 421 receives the address query failure information, it may notify the administrator through a display or a short message, so that the administrator can modify the error in time.
Step two, the host 420 modifies the virtual IP address of the second virtual machine in the second instruction into the physical IP address of the second virtual machine, and obtains a third instruction.
In implementation, after the virtual RDMA control module 421 determines the physical IP address corresponding to the virtual IP address of the second virtual machine, the virtual IP address of the second virtual machine in the second instruction may be deleted, and then the destination address in the second instruction is set as the physical IP address of the second virtual machine, so as to obtain the third instruction.
In step 804, the host 420 sets the attribute of the QP queue according to the modified QP attribute setting information, so that the HCA card 430 sends data to the second virtual machine based on the set QP queue.
In implementation, the virtual RDMA control module 421 sets the destination address in the QP queue to the physical IP address of the second virtual machine according to the modified QP attribute setting information. Meanwhile, the virtual RDMA control module 421 scans the bus, queries the physical IP address configured on the corresponding ethernet interface, and then sets the source address as the physical IP address of the ethernet interface. Subsequently, when the application program in the first virtual machine 410 sends data to the application program in the second virtual machine, the RDMA interface may read data from the memory corresponding to the application program of the first virtual machine 410, encapsulate the data with the physical IP address of the ethernet interface as a source address and the physical IP address of the second virtual machine as a destination address, obtain a data packet, and then send the data packet to the port of the HCA card 430 corresponding to the second virtual machine through the physical network between the first virtual machine 410 and the second virtual machine.
In the embodiment of the present invention, when the host 420 receives QP attribute setting information sent by the first virtual machine 410, the QP attribute setting information includes a first instruction and a second instruction, the first instruction is used to instruct the host 420 to set a source address in a QP queue as a virtual address of the first virtual machine 410, the second instruction is used to instruct the host 420 to set a destination address in the QP queue as a virtual address of the second virtual machine, the host 420 deletes the first instruction and modifies the second instruction into a third instruction, and the third instruction is used to instruct the host 420 to set the destination address in the QP queue as a physical address of the second virtual machine; and then setting the attribute of the QP queue according to the modified QP attribute setting information, so that the HCA card 430 sends data to the second virtual machine based on the set QP queue. Based on the invention, the virtual network and the physical network of the virtual machine can be isolated, the safety of the virtual machine is ensured, and the RoCE protocol can be compatible.
Based on the same technical concept, an embodiment of the present invention further provides an apparatus, as shown in fig. 5, the apparatus includes a first virtual machine 410, a host 420, and a host channel adapter HCA card 430, where the first virtual machine 410 includes a virtual ethernet card 411 and a virtual RDMA card 412, the virtual ethernet card 411 includes a port of the virtual ethernet card, and the virtual RDMA card 412 includes a port of the virtual RDMA card; wherein: the first virtual machine 410 is configured to determine, according to a virtual internet protocol address of the first virtual machine 410, a global identity GID of an RDMA network corresponding to the first virtual machine 410 when a data connection needs to be established with a second virtual machine; the first virtual machine 410 is further configured to send QP attribute setting information to the host 420 through the first port corresponding to the GID, where the QP attribute setting information includes a first instruction and a second instruction, the first instruction is used to instruct the host 420 to set a source address in a QP queue as a virtual address of the first virtual machine 410, and the second instruction is used to instruct the host 420 to set a destination address in the QP queue as a virtual address of the second virtual machine; the host 420 is used for deleting the first instruction and modifying the second instruction into a third instruction, wherein the third instruction is used for indicating the host 420 to set a destination address in the QP queue as a physical address of the second virtual machine; and the host 420 is configured to set an attribute of the QP queue according to the modified QP attribute setting information, so that the HCA card 430 sends data to the second virtual machine based on the set QP queue.
In a possible implementation manner, the host 420 is specifically configured to: determining a physical address of a second virtual machine according to a first mapping table stored in advance and the virtual address of the second virtual machine in a second instruction, wherein the first mapping table comprises a corresponding relation between the virtual address and the physical address; and modifying the virtual address of the second virtual machine in the second instruction into the physical address of the second virtual machine to obtain a third instruction.
In one possible implementation, the first virtual machine 410 is further configured to: acquiring an identifier of a preset virtual RDMA network card 412 and an identifier of a first port; determining the corresponding identifier of the second port according to the identifier of the virtual RDMA network card 412, the identifier of the first port and a second mapping table pre-stored in the host 420, wherein the second mapping table comprises the corresponding relationship among the identifier of the virtual RDMA network card 412, the identifier of the port of the virtual RDMA network card and the identifier of the port of the virtual Ethernet network card; determining a first GID of the first port according to a preset GID generation algorithm and a virtual address corresponding to the identifier of the second port;
the address of the first port is set to the first GID.
In a possible implementation manner, the first virtual machine 410 is specifically configured to: sending query request information to the host 420, where the query request carries the identifier of the virtual RDMA network card 412 and the identifier of the first port; the host 420 is specifically configured to: determining the identifier of the corresponding second port according to the query request information and a pre-stored second mapping table; and sending query response information to the first virtual machine 410, wherein the query response information carries the identifier of the second port.
In one possible implementation, the first virtual machine 410 is further configured to: when detecting the change information of the ethernet address carrying the identifier of the second port, determining the identifier of the corresponding virtual RDMA network card 412 and the identifier of the first port according to the identifier of the second port and a pre-stored third mapping table, where the third mapping table includes the correspondence between the identifier of the port of the virtual ethernet network card 411, the identifier of the virtual RDMA network card 412 and the identifier of the port of the virtual RDMA network card 412; and updating the address of the first port according to the event type carried in the Ethernet address change information.
In a possible implementation manner, the first virtual machine 410 is specifically configured to: when the event type is the network card starting, acquiring a virtual address corresponding to the second port identification; and determining a second GID of the first port according to the virtual address and a preset GID generation algorithm, and setting the address of the first port as the second GID.
In a possible implementation manner, the first virtual machine 410 is specifically configured to: and when the event type is the network card closing, deleting the address of the first port.
In the embodiment of the present invention, when the host 420 receives QP attribute setting information sent by the first virtual machine 410, the QP attribute setting information includes a first instruction and a second instruction, the first instruction is used to instruct the host 420 to set a source address in a QP queue as a virtual address of the first virtual machine 410, the second instruction is used to instruct the host 420 to set a destination address in the QP queue as a virtual address of the second virtual machine, the host 420 deletes the first instruction and modifies the second instruction into a third instruction, and the third instruction is used to instruct the host 420 to set the destination address in the QP queue as a physical address of the second virtual machine; and then setting the attribute of the QP queue according to the modified QP attribute setting information, so that the HCA card 430 sends data to the second virtual machine based on the set QP queue. Based on the invention, the virtual network and the physical network of the virtual machine can be isolated, the safety of the virtual machine is ensured, and the RoCE protocol can be compatible.
Based on the same technical concept, an embodiment of the present invention further provides an apparatus, as shown in fig. 9, where the apparatus 900 includes: memory 910, processing circuitry 920, and host channel adapter HCA card 930. The apparatus 900 can perform any one of the above methods described in the present application, and the system architecture, noun explanation, beneficial effect, etc. involved in any one of the above methods, please refer to the related paragraphs, which are not described herein again.
The memory 910 is used for storing codes.
The processing circuit 920 is configured to run a first virtual machine and a host according to the code stored in the memory 910.
The HCA card 930 is configured to send data to the second virtual machine.
The processing circuit 920 is further configured to implement, by the first virtual machine and the host, the following method:
the first virtual machine includes a port of a virtual remote direct data access, RDMA, network card, the method comprising:
the first virtual machine determines a Global Identity (GID) of an RDMA network corresponding to the first virtual machine according to the virtual address of the first virtual machine;
the first virtual machine sends queue pair QP attribute setting information to the host through a first port corresponding to the GID, wherein the first port is a port of the virtual RDMA network card of the first virtual machine, the QP attribute setting information comprises a first instruction and a second instruction, the first instruction is used for indicating the host to set a source address in a QP queue as a virtual address of the first virtual machine, the second instruction is used for indicating the host to set a destination address in the QP queue as a virtual address of the second virtual machine, the QP queue is a QP queue used for communication between the first virtual machine and the second virtual machine, and the destination address in the QP queue is a virtual address of the second virtual machine;
the host deletes the first instruction and modifies the second instruction into a third instruction, wherein the third instruction is used for indicating the host to set a destination address in a QP queue as a physical address of the second virtual machine;
and the host sets the attribute of the QP queue according to the modified QP attribute setting information so that the HCA card sends data to the second virtual machine based on the set QP queue.
In a possible implementation manner, the modifying the second instruction into a third instruction includes:
the host determines a physical address of the second virtual machine according to a first mapping table stored in advance and a virtual address of the second virtual machine in the second instruction, wherein the first mapping table comprises a corresponding relation between the virtual address and the physical address;
and the host modifies the virtual address of the second virtual machine in the second instruction into the physical address of the second virtual machine to obtain a third instruction.
In one possible implementation manner, the first virtual machine further includes a port of a virtual ethernet network card, and the method further includes:
the first virtual machine acquires a preset identifier of the virtual RDMA network card and an identifier of the first port;
the first virtual machine determines the identifier of a corresponding second port according to the identifier of the virtual RDMA network card, the identifier of the first port and a second mapping table prestored in the host, wherein the second mapping table comprises the corresponding relation among the identifier of the virtual RDMA network card, the identifier of the port of the virtual RDMA network card and the identifier of the port of the virtual Ethernet network card, and the second port is the port of the virtual Ethernet network card of the first virtual machine;
the first virtual machine determines a first GID of the first port according to a preset GID generation algorithm and a virtual address corresponding to the identifier of the second port;
the first virtual machine sets an address of the first port to the first GID.
In one possible implementation manner, the determining, by the first virtual machine, an identifier of a corresponding second port according to the identifier of the virtual RDMA network card, the identifier of the first port, and a second mapping table pre-stored in the host includes:
the first virtual machine sends query request information to the host, wherein the query request carries the identifier of the virtual RDMA network card and the identifier of the first port;
the host determines the identifier of the corresponding second port according to the query request information and a pre-stored second mapping table;
and the host sends query response information to the first virtual machine, wherein the query response information carries the identifier of the second port.
In one possible implementation, the method further includes:
when the first virtual machine detects Ethernet address change information carrying the identifier of the second port, determining the identifier of the corresponding virtual RDMA network card and the identifier of the first port according to the identifier of the second port and a prestored third mapping table, wherein the third mapping table comprises the corresponding relation among the identifier of the port of the virtual Ethernet network card, the identifier of the virtual RDMA network card and the identifier of the port of the virtual RDMA network card;
and the first virtual machine updates the address of the first port according to the event type carried in the Ethernet address change information.
In a possible implementation manner, the updating, by the first virtual machine, the address of the first port according to the event type carried in the ethernet address change information includes:
when the event type is the network card starting, the first virtual machine acquires a virtual address corresponding to the second port identifier;
and the first virtual machine determines a second GID of the first port according to the virtual address and a preset GID generation algorithm, and sets the address of the first port as the second GID.
In a possible implementation manner, the updating, by the first virtual machine, the address of the first port according to the event type carried in the ethernet address change information includes:
and when the event type is network card closing, the first virtual machine deletes the address of the first port.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.

Claims (18)

1. A method for data transmission, wherein a device applied by the method comprises a first virtual machine and a host, and the method comprises the following steps:
the host obtains a first instruction and a second instruction from the first virtual machine, wherein the first instruction is used for indicating that a source address of a queue pair QP queue is a virtual address of the first virtual machine, the second instruction is used for indicating that a destination address of the QP queue is a virtual address of the second virtual machine, and the QP queue is used for communication between the first virtual machine and the second virtual machine;
the host deletes the first instruction and modifies the second instruction into a third instruction, wherein the third instruction is used for indicating that a destination address in the QP queue is a physical address of the second virtual machine;
and the host sets the QP queue according to the third instruction so as to send data to the second virtual machine through the set QP queue.
2. The method of claim 1, further comprising:
the first virtual machine sends the first instruction and the second instruction to the host through a first port, wherein the first port is a port of a virtual remote direct access (RDMA) network card of the first virtual machine, and the first port corresponds to a Global Identity (GID) of the first virtual machine in an RDMA network.
3. The method of claim 1, wherein modifying the second instruction into a third instruction comprises:
the host determines a physical address of the second virtual machine according to a first mapping table stored in advance and a virtual address of the second virtual machine in the second instruction, wherein the first mapping table comprises a corresponding relation between the virtual address and the physical address;
and the host modifies the virtual address of the second virtual machine in the second instruction into the physical address of the second virtual machine to obtain a third instruction.
4. The method of claim 2, wherein the first virtual machine further comprises a port of a virtual ethernet network card, the method further comprising:
the first virtual machine acquires the identifier of the virtual RDMA network card and the identifier of the first port;
the first virtual machine determines the identifier of a corresponding second port according to the identifier of the virtual RDMA network card, the identifier of the first port and a second mapping table prestored in the host, wherein the second mapping table comprises the corresponding relation among the identifier of the virtual RDMA network card, the identifier of the port of the virtual RDMA network card and the identifier of the port of the virtual Ethernet network card, and the second port is the port of the virtual Ethernet network card of the first virtual machine;
the first virtual machine determines a first GID of the first port according to a preset GID generation algorithm and a virtual address corresponding to the identifier of the second port;
the first virtual machine sets an address of the first port to the first GID.
5. The method of claim 4, wherein the first virtual machine determining the identity of the corresponding second port from the identity of the virtual RDMA network card, the identity of the first port, and a second mapping table pre-stored in the host comprises:
the first virtual machine sends query request information to the host, wherein the query request information carries the identifier of the virtual RDMA network card and the identifier of the first port;
the host determines the identifier of the corresponding second port according to the query request information and a pre-stored second mapping table;
and the host sends query response information to the first virtual machine, wherein the query response information carries the identifier of the second port.
6. The method of claim 4, further comprising:
when the first virtual machine detects information carrying the identifier of the second port and used for changing the Ethernet address, determining the identifier of the corresponding virtual RDMA network card and the identifier of the first port according to the identifier of the second port and a prestored third mapping table, wherein the third mapping table comprises the corresponding relation among the identifier of the port of the virtual Ethernet network card, the identifier of the virtual RDMA network card and the identifier of the port of the virtual RDMA network card;
and the first virtual machine updates the address of the first port according to the event type carried in the information for changing the Ethernet address.
7. The method according to claim 6, wherein the updating, by the first virtual machine, the address of the first port according to the event type carried in the information for changing the ethernet address includes:
when the event type is the network card starting, the first virtual machine acquires a virtual address corresponding to the second port identifier;
and the first virtual machine determines a second GID of the first port according to a virtual address corresponding to the second port identifier and a preset GID generation algorithm, and sets the address of the first port as the second GID.
8. The method according to claim 6, wherein the updating, by the first virtual machine, the address of the first port according to the event type carried in the ethernet address change information includes:
and when the event type is network card closing, the first virtual machine deletes the address of the first port.
9. A data sending device, characterized in that the device comprises a host, wherein the host is used for obtaining a first instruction and a second instruction from a first virtual machine, the first instruction is used for indicating that a source address of a queue pair QP queue is a virtual address of the first virtual machine, the second instruction is used for indicating that a destination address of the QP queue is a virtual address of a second virtual machine, and the QP queue is a QP queue used by the first virtual machine for communicating with the second virtual machine; deleting the first instruction, and modifying the second instruction into a third instruction, wherein the third instruction is used for indicating that a destination address in the QP queue is a physical address of the second virtual machine; and the host sets the QP queue according to the third instruction so as to send data to the second virtual machine through the set QP queue.
10. The apparatus of claim 9, further comprising a first virtual machine,
the first virtual machine is used for sending the first instruction and the second instruction to the host through a first port, the first port is a port of a virtual remote direct access (RDMA) network card of the first virtual machine, and the first port corresponds to a Global Identity (GID) of the first virtual machine in an RDMA network.
11. The data sending device of claim 9, wherein the host is specifically configured to:
determining a physical address of the second virtual machine according to a first mapping table stored in advance and a virtual address of the second virtual machine in the second instruction, wherein the first mapping table comprises a corresponding relation between the virtual address and the physical address;
and modifying the virtual address of the second virtual machine in the second instruction into the physical address of the second virtual machine to obtain a third instruction.
12. The data sending device of claim 9, wherein the first virtual machine further comprises a port of a virtual ethernet network card, and the first virtual machine is further configured to:
acquiring an identifier of the virtual RDMA network card and an identifier of a first port; determining a corresponding identifier of a second port according to the identifier of the virtual RDMA network card, the identifier of the first port and a second mapping table prestored in the host, wherein the second mapping table comprises a corresponding relation among the identifier of the virtual RDMA network card, the identifier of the port of the virtual RDMA network card and the identifier of the port of the virtual Ethernet network card, and the second port is the port of the virtual Ethernet network card of the first virtual machine; determining a first GID of the first port according to a preset GID generation algorithm and a virtual address corresponding to the identifier of the second port; setting an address of the first port to the first GID.
13. The data sending device of claim 9, wherein the first virtual machine is specifically configured to:
sending query request information to the host, wherein the query request information carries the identifier of the virtual RDMA network card and the identifier of the first port;
the host is specifically configured to: determining the identifier of a corresponding second port according to the query request information and a pre-stored second mapping table; and sending query response information to the first virtual machine, wherein the query response information carries the identifier of the second port.
14. The data sending device of claim 11, wherein the first virtual machine is further configured to:
when the first virtual machine detects information carrying an identifier of a second port and used for changing an Ethernet address, determining the identifier of a corresponding virtual RDMA network card and the identifier of a first port according to the identifier of the second port and a prestored third mapping table, wherein the third mapping table comprises the corresponding relation among the identifier of the port of the virtual Ethernet network card, the identifier of the virtual RDMA network card and the identifier of the port of the virtual RDMA network card;
and the first virtual machine updates the address of the first port according to the event type carried in the information for changing the Ethernet address.
15. The data sending device of claim 14, wherein the first virtual machine is specifically configured to:
when the event type is the starting of the network card, acquiring a virtual address corresponding to the second port identifier;
and determining a second GID of the first port according to the virtual address corresponding to the second port identifier and a preset GID generation algorithm, and setting the address of the first port as the second GID.
16. The data sending device of claim 14, wherein the first virtual machine is specifically configured to:
and when the event type is the network card closing, deleting the address of the first port.
17. A data transmission apparatus, characterized in that the apparatus comprises: a memory, a processing circuit and a host channel adapter HCA card;
wherein the memory is used for storing codes;
the processing circuit is used for operating a first virtual machine and a host according to the codes stored in the memory;
the processing circuitry, further to implement the method of any of claims 1 to 8 by the first virtual machine and the host;
and the HCA card is used for sending data to the second virtual machine.
18. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method of any of claims 1 to 8.
CN202011203280.8A 2018-01-23 2018-01-23 Data sending method and device Active CN112398817B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011203280.8A CN112398817B (en) 2018-01-23 2018-01-23 Data sending method and device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011203280.8A CN112398817B (en) 2018-01-23 2018-01-23 Data sending method and device
CN201810064378.6A CN110071900B (en) 2018-01-23 2018-01-23 Data sending method and device

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
CN201810064378.6A Division CN110071900B (en) 2018-01-23 2018-01-23 Data sending method and device

Publications (2)

Publication Number Publication Date
CN112398817A CN112398817A (en) 2021-02-23
CN112398817B true CN112398817B (en) 2022-02-25

Family

ID=67365132

Family Applications (2)

Application Number Title Priority Date Filing Date
CN202011203280.8A Active CN112398817B (en) 2018-01-23 2018-01-23 Data sending method and device
CN201810064378.6A Active CN110071900B (en) 2018-01-23 2018-01-23 Data sending method and device

Family Applications After (1)

Application Number Title Priority Date Filing Date
CN201810064378.6A Active CN110071900B (en) 2018-01-23 2018-01-23 Data sending method and device

Country Status (1)

Country Link
CN (2) CN112398817B (en)

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111193653B (en) * 2019-12-31 2021-08-06 腾讯科技（深圳）有限公司 Data transmission method, device, equipment and storage medium
CN113194045B (en) * 2020-01-14 2023-11-17 阿里巴巴集团控股有限公司 Data traffic analysis method, device, storage medium and processor
CN113535370A (en) * 2020-04-09 2021-10-22 深圳致星科技有限公司 Method and equipment for realizing multiple RDMA network card virtualization of load balancing
CN112583655B (en) * 2020-11-02 2023-04-14 北京迈格威科技有限公司 Data transmission method and device, electronic equipment and readable storage medium
CN113448634B (en) * 2021-05-31 2022-07-19 山东英信计算机技术有限公司 ROCE network card resource management method, device, equipment and readable medium
CN113766005B (en) * 2021-07-29 2023-04-28 苏州浪潮智能科技有限公司 RDMA-based method and system for batch creation of cloud hosts
CN113312155B (en) * 2021-07-29 2022-02-01 阿里云计算有限公司 Virtual machine creation method, device, equipment, system and computer program product
CN113395359B (en) * 2021-08-17 2021-10-29 苏州浪潮智能科技有限公司 File currency cluster data transmission method and system based on remote direct memory access
CN114050998A (en) * 2022-01-07 2022-02-15 阿里巴巴（中国）有限公司 Method, device, electronic equipment and medium for realizing remote direct memory access
CN115314437B (en) * 2022-06-16 2024-03-01 阿里巴巴（中国）有限公司 Container virtual network communication method and container virtual network
CN115277640B (en) * 2022-07-29 2023-11-24 迈普通信技术股份有限公司 Data processing method, device, intelligent network card and storage medium
CN115834296B (en) * 2023-02-16 2023-04-18 北京融为科技有限公司 Data transmission method and device, electronic equipment and storage medium

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7886139B2 (en) * 2007-02-23 2011-02-08 International Business Machines Corporation Method to enable firmware to boot a system from an ISCSI device
CN103384551A (en) * 2013-01-04 2013-11-06 华为技术有限公司 Virtual machine communication method and system based on PCIE network, and server
CN104636185A (en) * 2015-01-27 2015-05-20 华为技术有限公司 Service context management method, physical host, PCIE equipment and migration management equipment
CN104753816A (en) * 2015-03-27 2015-07-01 华为技术有限公司 RDMA (remote direct memory access) connection message processing method and related device
CN105637822A (en) * 2013-08-27 2016-06-01 甲骨文国际公司 System and method for providing a data service in an engineered system for middleware and application execution
WO2017091465A1 (en) * 2015-11-24 2017-06-01 Oracle International Corporation System and method for efficient virtualization in lossless networks
CN107113298A (en) * 2014-12-29 2017-08-29 Nicira股份有限公司 The method that many leases are supported is provided for RDMA
CN107193699A (en) * 2017-05-22 2017-09-22 郑州云海信息技术有限公司 One kind tests the wide time-delay method of HCA cassette tapes automatically by RDMA modes

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8832216B2 (en) * 2011-08-31 2014-09-09 Oracle International Corporation Method and system for conditional remote direct memory access write
US9723008B2 (en) * 2014-09-09 2017-08-01 Oracle International Corporation System and method for providing an integrated firewall for secure network communication in a multi-tenant environment
US9928093B2 (en) * 2015-02-24 2018-03-27 Red Hat Israel, Ltd. Methods and systems for establishing connections associated with virtual machine migrations
US9813338B2 (en) * 2015-12-10 2017-11-07 Cisco Technology, Inc. Co-existence of routable and non-routable RDMA solutions on the same network interface

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7886139B2 (en) * 2007-02-23 2011-02-08 International Business Machines Corporation Method to enable firmware to boot a system from an ISCSI device
CN103384551A (en) * 2013-01-04 2013-11-06 华为技术有限公司 Virtual machine communication method and system based on PCIE network, and server
CN105637822A (en) * 2013-08-27 2016-06-01 甲骨文国际公司 System and method for providing a data service in an engineered system for middleware and application execution
CN107113298A (en) * 2014-12-29 2017-08-29 Nicira股份有限公司 The method that many leases are supported is provided for RDMA
CN104636185A (en) * 2015-01-27 2015-05-20 华为技术有限公司 Service context management method, physical host, PCIE equipment and migration management equipment
CN104753816A (en) * 2015-03-27 2015-07-01 华为技术有限公司 RDMA (remote direct memory access) connection message processing method and related device
WO2017091465A1 (en) * 2015-11-24 2017-06-01 Oracle International Corporation System and method for efficient virtualization in lossless networks
CN107193699A (en) * 2017-05-22 2017-09-22 郑州云海信息技术有限公司 One kind tests the wide time-delay method of HCA cassette tapes automatically by RDMA modes

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
基于KVM虚拟机双机热备技术研究与实现;万仔仁;《中国优秀硕士学位论文全文数据库信息科技辑》;20170531;第1-50页 *
基于全局地址空间的高效 I/O 虚拟化方法研究;李波，孟丹，霍志刚;《小型微型计算机系统》;20111130;第2274-2279页 *

Also Published As

Publication number Publication date
CN112398817A (en) 2021-02-23
CN110071900A (en) 2019-07-30
CN110071900B (en) 2020-11-17

Similar Documents

Publication Publication Date Title
CN112398817B (en) Data sending method and device
US11283650B2 (en) Method for sending virtual extensible local area network packet, computer device, and computer readable medium
US11372802B2 (en) Virtual RDMA switching for containerized applications
US9462047B2 (en) Network interface controller supporting network virtualization
US7996569B2 (en) Method and system for zero copy in a virtualized network environment
US9274825B2 (en) Virtualization gateway between virtualized and non-virtualized networks
US9588807B2 (en) Live logical partition migration with stateful offload connections using context extraction and insertion
KR101245213B1 (en) Method and system for network configuration and/or provisioning based on open virtualization format(ovf) metadata
US8255475B2 (en) Network interface device with memory management capabilities
US8830870B2 (en) Network adapter hardware state migration discovery in a stateful environment
US20220070095A1 (en) Data transmission method and apparatus, network adapter, and storage medium
EP3828709A1 (en) Communication method and network card
EP4160424A2 (en) Zero-copy processing
US9973351B2 (en) Data processing method and apparatus, storage controller, and cabinet
CN109698845B (en) Data transmission method, server, unloading card and storage medium
WO2015024373A1 (en) Virtual machine communication method and device
CN108063737B (en) FCoE storage area network read request processing method and system
WO2014172869A1 (en) Method, device and system for communication in virtual local area network
WO2020215455A1 (en) Data transmission method and system employing virtio port
US20240056391A1 (en) Communication method, apparatus, device, and system, and computer-readable storage medium
WO2022089645A1 (en) Communication method, apparatus, device, system, and computer readable storage medium
CN114928589A (en) Data transmission method, data transmission device, computer readable medium and equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant