WO2014106321A1 - Procédé, serveur et système de communication entre machines virtuelles basés sur un réseau pcie - Google Patents

Procédé, serveur et système de communication entre machines virtuelles basés sur un réseau pcie Download PDF

Info

Publication number
WO2014106321A1
WO2014106321A1 PCT/CN2013/070016 CN2013070016W WO2014106321A1 WO 2014106321 A1 WO2014106321 A1 WO 2014106321A1 CN 2013070016 W CN2013070016 W CN 2013070016W WO 2014106321 A1 WO2014106321 A1 WO 2014106321A1
Authority
WO
WIPO (PCT)
Prior art keywords
virtual machine
pcie
host
destination
server
Prior art date
Application number
PCT/CN2013/070016
Other languages
English (en)
Chinese (zh)
Inventor
李思聪
邱军
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to CN201380000042.9A priority Critical patent/CN103384551B/zh
Priority to PCT/CN2013/070016 priority patent/WO2014106321A1/fr
Publication of WO2014106321A1 publication Critical patent/WO2014106321A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/45595—Network integration; Enabling network access in virtual machine instances

Definitions

  • the invention belongs to the technical field of computer internet communication, and relates to a virtual machine communication method, in particular to a virtual machine communication method, server and system based on a Peripheral Component Interconnect Express (PCIE) network.
  • PCIE Peripheral Component Interconnect Express
  • inter-VM communication is primarily based on Ethernet.
  • the user mode application in the virtual machine first calls the Socket interface through the Transmission Control Protocol (TCP) / Internet Protocol (IP) stack.
  • TCP Transmission Control Protocol
  • IP Internet Protocol
  • the actual forwarding and processing is performed in the Virtual Machine Monitor (VMM).
  • VMM Virtual Machine Monitor
  • the TCP/IP protocol is more complicated, and the protocol processing of the operating system (OS) occupies a large part of the network delay.
  • OS operating system
  • CPU Central Processing Unit
  • an object of the present invention is to provide a virtual machine communication method, a server and a system based on a PCIE network, which are used to solve the slow communication speed between virtual machines on different physical machines in the prior art. problem.
  • an embodiment of the present invention provides a virtual machine communication method based on a high-speed peripheral component interconnect PCIE network, including:
  • the source virtual machine on the source host intercepts the network request on the TCP/IP protocol stack, and sends the data contained in the network request to the virtual machine monitor on the source host;
  • the virtual machine monitor on the source host determines the PCIE NTB address space of the destination host corresponding to the source host on the PCIE switch according to the destination address of the network request, where the destination host is the purpose of the network request
  • the host to which the virtual machine belongs, the PCIE NTB address space of the destination host corresponding to the source host on the PCIE switch is the PCIE corresponding to the source host in the PCIE NTB address space allocated to the destination host on the PCIE switch. a NTB address space; the virtual machine monitor on the source host sends a PCIE message containing the data to the
  • the virtual machine monitor on the source host determines, according to the destination address of the network request, a PC IE NTB address of the destination host corresponding to the source host on the PCIE switch. Space, specifically:
  • the PCIE NTB address space information of the destination host corresponding to the source host is cached by the virtual machine monitor on the source host when the source virtual machine on the source host sends an address resolution protocol request for address resolution. .
  • the method when the source virtual machine on the source host sends an address resolution protocol request for address resolution, the method includes: the source virtual machine The monitor intercepts the destination virtual machine that the source virtual machine requests from the network After the request for the address resolution protocol is sent, the request message for querying the PCIE NTB address space information of the destination host corresponding to the source virtual machine is initiated to the subnet management server;
  • the virtual machine monitor on the source host receives the response message returned by the subnet management server, and caches the PCIE NTB address space information of the destination host corresponding to the source virtual machine included in the response message locally.
  • the PCIE NTB address space of the destination host corresponding to the source virtual machine is the same as the PCIE NTB address space of the destination host corresponding to the source host.
  • the virtual machine monitor on the source host The PCI IE packet containing the data in the network request is sent to the PC IE NTB address space of the destination host corresponding to the source host on the PCIE switch, and includes:
  • the virtual machine monitor on the source host puts the data included in the network request, the ID of the source virtual machine, and the source NTB port ID into a data payload of the PCIE packet, and virtualizes the destination of the network request. Sending the PCIE message to the PCIE switch as the destination address of the PCIE message, so that the PCIE switch writes the PCIE file to the PCIE NTB address space of the destination host. a PCIE NTB address space of the source host;
  • the source NTB port ID is an ID of an NTB port in the PCIE switch that is connected to a virtual machine monitor on the source host.
  • the method further includes:
  • the PCIE NTB address space is divided into the same number of PCIE NTB address subspaces as the number of hosts connected to the PCIE switch;
  • Each of the address subspaces corresponds to a host, and is configured to receive data sent to the corresponding host;
  • Each of the other hosts is assigned an address micro space in each of the address subspaces, and each of the address micro spaces corresponds to one of the other hosts for receiving from the other host
  • the address subspace corresponds to data of the host, where the other host is a host connected to the PCIE switch except the host corresponding to the address subspace.
  • the method further includes:
  • the subnet management server allocates all the PCIE NTB address spaces when the PCIE network is initialized, and notifies all hosts of its own PCIE NTB address space information, and the PC IE of the subnet management server corresponding to each host cache NTB address space information, so that the request message for querying the host PCIE NTB address space of the destination destination can be sent to the subnet management server.
  • the method further includes:
  • the subnet management server maintains a correspondence between the host and the virtual machine, and updates the corresponding relationship according to the creation, modification, and destruction of the virtual machine.
  • the virtual machine monitor of the destination host is configured according to the PCIE packet
  • the ID of the destination virtual machine of the network request distributes the data to the destination virtual machine of the network request, including:
  • the virtual machine monitor of the destination host reads the PCIE packet in the PCIE NTB address space corresponding to the source host in the PCIE NTB address space of the destination host, according to the PCIE packet.
  • the ID of the destination virtual machine of the network request distributes the data to the destination virtual machine of the network request.
  • an embodiment of the present invention provides a virtual machine communication system based on a high-speed peripheral component interconnection network, including: at least two hosts and one PCIE switch, wherein the at least two hosts are communicatively connected through the PCIE switch;
  • Each of the at least two hosts is configured to run a virtual machine and a virtual machine monitoring module, wherein the source virtual machine on the source host of the at least two hosts intercepts the network on the TCP/IP protocol stack Requesting, and transmitting data contained in the network request to the virtual machine monitoring of the source host
  • the virtual machine monitor of the source host determines, according to the destination address of the network request sent by the source virtual machine, a PCIE NTB address space of the destination host corresponding to the source host on the PCIE switch, and includes a The PCIE packet of the data in the network request is sent to the PCIE NTB address space of the destination host corresponding to the source host on the PCIE switch, so that the virtual machine monitor of the destination host is in the PCIE packet.
  • the ID of the destination virtual machine of the network request distributes the data to a destination virtual machine of the network request;
  • the destination host is a host to which the destination virtual machine of the network belongs, and the PCIE NTB address space of the destination host corresponding to the source host on the PCIE switch is allocated to the destination host on the PCIE switch.
  • the PCIE NTB address space in the PCIE NTB address space corresponds to the source host.
  • each of the hosts includes:
  • a virtual machine running module configured to run the virtual machine
  • a virtual machine monitoring module configured to run the virtual machine monitor.
  • each of the hosts includes: at least one processor, a memory;
  • the memory is configured to store an execution instruction
  • the processor is configured to communicate with the memory when the host is running, and execute the execution instruction to cause the host to run the virtual machine and the virtual machine monitor.
  • the virtual machine monitor of the source host is configured according to the source virtual machine Determining, by the destination address of the network request, the PC IE NTB address space of the destination host corresponding to the source host on the PCIE switch, specifically:
  • the virtual machine monitor on the source host determines the destination host of the network request according to the destination address of the network request sent by the source virtual machine, and according to the PCIE of the destination host corresponding to the source host locally cached Determining, by the NTB address space information, a PCIE NTB address space of the destination host corresponding to the source host on the PCIE switch; The PCIE NTB address space information of the destination host corresponding to the source host is cached by the virtual machine monitor on the source host when the source virtual machine on the source host sends an address resolution protocol request for address resolution. .
  • the subnet management server is further included; then, the source of the virtual machine monitor on the source host on the source host
  • the cache is specifically: when the source virtual machine does not have the MAC address information of the destination virtual machine requested by the network, the destination virtual machine requests the network The virtual machine sends the address resolution protocol request, and after the virtual machine monitor on the source host intercepts the address resolution protocol request sent by the virtual machine, initiating a query to the subnet management server to query the destination virtual machine.
  • the subnet management server is configured to: after receiving a request message sent by the virtual machine monitor of the source host to query a PCIE NTB address space corresponding to the destination virtual machine, determine, according to a correspondence between the host and the virtual machine, The destination host to which the destination virtual machine belongs, obtains the PCIE NTB address space of the destination host corresponding to the source host, and sends the response message to the virtual machine monitor of the source host.
  • the virtual machine monitor on the source host is included in the network request
  • the PCIE packet of the data is sent to the PC IE NTB address space of the destination host corresponding to the source host on the PCIE switch, and includes:
  • the virtual machine monitor on the source host puts the data included in the network request, the ID of the source virtual machine, and the source NTB port ID into a data payload of the PCIE packet, and virtualizes the destination of the network request. Sending the PCIE message to the PCIE switch as the destination address of the PCIE message, so that the PCIE switch writes the PCIE file to the PCIE NTB address space of the destination host.
  • a PCIE NTB address space of the source host where the source NTB port ID is an ID of an NTB port of the PCIE switch connected to a virtual machine monitor on the source host;
  • the PCIE switch is further configured to write the PCE E NTB address space corresponding to the source host in the PCIE NTB address space of the destination host.
  • the subnet management server is further configured to maintain and manage all PCIE NTB address spaces, where all PCIE NTB address spaces are divided into PCIE NTB address subspaces of the same number of hosts connected to the PCIE switch, each of which is The address subspace corresponds to a host, and is configured to receive data sent to the corresponding host, and allocate an address micro space to each of the other hosts in each of the address subspaces, where each of the address micro spaces corresponds to one
  • the other host is configured to receive data sent by the other host to the host corresponding to the address subspace, where the other host is connected to the PCIE switch except the host corresponding to the address subspace. Host.
  • the subnet management server is further configured to allocate the all when the PCIE network is initialized
  • PC IE NTB address space and informs all hosts of its own PC IE NTB address space information, so that each host caches the PCIE NTB address space information of the subnet management server corresponding to the host, so that the subnet management server can be sent to the subnet management server.
  • the request message for querying the destination host PCIE NTB address space is described.
  • the subnet management server is further configured to maintain a correspondence between the host and the virtual machine, and update the corresponding relationship in real time according to the creation, modification, and destruction of the virtual machine.
  • the virtual machine monitor of the destination host is configured according to the PCIE packet
  • the ID of the destination virtual machine requested by the network distributes the data to the destination virtual machine of the network request, including:
  • the virtual machine monitor of the destination host reads the PCIE packet in the PCIE NTB address space corresponding to the source host in the PCIE NTB address space of the destination host, according to the PCIE
  • the ID of the destination virtual machine of the network request in the message distributes the data to the destination virtual machine of the network request.
  • an embodiment of the present invention provides a server, including:
  • a virtual machine monitoring module configured to run the virtual machine monitor
  • the virtual machine is configured to intercept a network request on a TCP/IP protocol stack, and send the data included in the network request to a virtual machine monitor;
  • the virtual machine monitor is configured to determine, according to the destination address of the network request sent by the virtual machine, a PCIE NTB address space of a destination host corresponding to the server on the PCIE switch, and the data in the network request is included
  • the PCIE message is sent to the PCIE NTB address space of the destination host corresponding to the server on the PCIE switch, so that the virtual machine monitor of the destination host according to the purpose of the network request in the PCIE packet
  • the ID of the virtual machine distributes the data to the destination virtual machine of the network request;
  • the destination host is a host to which the destination virtual machine of the network belongs, and the PCIE NTB address space of the destination host corresponding to the server on the PCIE switch is a PCIE allocated to the destination host on the PCIE switch.
  • the NTB address space corresponding to the server is a host to which the destination virtual machine of the network belongs, and the PCIE NTB address space of the destination host corresponding to the server on the PCIE switch is a PCIE allocated to the destination host on the PCIE switch.
  • the virtual machine monitor is configured to determine, according to a destination address of the network request sent by the virtual machine, a PCIE NTB of a destination host corresponding to the server on a PCIE switch.
  • Address space specifically:
  • the virtual machine monitor is configured to determine, according to the destination address of the network request sent by the virtual machine, a destination host that is requested by the network, and according to the PCIE NTB of the destination host corresponding to the locally cached server Address space information, determining a PC IE NTB address space of the destination host corresponding to the server on the PCIE switch;
  • the PCIE NTB address space information of the destination host corresponding to the server is cached by the virtual machine monitor when the virtual machine sends an address resolution protocol request for address resolution.
  • the virtual machine monitor caches when the virtual machine sends an address resolution protocol request for address resolution, specifically:
  • the virtual machine is further configured to: when the virtual machine does not have the MAC address information of the destination virtual machine requested by the network, send the address resolution protocol request to the destination virtual machine requested by the network;
  • the virtual machine monitor is further configured to: after intercepting the address resolution protocol request sent by the virtual machine, initiate a request for querying a PCIE NTB address space corresponding to the destination virtual machine to the subnet management server, and After receiving the response message returned by the subnet management server, the PCIE NTB address space information of the destination host corresponding to the virtual machine included in the response message is cached locally, where the virtual machine corresponds to the The PCIE NTB address space of the destination host is the same as the PCIE NTB address space of the destination host corresponding to the server.
  • the virtual machine monitor on the source host The PCIE packet containing the data in the network request is sent to the PCIE NTB address space of the destination host corresponding to the server on the PCIE switch, including:
  • the virtual machine monitor is configured to put data included in the network request, an ID of the virtual machine, and a source NTB port ID into a data payload of the PCIE packet, where the network request destination virtual machine Sending the PCIE message to the PCIE switch as the destination address of the PCIE message, so that the PCIE switch writes the PCIE file to the PCIE NTB address space of the destination host corresponding to the The PCIE NTB address space of the server; wherein the source NTB port ID is an ID of an NTB port of the PCIE switch connected to the virtual machine monitor.
  • the PC IE NTB address space of the destination host corresponding to the server is maintained and managed by the subnet management server, and is configured to receive data sent by the server to the destination host;
  • the PCIE NTB address space is divided into the same number of PCIE NTB address subspaces as the number of hosts connected to the PCIE switch, and each of the address subspaces corresponds to a host, and is configured to receive and send to the corresponding host.
  • each of the other hosts is allocated an address space in each of the address subspaces, and each of the address spaces corresponds to one of the other hosts, and is configured to receive from the other hosts and send the address to the address
  • the space corresponds to the data of the host, and the other host is a host connected to the PCIE switch except the host corresponding to the address subspace.
  • the virtual machine monitor is further configured to receive and cache a PC IE NTB address of the subnet management server corresponding to the server Spatial information.
  • the virtual machine monitor of the destination host is configured according to the purpose in the PCIE packet
  • the virtual machine ID distributes the data to the destination virtual machine of the network request, including: the virtual machine monitor of the destination host reads a PCIE NTB corresponding to the server in a PCIE NTB address space of the destination host.
  • the PCIE message in the address space distributes the data to the destination virtual machine requested by the network according to the ID of the destination virtual machine in the PCIE message.
  • an embodiment of the present invention provides a server, including: a processor and a memory, where the memory stores an execution instruction, and when the server is running, the processor communicates with the memory, Executing the execution instruction by the processor causes the server to perform the following method:
  • the virtual machine on the server intercepts the network request on the TCP/IP protocol stack, and sends the data contained in the network request to the virtual machine monitor on the server;
  • the virtual machine monitor on the server determines the PCIE NTB address space of the destination host corresponding to the server on the PCIE switch according to the destination address of the network request, where the destination host is the destination virtual machine requested by the network.
  • An associated host, a PCIE NTB address space of the destination host corresponding to the server on the PCIE switch is a PCIE NTB address space corresponding to the server in a PCIE NTB address space allocated to the destination host on the PCIE switch;
  • the virtual machine monitor on the server sends a PCIE message containing the data to a PCIE NTB address space of the destination host corresponding to the server on the PCIE switch, so that the virtual machine monitor of the destination host is The destination virtual of the network request in the PCIE packet
  • the ID of the machine distributes the data to the destination virtual machine of the network request.
  • the virtual machine monitor on the server is configured according to a destination address of the network request. Determine the PCIE NTB address space of the destination host corresponding to the server on the PCIE switch, specifically:
  • the PCIE NTB address space information of the destination host corresponding to the server is cached when the virtual machine monitor on the server sends an address resolution protocol request for address resolution on the virtual machine on the server.
  • the executing, by the processor, the execution instruction, in the method performed by the server, the virtual machine sending address on the server The protocol is resolved when the protocol request is used for address resolution, including:
  • the virtual machine monitor intercepts the address resolution protocol request sent by the virtual machine to the destination virtual machine requested by the network, and then initiates querying the PC IE NTB address space information of the destination host corresponding to the virtual machine. Request message to the subnet management server;
  • the virtual machine monitor on the server receives the response message returned by the subnet management server, and caches the PCIE NTB address space information of the destination host corresponding to the virtual machine included in the response message, where The PCIE NTB address space of the destination host corresponding to the virtual machine is the same as the PC IE NTB address space of the destination host corresponding to the server.
  • the processor executes the execution instruction In the method performed by the server, the virtual machine monitor of the server sends a PCIE message including the data in the network request to the PC IE NTB address space of the destination host corresponding to the server on the PCIE switch, including :
  • the virtual machine monitor on the server puts the data contained in the network request, the ID of the virtual machine, and the source NTB port ID into a data payload of the PCIE packet, and the destination of the network request
  • the ID of the virtual machine is used as the destination address of the PCIE message, and the PCIE certificate is sent to the PCIE switch, so that the PCIE switch writes the PCIE file to the PCIE NTB address space of the destination host.
  • the PCIE NTB address space of the server corresponds to the PCIE NTB address space of the server;
  • the source NTB port ID is an ID of an NTB port in the PCIE switch that is connected to a virtual machine monitor on the server.
  • the method that the processor executes the execution instruction to cause the server to perform further includes: the server corresponding to The PC IE NTB address space of the destination host is maintained and managed by the subnet management server, and is configured to receive data sent by the server to the destination host; wherein, all PCIE NTB address spaces are divided into the PCIE a PCIE NTB address subspace of the same number of hosts connected to the switch, each of the address subspaces corresponding to a host, configured to receive data sent to the corresponding host, in each of the address subspaces Each other host allocates an address space, and each of the address spaces corresponds to one of the other hosts, and is configured to receive data sent by the other host to the host corresponding to the address subspace, where the other hosts are The address subspace corresponds to a host connected to the PCIE switch except the host.
  • the method, by the processor, executing the execution instruction, the method performed by the server further includes:
  • the virtual machine monitor of the server receives and caches PCIE NTB address space information of the subnet management server corresponding to the server, so that the request message of the host PCIE NTB address space of the query destination can be sent to the subnet management server.
  • the processor executes the execution instruction to cause the server to perform
  • the purpose The virtual machine monitor of the host distributes the data to the destination virtual machine of the network request according to the ID of the destination virtual machine in the PCIE packet, including:
  • the virtual machine monitor of the destination host reads the PCIE address in the PCIE NTB address space of the destination host in the PCIE NTB address space of the destination host, according to the destination virtual in the PCIE packet
  • the ID of the machine distributes the data to the destination virtual machine of the network request.
  • an embodiment of the present invention provides a computer readable medium, including computer execution instructions, where the computer execution instructions are used to cause a server to execute the following method:
  • the virtual machine on the server intercepts the network request on the TCP/IP protocol stack, and sends the data contained in the network request to the virtual machine monitor on the server;
  • the virtual machine monitor on the server determines the PCIE NTB address space of the destination host corresponding to the server on the PCIE switch according to the destination address of the network request, where the destination host is the destination virtual machine requested by the network.
  • An associated host, a PCIE NTB address space of the destination host corresponding to the server on the PCIE switch is a PCIE NTB address space corresponding to the server in a PCIE NTB address space allocated to the destination host on the PCIE switch;
  • the virtual machine monitor on the server sends a PCIE message containing the data to a PCIE NTB address space of the destination host corresponding to the server on the PCIE switch, so that the virtual machine monitor of the destination host is The ID of the destination virtual machine of the network request in the PCIE message distributes the data to the destination virtual machine of the network request.
  • the virtual machine monitor on the server determines the PCIE switch according to the destination address of the network request.
  • the PCIE NTB address space of the destination host corresponding to the server is specifically:
  • the PCIE NTB address space information of the destination host corresponding to the server is cached when the virtual machine monitor on the server sends an address resolution protocol request for address resolution on the virtual machine on the server.
  • the virtual machine on the server sends an address resolution protocol request Cache at address resolution, including:
  • the virtual machine monitor intercepts the address resolution protocol request sent by the virtual machine to the destination virtual machine requested by the network, and then initiates querying the PC IE NTB address space information of the destination host corresponding to the virtual machine. Request message to the subnet management server;
  • the virtual machine monitor on the server receives the response returned by the subnet management server
  • the PCIE NTB address space information of the destination host corresponding to the virtual machine included in the response message is cached locally, where the PCIE NTB address space of the destination host corresponding to the virtual machine is The PC IE NTB address space of the destination host corresponding to the server is the same.
  • the computer executing the instruction, in a method for causing a server to execute, the virtual machine of the server
  • the monitor sends a PCIE message containing the data in the network request to the PCIE NTB address space of the destination host corresponding to the server on the PCIE switch, including: the virtual machine monitor on the server will use the network
  • the data included in the request, the ID of the virtual machine, and the source NTB port ID are placed in the data payload of the PCIE packet, and the ID of the destination virtual machine requested by the network is sent as the destination address of the PCIE message.
  • the PCIE switch sends the PCIE switch to write the PCIE switch to the PCIE NTB address space of the destination host in the PCIE NTB address space corresponding to the PCIE NTB address space of the server;
  • the source NTB port ID is an ID of an NTB port in the PCIE switch that is connected to a virtual machine monitor on the server.
  • the method for the computer to execute the instruction for causing the server to perform further includes:
  • the PC IE NTB address space of the destination host corresponding to the server is maintained and managed by the subnet management server, and is configured to receive data sent by the server to the destination host; wherein, all PCIE NTB address spaces are divided into a PCIE NTB address subspace of the same number of hosts connected to the PCIE switch, each of the address subspaces corresponding to a host, for receiving data sent to the corresponding host, at each of the addresses
  • Each of the other hosts is allocated an address space, and each of the other addresses is corresponding to one of the other hosts, and is configured to receive data sent by the other host to the host corresponding to the address subspace, the other
  • the host is a host connected to the PCIE switch in addition to the host corresponding to the address subspace.
  • the method for the computer to execute the instruction for the server to perform further includes:
  • the virtual machine monitor of the server receives and caches PCIE NTB address space information of the subnet management server corresponding to the server, so that the request message for querying the host host PCIE NTB address space can be sent to the subnet management server.
  • the computer executing the instruction, in a method for causing a server to execute, the virtual machine of the destination host The monitor distributes the data to the destination virtual machine of the network request according to the ID of the destination virtual machine in the PCIE packet, including:
  • the virtual machine monitor of the destination host reads the PCIE address in the PCIE NTB address space of the destination host in the PCIE NTB address space of the destination host, according to the destination virtual in the PCIE packet
  • the ID of the machine distributes the data to the destination virtual machine of the network request.
  • the PCIE network-based virtual machine communication method, server, and system according to the present invention have the following beneficial effects:
  • FIG. 1 is a schematic diagram showing the main structure of a virtual machine communication system based on a PCIE network according to the present invention.
  • FIG. 2 is a schematic structural diagram of a host in a virtual machine communication system based on a PCIE network according to the present invention.
  • FIG. 3 is a schematic structural diagram of a virtual machine running on a host in a virtual machine communication system based on a PCIE network according to the present invention.
  • FIG. 4 is a schematic structural diagram of a virtual machine monitor running on a host in a virtual machine communication system based on a PCIE network according to the present invention.
  • FIG. 5 is a schematic structural diagram of a PCIE switch in a virtual machine communication system based on a PCIE network according to the present invention.
  • FIG. 6 is a schematic structural diagram of a subnet management server in a virtual machine communication system based on a PC I E network according to the present invention.
  • FIG. 7 is a schematic flowchart diagram of a PCIE network-based virtual machine communication method according to the present invention.
  • FIG. 8 is a schematic flowchart diagram of step S1 in a PCIE network-based virtual machine communication method according to the present invention.
  • FIG. 9 is a schematic flow chart of address resolution in a virtual machine communication method based on PC I E network according to the present invention.
  • FIG. 10 is a schematic structural diagram of a server according to another embodiment of the present invention. detailed description
  • the PCIE bus is further expanded into a data communication bus between two hosts.
  • the PCIE transport packet is addressed by the address range, that is, when the processor accesses the peripheral device, it reads and writes the address space allocated to the peripheral device, and finally the PCIE bus routes the data packet to the peripheral device through the address.
  • This routing mechanism requires that the address space allocated to peripheral devices in a host system cannot be repeated, but there is no guarantee that the address space will not be duplicated when PCIE interacts between two hosts.
  • the PCIE bus introduces a non-transparent bridge. NTB mechanism.
  • the NTB mechanism converts the address of a data message sent from one host to another by using a preset address translation rule, and converts it into a range of addresses that do not conflict in the target host, thereby realizing data between the hosts of the PCIE bus. routing.
  • NTB is generally integrated into the PCIE switch chip to provide high-bandwidth and low-latency inter-host communication links and features PCIE devices such as interrupt and error handling.
  • the PCIE switch mainly implements the following two functions for the implementation of Host2Host (host-to-host) communication: 1) The reading and writing of the PI0 (Programming Input Output) is identified by the NTB (identity, ID for short). Routing and address routing; 2) For DMA (Direct Memory acces s, direct memory access) is sent and received via ID routing.
  • PI0 Programming Input Output
  • the present invention proposes a PCIE network-based virtual machine communication method, which constructs one for all VMs.
  • the data exchange service of the PC IE network, the host's VMM uses the PC IE NTB's address window and the interrupt mechanism to communicate, thereby accelerating communication between the VMs.
  • This embodiment provides a virtual machine communication system based on a PC IE network.
  • the system includes at least two hosts 10 (Hos t ), one PCIE switch 20 (PC IE Swi t ch ), and a subnet management server. 30;
  • Each host 10 runs a virtual machine monitor (VMM) 111 and at least one virtual machine (VM) 101;
  • the PC IE switch 20 is connected to all hosts to implement data forwarding between the hosts.
  • the subnet management server 30 is connected to the PCIE switch 20 and is used to maintain the correspondence between the host and the virtual machine in the system.
  • the host of the system provided in the embodiment of the present invention may be a general computer, a mobile terminal, a workstation or a server, a dedicated server, etc.
  • the subnet management server may be an ordinary host in the system, or may be a dedicated host. The invention is not specifically limited.
  • the host PCIE switch, and subnet management server are described in further detail below.
  • each of the hosts 10 includes: a virtual machine running module 100, a virtual machine monitoring module 110, and a hardware platform 120.
  • the virtual machine running module 100 and the virtual machine monitoring module 110 respectively provide a hardware platform and a software platform of the virtual machine, and the virtual machine monitoring module 110 includes a virtual machine monitor 111, which is an intermediate layer between the physical hardware and the virtual machine, and is responsible for coordination. Access to the hardware platform by each virtual machine.
  • a plurality of virtual machines 101 are running in the virtual machine running module 100.
  • the virtual machine monitor 111 on each host communicates with all the virtual machines 101 on the host, and sends the data sent by the virtual machine 101 to the PC IE switch 20 in the PCIE protocol.
  • the host's hardware platform 120 may include high speed I/O devices such as CPUs, memory, network cards, storage and interconnect devices, and low speed devices such as basic input and output.
  • the virtual machine runs on the hardware of the host On the platform, the communication inside the network cluster is not perceived to be completed through the PCIE network.
  • the virtual machine 101 is configured to intercept a network request on the TCP/IP protocol stack, and send the data included in the network request to the virtual machine monitor 111.
  • the virtual machine 101 is configured to intercept the network request on the TCP/1 P protocol stack, obtain data included in the network request, and put the data included in the network request into its corresponding virtual machine.
  • the packet buffers the queue and notifies the virtual machine monitor 111.
  • the virtual machine 101 can intercept and perform subsequent processing only for the network request of the data type message, and the network request of the non-data type message is further processed according to the original TCP/IP protocol stack mode.
  • the virtual machine monitor 111 is configured to determine, after receiving the network request sent by the virtual machine 101, the PCIE NTB address space of the destination host corresponding to the source host on the PCIE switch according to the destination address of the network request, and include the The PCIE packet of the data in the network request is sent to the PCIE NTB address space of the destination host corresponding to the source host on the PCIE switch, so that the virtual machine monitor of the destination host is in the PCIE packet.
  • the destination virtual machine's ID distributes the data to the destination virtual machine of the network request.
  • the destination host is a host to which the destination virtual machine of the network belongs, and the PCIE NTB address space of the destination host corresponding to the source host on the PCIE switch is allocated to the destination host on the PCIE switch.
  • the PCIE NTB address space in the PCIE NTB address space corresponds to the source host.
  • the VMM 111 is configured to: after receiving the network request sent by the virtual machine 101, determine the PCIE NTB address of the destination host corresponding to the source host according to the destination address in the network request, that is, the IP address of the destination virtual machine requested by the network. Space, that is, the PCIE NTB address space of the destination host that needs to be written when the source host communicates with the destination host (here, the source VM on the source host communicates with the destination VM on the destination host).
  • the host where the virtual machine that issued the network request is located is referred to as the source host
  • the destination host the host where the destination VM is requested by the network.
  • the V 111 may determine the network.
  • the corresponding PCIE NTB address space corresponding to the destination host is requested, that is, the PCIE NTB address space of the destination host that the source host should write.
  • the virtual machine 101 sends an Address Resolution Protocol (ARP) request to perform the address resolution protocol (ARP) request.
  • ARP Address Resolution Protocol
  • the VMM 111 caches the PC IE NTB address space of the destination host corresponding to the source host.
  • the virtual machine 101 when the virtual machine 101 is to issue a TCP/IP network request, if the virtual machine 101 does not have the MAC address information of the destination virtual machine requested by the network, the virtual machine 111 sends an ARP request according to the TCP/IP protocol to perform the address. Analysis. Specifically:
  • the virtual machine 101 is further configured to send an Ethernet ARP request to the destination virtual machine requested by the network when there is no MAC address information of the destination virtual machine requested by the network on the virtual machine 101.
  • the ARP request contains the IP address of the destination virtual machine.
  • the VLi 111 is further configured to: after receiving the ARP request sent by the virtual machine 101, initiate a request message for querying the PCIE NTB address space corresponding to the destination virtual machine to the subnet management server 30, where the request message has a destination virtual The IP address of the machine, and after receiving the response message of the PCIE NTB address space information of the destination host corresponding to the source VM returned by the subnet management server 30, the destination host corresponding to the source VM The PCIE NTB address space information is cached locally.
  • the subnet management server 30 is configured to: after receiving the request message of the PCIE NTB address space corresponding to the virtual machine that is inquired by the VMM, determine the destination host to which the destination virtual machine belongs according to the correspondence between the host and the virtual machine, and obtain the source host. Corresponding to the PC IE NTB address space of the destination host, a response message is sent to the virtual machine monitor 111.
  • the subnet management server 30 is configured to maintain and manage all PCIE NTB address spaces, and the PCIE NTB address space is configured on a PCIE switch connected to the source host and the destination host, and the PCIE NTB address space is divided into PCIE switches.
  • the number of connected hosts is the same number of PCIE NTB address subspaces, and each of the address subspaces corresponds to one host, and is used for receiving data sent to the host, in the address subspace corresponding to each host for each other
  • the host allocates an address micro space, and each of the address micro spaces corresponds to one other host, for receiving data from the other host corresponding to the host corresponding to the address subspace, where the other hosts are A host connected to a PCIE switch, other than the host corresponding to the address subspace.
  • a PCIE switch can access N hosts (host 1 and host N)
  • the PCIE NTB address space maintained by the subnet management server will be equally divided into N PCIE NTB address subspaces, and one PCIE NTB address subspace correspondingly.
  • a host Further, a PCIE NTB address subspace can further separate N-1 PCIE NTB address microspaces, and each PCIE NTB address microspace corresponds to one other host, and the host and the host write the specified PCIE NTB address space.
  • Can achieve each other Communication for example, the entire PC I ⁇ NTB address space is divided into N PC IE NTB address subspaces, host
  • the host 2 when the host 2 communicates with the host 1, the host 2 writes data to the H2 area (address micro space) in the PC IE NTB address space (address subspace) corresponding to the host 1 to complete the data from the host 2 to the host.
  • the subnet management server 30 notifies all Hos t's own address space in the network cluster when the PCIE network is initialized. In this embodiment of the present invention, multiple hosts can communicate with one host concurrently through the mechanism of the address space.
  • the subnet management server 30 is further configured to maintain a correspondence between each host and a virtual machine running on the host, and update the corresponding relationship in real time according to the creation, modification, and destruction of the virtual machine.
  • the mapping maintained by the subnet management server may be a mapping relationship between an IP address or a virtual machine ID, or another type of mapping relationship, for example, a correspondence between an IP address of the host and an ID of the virtual machine. , or the correspondence between the PC IE NTB address space corresponding to the host and the ID of the virtual machine.
  • Each VM's creation, destruction, and change will notify the subnet management server to refresh its mappings.
  • the subnet management server may be a physical host or a virtual machine.
  • the subnet management server 30 is configured to receive the request message sent by the source V, and determine, according to the IP address of the destination virtual machine carried in the message, and the corresponding relationship between the destination host and the destination virtual machine saved and maintained locally.
  • the destination host to which the virtual machine belongs and further determines the PCIE NTB address space of the destination host corresponding to the source VM, that is, the PCIE NTB address space of the destination VM corresponding to the source VM, and sends a response message to the source virtual machine monitor, corresponding to the source VM.
  • the destination host's PCIE NTB address space information is returned to the source V.
  • the virtual machine monitor 111 is further configured to continue the ARP process of the source VM, and the subsequent ARP process is similar to the existing ARP process:
  • the virtual machine monitor sends the ARP request of the Ethernet to the destination virtual machine of the network request, that is, the source V MN sends the ARP request to the V MN of the host where the destination virtual machine is located, and the virtual machine monitors the destination host.
  • the device After receiving the ARP request, the device sends the ARP request to the destination virtual machine; the destination virtual machine returns an ARP response, where the MAC address of the destination virtual machine is included; and the source virtual machine receives the destination virtual machine The ARP response sent, and the ARP address resolution process of the destination VM is completed.
  • the virtual machine monitor 111 is configured to send the network request to the PCIE of the destination host.
  • the PCIE NTB address space corresponding to the source host in the NTB address space is specifically: the virtual machine monitor 111, configured to put the data, the source virtual machine ID, and the source NTB port ID included in the network request into the PCIE packet.
  • the data payload, the ID of the destination virtual machine requested by the network is used as the destination address of the PCIE packet, and the PCIE packet is sent to the PCIE switch 20, so that the PCIE switch 20 writes the data to the destination host.
  • the PCIE NTB address space corresponds to the PCIE NTB address space of the source host.
  • the source virtual machine ID is an ID of a virtual machine that sends the data
  • the source NTB port ID is an ID of an NTB port of the PCIE switch that is connected to a virtual machine monitor on the source host.
  • the virtual machine monitor of the destination host is further configured to distribute the data to the destination virtual machine according to the ID of the destination virtual machine. Specifically, the virtual machine monitor of the destination host reads the data in the PC IE NTB address space corresponding to the source host in the PC IE NTB address space corresponding to the destination host, and the ID of the virtual machine according to the destination of the data. Data is distributed to a buffer queue of the destination virtual machine of the data, and the destination virtual machine of the data is notified to receive data from the buffer queue.
  • the virtual machine 101 includes: a network request interception module 1011, a kernel protocol module 1012, and a virtual device driver module 1013;
  • the network request interception module 1011 is configured to intercept the network request on the TCP/IP protocol stack.
  • the kernel protocol module 1012 is communicatively coupled to the network request interception module 1011 for processing the network request intercepted by the network interception module 1011.
  • the specific processing procedure is as follows: The kernel protocol module 1012 receives the network request, parses the network request, and acquires data included in the network request.
  • the virtual device driver module 1013 is communicably connected to the kernel protocol module 1012, and configured to put data included in the network request into a data packet buffer queue corresponding to the virtual machine, and notify the virtual machine A virtual machine monitor running on the same host. Specifically, the virtual device driver module 1013 in the virtual machine puts the data included in the network request into a packet buffer queue (hereinafter referred to as a buffer queue) that is supervised by the virtual machine monitor and belongs to the virtual machine. Also inform the virtual machine monitor that there is a new request to send. Each virtual machine has a private buffer ⁇ ' J. Since each virtual machine can have multiple virtual machines, the virtual machine monitor on each host supervises all the virtual machines on the host. Punch the queue. Further, as shown in FIG. 4, the virtual machine monitor 111 includes a PCIE packet distributor 1111 and a PCIE communication module 1112 that are in communication with the virtual machine 101.
  • the PCIE data packet distributor 1111 is configured to put the data included in the network request, the source virtual machine ID, and the source NTB port ID into a data payload of the PCIE packet, and the destination virtual machine of the network request
  • the ID is used as the destination address of the PCIE message.
  • the virtual machine that intercepts the network request is called the source virtual machine, and the virtual machine that receives the network request is the destination virtual machine, the host to which the source virtual machine belongs is the source host, and the host to which the destination virtual machine belongs is the destination host, the source virtual machine.
  • the ID is the ID of the virtual machine that sends the data
  • the source NTB port ID is the ID of the NTB port connected to the virtual machine monitor on the source host. Since multiple VMs may be running on a single host, the function of the PCIE packet distributor 1111 is to distinguish the data flow of each VM by adding the source virtual machine ID and the source NTB port ID to the network request of a VM.
  • the PCIE communication module 1112 is configured to send the PCIE packet generated by the PCIE packet distributor 1111 to the PCIE switch 20, so that the PCIE switch 20 writes the data to a PCIE NTB address corresponding to the destination host.
  • the PCIE communication module 1112 on the virtual machine monitor 111 on the destination host is further configured to receive the PCIE message from the PCIE switch 20, and the PCIE data packet distributor 1111 is further configured to receive the PCIE communication module 1112.
  • the PCIE packet distributes the data in the PCIE packet to the buffer queue of the destination virtual machine corresponding to the packet according to the destination VM ID in the packet.
  • the PCIE switch 20 mainly includes a non-transparent bridge connected to the host Host.
  • NTB NTB provides an address window mechanism for sending and receiving data between hosts.
  • the PCIE switch 20 includes at least two NTB ports, and the NTB port is connected to the host and the subnet management server. For example, in FIG.
  • the host 1 writes the address window of the NTB of the host 1, and the data is sent by the PCIE switch 20 from the NTB of the host 1 to the NTB of the host 2 in the form of data packets, and finally sent to the memory of the host 2
  • the host 1 can also write the doorbell register corresponding to the host 2, so that the NTB connected to the host 2 sends an interrupt to the host 2;
  • the PCIE switch 20 has more advanced remote direct access (remote direct memory access, Referred to as RDMA) function, with this function, PCIE switch 20 can realize copy memory between hosts connected by NTB.
  • the subnet management server 30 includes a host and virtual machine correspondence management module 301, a PCIE NTB address space management module 302, and a PCIE communication module 303.
  • the host and virtual machine correspondence management module 301 is configured to maintain a correspondence between the host and the virtual machine in the system. Specifically, the host and virtual machine correspondence management module 301 maintains a correspondence between each Hos t in the current system and a VM running thereon.
  • the mapping between the host and the virtual machine correspondence management module 301 may be an IP mapping relationship or an ID mapping relationship, or other types of mapping relationships, such as a affiliation relationship between the IP address of the host and the ID of the virtual machine or a host corresponding to the host. Correspondence between the PC IE NTB address space and the ID of the virtual machine.
  • the host and virtual machine correspondence management module 301 updates the correspondence between the host and the virtual machine in real time according to the creation, change, and destruction of the virtual machine.
  • the PCIE NTB address space management module 302 is configured to maintain and manage a PCIE NTB address space pre-configured for each host. All PC IE NTB address spaces are divided into PCIE NTB address sub-spaces of the same number of hosts connected to the PC IE switch 20, and each of the address sub-spaces corresponds to a host for receiving and transmitting to the address sub-address.
  • the data of the host corresponding to the space further, assigning an address micro space to the other host in each of the address subspaces, each of the address micro spaces corresponding to one of the other hosts, for receiving from the other
  • the PC IE NTB address space maintained by the subnet management server will be equally divided into N PC IE NTB address subspaces, and one PCIE NTB address subcard.
  • the space corresponds to a host.
  • a PC IE NTB address subspace can further separate N-1 PCIE NTB address microspaces, and each PC IE NTB address micro space corresponds to one other host, and the specified PC IE NTB address is written between the host and the host.
  • the space can be communicated with each other. For example, the entire PC IE NTB address space is divided into N PCIE NTB address subspaces.
  • the PCIE NTB address subspace corresponding to host 1 is divided into N-1 address spaces, which are respectively identified as ⁇ 2, ⁇ 3. ⁇
  • the host 1 writes data to the H2 area (address micro space) in the PC I ⁇ ⁇ address space (address subspace) corresponding to the host 1 to complete the data from the host 2 The delivery of host 1.
  • the PCIE NTB address space management module 302 notifies the address space corresponding to all Hos t subnet management servers 30 in the network cluster when the PC IE network is initialized, and each host
  • the VMM caches the address space information of the subnet management server corresponding to the host, so that the VMM can correctly send the PCIE NTB address of the destination host corresponding to the source VM to the subnet management server when receiving the address resolution request of the source VM. Space request message.
  • the PCIE communication module 303 is configured to communicate with the PCIE switch 20 to send and receive PCIE messages.
  • the PCIE communication module 303 is configured to receive a request message sent by the source VLi through the PCIE switch, and the host and the virtual machine correspondence management module 301 is configured to: according to the IP address of the destination virtual machine carried in the request message, And determining, by the mapping between the destination host and the destination virtual machine, the destination host to which the destination virtual machine belongs; the PCIE NTB address space management module 302, configured to determine, according to the information of the destination host and the source host, the source VM The PCIE NTB address space of the destination host, that is, the PCIE NTB address space of the destination VM corresponding to the source VM; the PCIE communication module 303 sends a response message to the source virtual machine monitor, and the PCIE NTB address space of the destination host corresponding to the source VM The information is returned to the source V.
  • the invention uses the PCIE network as a communication mechanism between virtual machines, skips the transmission of the Ethernet, and utilizes the advantages of high bandwidth and low delay of the PCIE network, greatly reduces the transmission delay, and realizes the fast between virtual machines. Communication. Moreover, the invention also breaks through the limitation of the PCIE network, realizes the subnet management based on the PC IE network, so that the virtual machine on all the hosts in the network bypasses the TCP/1 P protocol stack in the OS, and maintains the host in the system. The corresponding relationship with the virtual machine realizes unified management and maintenance of the entire system, and further improves the integrity of the virtual machine communication system based on the PCIE network. In addition, the present invention realizes the sharing of the PCIE NTB address space by the virtual machine through the subnet management server, solves the concurrent read and write problem of the PCIE network, and further improves the data receiving speed between the virtual machines.
  • the embodiment further provides a PCIE network-based virtual machine communication method, which may be implemented by the PCIE network-based virtual machine communication system, but the implementation structure is not limited to the system described in the embodiment.
  • the PCIE network-based virtual machine communication method includes:
  • the source virtual machine on the source host intercepts the network request on the TCP/IP protocol stack, and sends the data contained in the network request to the virtual machine monitor on the source host.
  • the step S1 includes:
  • the source virtual machine intercepts the network request on the TCP/IP protocol stack.
  • the network request of the data type packet may be intercepted and subsequently processed, and the network request of the non-data type message is further processed according to the original TCP/IP protocol stack manner.
  • the control packet is still controlled. All of the data is forwarded to the existing TCP/IP protocol stack for subsequent processing, and the intercepted data type packet is subjected to subsequent processing in the embodiment of the present invention.
  • the source virtual machine parses the network request, acquires data included in the network request, and notifies the virtual machine monitor on the source host.
  • the virtual machine monitor on the source host determines, according to the destination address of the network request, a PC IE NTB address space of the destination host corresponding to the source host on the PCIE switch, where
  • the PC IE NTB address space of the destination host is a PCIE NTB address space allocated to the destination host on the PC IE switch, and the PCIE NTB address space of the destination host corresponding to the source host on the PCIE switch is allocated to the PCIE switch.
  • the PCIE NTB address space of the destination host corresponds to the PCIE NTB address space of the source host.
  • the VMM on the source host After receiving the network request, the VMM on the source host determines the destination host corresponding to the network request according to the destination address in the network request (that is, the IP address of the destination virtual machine requested by the network) (the physical host to which the destination virtual machine belongs) And further determining the PCIE NTB address space of the destination host corresponding to the source host, that is, determining that the source host communicates with the destination host (here, the source VM on the source host communicates with the destination VM on the destination host) Write the PCIE NTB address space of the destination host.
  • the PCIE NTB address space is configured on a PCIE switch that is in communication with the source and destination hosts.
  • the VMM on the source host is used.
  • the PCIE NTB address space corresponding to the destination host corresponding to the network request may be determined, that is, the source host corresponds to the PC IE NTB address space of the destination host to be written.
  • the VMM on the source host caches the PCIE NTB address space of the destination host corresponding to the source host, where the destination host is the IP address of the destination virtual machine requested by the network. The physical host to which the address corresponds.
  • the source virtual machine when the source virtual machine is to be sent When the TCP/IP network request is made, if the source virtual machine does not have the MAC address information of the destination virtual machine requested by the network, the source virtual machine sends an ARP request for address resolution according to the TCP/IP protocol, as shown in Figure 9, where the source VLi refers to the virtual machine monitor on the source host, and the destination V refers to the virtual machine monitor on the destination host.
  • the specific implementation process of address resolution is:
  • the source virtual machine sends an Ethernet ARP request to the destination virtual machine requested by the network by using a virtual machine monitor on the source host, where the ARP request is a standard request in the TCP/IP protocol, and the IP address of the destination virtual machine is included therein. .
  • the source VMM After receiving the ARP request with the destination virtual machine IP address, the source VMM initiates a request message for querying the PCIE NTB address space corresponding to the target virtual machine to the subnet management server, where the request message carries the IP address of the destination virtual machine.
  • the source VMM stores the PCIE NTB address space information of the subnet management server, and sends a request message for querying the PCIE NTB address space of the destination host corresponding to the source VM.
  • the subnet management server determines, according to the correspondence between the host and the virtual machine that is maintained and managed by the maintenance, the destination host to which the destination virtual machine belongs, and the PC I E NTB address space of the destination host corresponding to the source host.
  • the subnet management server maintains and manages all PCIE NTB address spaces, and all PCIE NTB address spaces are divided into PCIE NTB address subspaces of the same number of hosts connected to the PCIE switch, and each of the address subspaces corresponds to one host.
  • the other host sends the data of the host corresponding to the address subspace.
  • the other host mentioned here is a host connected to the PC IE switch except the host corresponding to the address subspace.
  • the PCIE NTB address space maintained by the subnet management server will be equally divided into N PCIE NTB address subspaces, and one PCIE NTB address subspace correspondingly.
  • a host Further, a PCIE NTB address subspace can further separate N-1 PCIE NTB address microspaces, and each PCIE NTB address microspace corresponds to one other host, and the host and the host write the specified PC IE NTB address space.
  • the method can communicate with each other. For example, the entire PCIE NTB address space is divided into N PCIE NTB address subspaces, and the PCIE NTB address subspace corresponding to host 1 is divided into N-1 address microspaces.
  • H2 HN When host 1 communicates with host 1, host 1 can write data by writing data to H2 area (address micro space) in PCIE NTB address space (address subspace) corresponding to host 1. The transfer from the host 2 to the host 1.
  • H2 area address micro space
  • PCIE NTB address space address subspace
  • the subnet management server maintains a corresponding relationship between each host and the virtual machine running thereon, and updates the corresponding relationship in real time according to the creation, modification, and destruction of the virtual machine.
  • the mapping maintained by the subnet management server may be a mapping relationship between an IP address or a virtual machine ID, or another type of mapping relationship, for example, a correspondence between an IP address of the host and an ID of the virtual machine. , or the correspondence between the PC IE NTB address space corresponding to the host and the ID of the virtual machine.
  • the subnet management server receives the request message sent by the source V, and the subnet management server determines, according to the IP address of the destination virtual machine carried in the message, and the corresponding relationship between the destination host and the destination virtual machine saved and maintained locally.
  • the destination host to which the virtual machine belongs and further determines the PCIE NTB address space of the destination host corresponding to the source VM, that is, the PCIE NTB address space of the destination VM corresponding to the source VM.
  • the subnet management server sends a response message to the source virtual machine monitor, and returns the PC IE NTB address space information of the destination host corresponding to the source VM to the source V.
  • the virtual machine monitor on the source host caches the PCIE NTB address space information of the destination host corresponding to the source VM. That is, the PC IE NTB address space information of the destination host corresponding to the source host is cached.
  • the ARP process of the source VM is continued, and the subsequent ARP process is similar to the existing ARP process:
  • the virtual machine monitor on the source host sends an ARP request of the Ethernet to the destination virtual machine of the network request, that is, the source VMM sends an ARP request to the VMM of the host where the destination virtual machine is located. After receiving the ARP request, the virtual machine monitor on the host sends the ARP request to the destination virtual machine.
  • the destination virtual machine returns an ARP response, where the source MAC address of the destination virtual machine is included.
  • the virtual machine receives the ARP response sent by the destination virtual machine, and the ARP address resolution process of the destination virtual machine is completed.
  • the subnet management server pre-configures the PCIE NTB address space when the PCIE network is initialized, and notifies all hosts in the subnet to the address space information corresponding to the subnet management server, and each host receives the subnet management server. After the information, the address space information of the subnet management server corresponding to it is cached, so that the request message for querying the corresponding PCIE NTB address space can be correctly sent to the subnet management server.
  • the virtual machine monitor on the source host sends a PCIE packet containing the data in the network request to a PCIE NTB address space of the destination host corresponding to the source host on the PCIE switch, so that The virtual machine monitor of the destination host distributes the data to the destination virtual machine requested by the network according to the ID of the destination virtual machine in the PCIE message.
  • the virtual machine monitor on the source host sends the network request to the PCIE NTB address space corresponding to the source host in the PCIE NTB address space of the destination host, and the detailed implementation process is as follows:
  • the virtual machine monitor puts the data contained in the network request, the ID of the source virtual machine, and the source NTB port ID into a data payload of the PCIE packet, and uses the ID of the destination virtual machine requested by the network as the PCIE.
  • the destination address of the message sending the PCIE message to the PCIE switch, causing the PCIE switch to write the data to the PCIE NTB address space of the destination host in the PCIE NTB address space corresponding to the source host .
  • the source NTB port ID is an ID of an NTB port of the PCIE switch connected to a virtual machine monitor on the source host.
  • the PCIE network-based virtual machine communication method further includes a step S4, wherein the virtual machine monitor of the destination host distributes the data to the destination virtual machine according to the ID of the destination virtual machine.
  • the virtual machine monitor of the destination host reads data in the PCIE NTB address space corresponding to the source host in the PCIE NTB address space corresponding to the destination host, according to the The ID of the destination virtual machine distributes the data to a buffer queue of the destination virtual machine of the data, and notifies the destination virtual machine of the data to receive data from the buffer queue. That is, after receiving the doorbe ll interrupt, the VMM of the destination host will quickly parse the buffer data, and distribute the data to the buffer queue of the target VM according to the target VM's doma in ID, and the target VMM will The injection interrupt informs the target VM that new data is coming.
  • the target VM will eventually The data is received from the buffer queue by a virtual device driver module in its kernel.
  • Each virtual machine has a private buffer queue. Since there are multiple virtual machines on each host, the virtual machine monitor in each host supervises all the virtual machine private buffer queues on the host.
  • the embodiment of the present invention maintains the correspondence between the host and the virtual machine in the PCIE network by using the subnet management server, implements the address resolution function of the target VM, further realizes unified management and maintenance of the entire PCIE network, and improves the integration of the PCIE network. .
  • the server 40 of this embodiment includes at least one processor 401, a memory 405, at least one network interface card 404, and at least one communication bus 402.
  • the server 40 optionally includes a user interface 403 including a display, a keyboard or a pointing device.
  • the memory 405 stores execution instructions.
  • the processor 401 communicates with the memory 405, and the processor 401 executes the execution instructions so that the server 40 executes the host implementation method described in the previous embodiment of the present invention, specifically: As shown in FIG.
  • the source virtual machine on the server intercepts the network request on the TCP/IP protocol stack, and sends the data contained in the network request to the virtual machine monitor on the server; the virtual machine monitoring on the server After receiving the network request, determining, according to the destination address of the network request, the PCIE NTB address space of the destination host corresponding to the server on the PCIE switch; the virtual machine monitor on the server will include the network request
  • the PCIE packet of the data is sent to the PCIE NTB address space of the destination host corresponding to the server on the PCIE switch, so that the virtual machine monitor of the destination host according to the ID of the destination virtual machine in the PCIE packet
  • the data is distributed to the destination virtual machine requested by the network.
  • Operating system 406 which contains various programs for implementing various basic services and handling hardware-based tasks.
  • the server 40 also includes a display interface card.
  • the embodiment of the present invention further provides a computer readable medium, which includes a computer execution instruction, and the computer execution instruction enables the server 40 to execute the host implementation method described in the previous embodiment of the present invention, and the implementation principle and technical effect thereof are similar. , will not repeat them here.
  • the server 40 in this embodiment may further replace the host 10 in the virtual machine communication system of the PCIE network in the foregoing embodiment, and implement the functions of the host 10 in the foregoing embodiment, as a virtual machine communication system constituting the PCIE network. Part of communicating with other components in the virtual machine communication system of the PCIE network, enabling the virtual machine communication system of the PCIE network to be implemented
  • the PCIE network acts as a communication mechanism between virtual machines to achieve fast communication between virtual machines.
  • the implementation principle and technical effects are similar to those of the communication system and the communication method described in the previous embodiments, and are not described herein again.
  • the invention avoids the delay caused by the TCP/IP protocol stack and the performance loss of the host, utilizes the advantages of low latency and high bandwidth of the PCIE network, greatly reduces the transmission delay, and realizes fast communication between virtual machines;
  • the standard-based PC IE implementation solves the concurrent read and write problems of the PCIE network, and further improves the speed of data reception between virtual machines.
  • the present invention effectively overcomes various shortcomings in the prior art and has a high industrial utilization value.
  • Computer readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one location to another.
  • a storage medium may be any available media that can be accessed by a computer.
  • the computer readable medium can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage media or other magnetic storage device, or can be used to carry or store an instruction or data structure.
  • Any connection may suitably be a computer readable medium.
  • the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave
  • coaxial cable , fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, wireless, and microwaves are included in the fixing of the associated media.
  • a disc (Di sk ) and a disc (CD) include a compact disc (CD), a laser disc, a compact disc, a digital versatile disc (DVD), a floppy disc, and a Blu-ray disc, wherein the disc is usually magnetically replicated, and The disc uses a laser to optically replicate the data. Combinations of the above should also be included within the scope of the computer readable media.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)
  • Computer And Data Communications (AREA)

Abstract

L'invention concerne un système et un procédé de communication entre machines virtuelles basés sur un réseau PCIe. Dans ledit système : un hôte source comprend un hyperviseur et au moins une machine virtuelle, l'hyperviseur sur l'hôte source étant connecté à la machine virtuelle sur l'hôte source de manière à pouvoir communiquer avec elle ; une machine virtuelle sur l'hôte source, qui sert de machine virtuelle source, traite une demande réseau interceptée sur une pile de protocoles TCP/IP et envoie cette demande à l'hyperviseur sur l'hôte source ; et l'hyperviseur sur l'hôte source envoie les données contenues dans la demande réseau à un espace adresse PCIe NTB correspondant à l'hôte source dans l'espace adresse PCIe NTB correspondant à un hôte cible auquel la machine virtuelle cible est subordonnée, selon l'ID d'une machine virtuelle cible de la demande réseau. La présente invention utilise le réseau PCIe comme mécanisme de communication entre des machines virtuelles, ce qui permet une communication rapide entre les machines virtuelles, et le réseau PCIe n'est plus limité car l'espace adresse PCIe NTB dans un serveur de gestion de sous-réseau est géré et partagé sur la base d'un sous-réseau dudit réseau PCIe, améliorant ainsi la vitesse de communication entre les machines virtuelles.
PCT/CN2013/070016 2013-01-04 2013-01-04 Procédé, serveur et système de communication entre machines virtuelles basés sur un réseau pcie WO2014106321A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201380000042.9A CN103384551B (zh) 2013-01-04 2013-01-04 一种基于pcie网络的虚拟机通信方法、服务器及系统
PCT/CN2013/070016 WO2014106321A1 (fr) 2013-01-04 2013-01-04 Procédé, serveur et système de communication entre machines virtuelles basés sur un réseau pcie

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2013/070016 WO2014106321A1 (fr) 2013-01-04 2013-01-04 Procédé, serveur et système de communication entre machines virtuelles basés sur un réseau pcie

Publications (1)

Publication Number Publication Date
WO2014106321A1 true WO2014106321A1 (fr) 2014-07-10

Family

ID=49492133

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2013/070016 WO2014106321A1 (fr) 2013-01-04 2013-01-04 Procédé, serveur et système de communication entre machines virtuelles basés sur un réseau pcie

Country Status (2)

Country Link
CN (1) CN103384551B (fr)
WO (1) WO2014106321A1 (fr)

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9672167B2 (en) 2013-07-22 2017-06-06 Futurewei Technologies, Inc. Resource management for peripheral component interconnect-express domains
CN103763173B (zh) * 2013-12-31 2017-08-25 华为技术有限公司 数据传输方法和计算节点
TWI540871B (zh) * 2014-04-29 2016-07-01 緯創資通股份有限公司 混合式資料傳輸之方法及其相關混合式系統
CN104486365B (zh) * 2014-09-28 2018-02-02 浪潮(北京)电子信息产业有限公司 双控之间通信方法和系统
CN105786398B (zh) * 2014-11-18 2019-04-05 财团法人工业技术研究院 存储器映射方法和存储器映射系统
CN105704098B (zh) * 2014-11-26 2019-03-01 杭州华为数字技术有限公司 一种虚拟化网络的数据传输方法，节点控制器及系统
CN108351795A (zh) * 2015-10-30 2018-07-31 华为技术有限公司 用于映射虚拟机通信路径的方法和系统
CN109688062B (zh) * 2017-10-19 2021-07-13 中兴通讯股份有限公司 一种路由方法和路由设备
CN112398817B (zh) * 2018-01-23 2022-02-25 华为技术有限公司 数据发送的方法及设备
WO2019161557A1 (fr) * 2018-02-24 2019-08-29 华为技术有限公司 Procédé et appareil de communication
CN109002414A (zh) * 2018-07-26 2018-12-14 郑州云海信息技术有限公司 一种跨节点虚拟机通信方法及相关装置
CN109213574A (zh) * 2018-09-26 2019-01-15 郑州云海信息技术有限公司 一种用于存储系统之间的信息传输方法、装置及介质
CN109460286A (zh) * 2018-11-05 2019-03-12 郑州云海信息技术有限公司 基于ntb硬件的虚拟机通信方法、装置、设备及存储介质
CN114389995A (zh) * 2021-12-03 2022-04-22 阿里巴巴（中国）有限公司 资源共享的方法、装置以及电子设备
CN117827726B (zh) * 2024-03-06 2024-05-31 井芯微电子技术(天津)有限公司 非透明桥传输的实现方法、装置、电子设备和存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070288921A1 (en) * 2006-06-13 2007-12-13 King Steven R Emulating a network-like communication connection between virtual machines on a physical device
US20080148281A1 (en) * 2006-12-14 2008-06-19 Magro William R RDMA (remote direct memory access) data transfer in a virtual environment
CN101470633A (zh) * 2007-12-24 2009-07-01 联想(北京)有限公司 一种虚拟机监视器、虚拟机系统及其内存处理方法
CN102707991A (zh) * 2012-05-17 2012-10-03 中国科学院计算技术研究所 多根i/o 虚拟化共享方法和系统

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101299757A (zh) * 2008-05-23 2008-11-05 华为技术有限公司 一种数据共享方法及通讯系统以及相关设备
US8463934B2 (en) * 2009-11-05 2013-06-11 Rj Intellectual Properties, Llc Unified system area network and switch

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070288921A1 (en) * 2006-06-13 2007-12-13 King Steven R Emulating a network-like communication connection between virtual machines on a physical device
US20080148281A1 (en) * 2006-12-14 2008-06-19 Magro William R RDMA (remote direct memory access) data transfer in a virtual environment
CN101470633A (zh) * 2007-12-24 2009-07-01 联想(北京)有限公司 一种虚拟机监视器、虚拟机系统及其内存处理方法
CN102707991A (zh) * 2012-05-17 2012-10-03 中国科学院计算技术研究所 多根i/o 虚拟化共享方法和系统

Also Published As

Publication number Publication date
CN103384551B (zh) 2015-12-09
CN103384551A (zh) 2013-11-06

Similar Documents

Publication Publication Date Title
WO2014106321A1 (fr) Procédé, serveur et système de communication entre machines virtuelles basés sur un réseau pcie
US11934341B2 (en) Virtual RDMA switching for containerized
US9893977B2 (en) System and method for supporting live migration of virtual machines in a virtualization environment
US8774055B2 (en) Switching method
US9329783B2 (en) Data processing system and data processing method
US7996569B2 (en) Method and system for zero copy in a virtualized network environment
US20140032796A1 (en) Input/output processing
US9154451B2 (en) Systems and methods for sharing devices in a virtualization environment
US9864717B2 (en) Input/output processing
US11940933B2 (en) Cross address-space bridging
US20070050520A1 (en) Systems and methods for multi-host extension of a hierarchical interconnect network
US11693804B2 (en) Cross bus memory mapping
CN111221758A (zh) 处理远程直接内存访问请求的方法和计算机设备
WO2021051919A1 (fr) Puce de transfert de données et serveur
WO2012141695A1 (fr) Traitement de données d'entrée/de sortie
US10397140B2 (en) Multi-processor computing systems
US11979459B1 (en) Configuration of data connections between a host and a shared network adapter

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 13869867

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 13869867

Country of ref document: EP

Kind code of ref document: A1