CN103873489A

CN103873489A - Device sharing system with PCIe interface and device sharing method with PCIe interface

Info

Publication number: CN103873489A
Application number: CN201210525640.5A
Authority: CN
Inventors: 杨佳儒
Original assignee: Hongfujin Precision Industry Shenzhen Co Ltd; Hon Hai Precision Industry Co Ltd
Current assignee: Hongfujin Precision Industry Shenzhen Co Ltd; Hon Hai Precision Industry Co Ltd
Priority date: 2012-12-10
Filing date: 2012-12-10
Publication date: 2014-06-18

Abstract

A device sharing system with a PCIe interface, running in a requesting server and a sharing server that provides a PCIe interface device as a device to be shared, including: the sharing drive obtains the access of the virtual machine in the requesting server to the device to be shared, and the sharing management will The model of the device to be shared and the memory address of the PCIe base address register of the device to be shared are sent back to the sharing agent. The shared driver creates a first window in the designated storage space of the virtual machine to map to the PCIe base address register memory of the device to be shared. The sharing management establishes a second window in the storage device of the sharing server to be mapped to a specified storage space in the virtual machine. The invention also provides a device sharing method with a PCIe interface. The invention can realize sharing of devices with PCIe interfaces among multiple servers.

Description

There is device shared system and the method for PCIe interface

Technical field

The present invention relates to hardware technology of sharing, relate in particular to a kind of device shared system and method with quick peripheral component interconnection PCIe interface.

Background technology

Input-output equipment can be used to share between multiple devices, but, in the time that a device need to be shared the input-output equipment of other devices, often need to revise the corresponding driver of this device local side.For example, take network interface card as example, when server wants is used the network interface card on other servers to realize transfer of data, need to use specific driver, must cause modifying and could realizing calling dissimilar network interface card the NIC driver of local side, and the final device with PCIe interface of realizing shares, so, must cause the service efficiency of network interface card low.Or, if this server is not modified to the NIC driver of local side, so, must only can use the network interface card corresponding with this NIC driver, must make the network interface card type that can call be very restricted, thereby affect network interface card sharing between multiple devices.

Summary of the invention

In view of above content, be necessary to provide a kind of device shared system and method for the PCIe of having interface, can realize and between multiple devices, carry out sharing of network interface card.

A kind of device shared system with quick peripheral component interconnection PCIe interface, run in multiple servers, at least one request server is connected with another share service device, described share service device provides has the device of PCIe interface and using the device of the described PCIe of having interface as treating sharing means, this system comprises shares driving, share agency and Sharing Management, wherein: described shared driving, the access of the configuration space of the PCIe that treats sharing means for the virtual machine that obtains described request server to described share service device is to treat the model of sharing means described in obtaining, and this access is passed to described shared agency, described shared agency, for sending this access to the Sharing Management of described share service device, described Sharing Management, for receiving this access, treats this model of sharing means and this is treated to the memory address of the base register BAR of the PCIe of sharing means returns to described shared agency, described shared agency, also for this is treated to the model of sharing means feeds back to described virtual machine, described virtual machine treats that according to this model of sharing means determines that corresponding this treat the driver of sharing means, described shared driving, also sets up first window for the designated memory space at described virtual machine, treats that according to this memory address of the base register of the PCIe of sharing means is mapped to this first window the base register internal memory of this PCIe that treats sharing means, and described Sharing Management, also, for setting up Second Window at the storage device of described share service device, this Second Window is mapped in the designated memory space in described virtual machine.

A kind of device with quick peripheral component interconnection PCIe interface is shared the shared method of the device with PCIe interface that method is carried out, run in multiple servers, at least one request server is connected with another share service device, described share service device provides has the device of PCIe interface and using the device of the described PCIe of having interface as treating sharing means, the method comprises the steps: to share the access that drives the configuration space that obtains virtual machine in the described request server PCIe that treats sharing means to described share service device to treat the model of sharing means described in obtaining, and this access is passed to described shared agency, share agency and send this access to Sharing Management in described share service device, described Sharing Management receives this access, and this is treated to the model of sharing means and this is treated to the memory address of the base register BAR of the PCIe of sharing means returns to described shared agency, described shared agency treats that by this model of sharing means feeds back to described virtual machine, and described virtual machine treats that according to this model of sharing means determines that corresponding this treat the driver of sharing means, described shared driving is set up first window in the designated memory space of described virtual machine, treats that according to this memory address of the base register of the PCIe of sharing means is mapped to this first window the base register internal memory of this PCIe that treats sharing means, and described Sharing Management sets up Second Window in the storage device of described share service device, this Second Window is mapped in the designated memory space in described virtual machine.

Compared to prior art, described device shared system and the method with quick peripheral component interconnection interface, can realize and between multiple devices, carry out sharing of network interface card, and without revising corresponding NIC driver.

Accompanying drawing explanation

Fig. 1 is the applied environment figure that the present invention has the preferred embodiments of the device shared system of PCIe interface.

Fig. 2 is the functional block diagram that the present invention has the preferred embodiments of the device shared system of PCIe interface.

Fig. 3 is the flow chart that the present invention has the preferred embodiments of the shared method of device of PCIe interface.

Main element symbol description

Request server	1
		Virtual machine	10
Storage device	100
		There is the device shared system of PCIe interface	11
Share and drive	110
		Share agency	112
Sharing Management	114
		Processor	12
PCIe interface	2
		Share service device	3
Network interface card	30
		Processor	32
Storage device	33

Following embodiment further illustrates the present invention in connection with above-mentioned accompanying drawing.

Embodiment

As shown in Figure 1, be the running environment figure that the present invention has the preferred embodiments of the device shared system of quick peripheral component interconnection (Peripheral Component Interconnect Express, PCIe) interface.The described device shared system 11 with PCIe interface is applied in multiple servers, to realize, the device with PCIe interface in a given server is shared.This preferred embodiments can be used for the device of all PCIe of having interfaces to share, that is, this device with PCIe interface is regarded as treating sharing means.

The technical program is below described for convenience of description and clearly, be elaborated as an example of the network interface card with PCIe interface example, in practical application, there is the device of PCIe interface all applicable to the technical program, for example: Serial Attached SCSI (SAS) (the Serial Attached SCSI with PCIe interface, SAS) card or there is PCIe card or the device such as host bus adaptor (Host Bus Adapter, HBA) card of PCIe interface.Described SCSI is small computer system interface (Small Computer System Interface).

Further, for convenience of description, server is divided into multiple roles, for example, in multiple request server 1(Fig. 1, only illustrates two) and share service device 3.Wherein, described share service device 3 uses to described request server 1 in order to the network interface card (hereinafter referred is " network interface card ") 30 that the multiple PCIe of having interfaces are provided.The quantity of the request server 1 shown in Fig. 1, share service device 3 and network interface card 30 is only for for example, also unrestricted in practical application, below describes with a network interface card 30 in share service device 3.

In described request server 1, also comprise that multiple virtual machines 10, processor 12 and other are not shown in software or the hardware unit in figure, for example, storage device, input-output equipment etc.Described virtual machine 10 also comprises that storage device 100 is separately to store the related data of each virtual machine 10, for example, and file, image etc.

Described processor 12 has the device shared system 11 of PCIe interface and all kinds of softwares of described request server 1 interior installation, such as operating system etc. described in carrying out.

Described request server 1 interconnects with described share service device 3, and described connection can be PCIe interface, and in other embodiments, described request server 1 can also be connected by other means with described share service device 3, for example, and PCI connection etc.

Described in described share service device 3 is also provided with, there is the device shared system 11 of PCIe interface, in addition, also comprise multiple network interface cards 30, processor 32 and storage device 33.Described network interface card 30 can be PCIe network interface card, or the network interface card of other types.Processor 32 of living in for example, in order to carry out the device shared system 11 with PCIe interface and other software, the operating system in described share service device 3.Described storage device 33 is for storing the various data of described share service device 3, for example, has the data that the device shared system 11 of PCIe interface is set and obtain described in utilization.

As shown in Figure 2, be the functional block diagram that the present invention has the preferred embodiments of the device shared system of PCIe interface.The device shared system 11 of the described PCIe of having interface comprises multiple functional modules, respectively: share driving 110, share agency 112 and Sharing Management 114.

As described above, described in being all provided with, there is the device shared system 11 of PCIe interface in described request server 1 and share service device 3, below be described with the cooperation having in described request server 1 and share service device 3 between the disparate modules of device shared system 11 of PCIe interface, for example, below with the shared driving 110 in the device shared system 11 with PCIe interface in described request server 1 and shared agency 112, coordinating of carrying out with the Sharing Management 114 of the device shared system 11 with PCIe interface in described share service device 3 is that example describes.

First, the virtual machine 10 in described request server 1, before realizing the network interface card 30 of sharing described share service device 3, need to first be determined the model of this network interface card 30, drives to call corresponding network interface card.In order to obtain the model of network interface card 30, described virtual machine 10 can send the configuration space of the PCIe of this network interface card of having stored these network interface card 30 models.

Described shared driving 110, the access of the configuration space of the PCIe of the network interface card 30 for the virtual machine 10 that obtains described request server 1 to described share service device 3, and this access is passed to described shared agency 112.Described access is in order to obtain the model of described network interface card 30.

Described shared driving 110 can utilize Netlink that this access is given to the shared agency in described request server, this Netlink socket is to realize a kind of special interprocess communication (the Inter-Process Communication that consumer process is communicated by letter with kernel process, IPC), be also the most frequently used interface of web application and kernel communication.

Described shared agency 112, for sending this access to the Sharing Management 114 of described share service device 3.For example, described shared agency 112 also can send this access to described Sharing Management 114 by IPC.

Described Sharing Management 114, for receiving this access, and goes the PCIe configuration space of described network interface card 30 to read the model of this network interface card 30 according to this access.

Described Sharing Management 114 is further used for the model of the network interface card 30 that this is read and the memory address of the base register of the PCIe of this network interface card 30 (Base Address Register, BAR) is returned to described shared agency 112.

Described shared agency 112, also for feeding back to the model of this network interface card 30 described virtual machine 10.Described virtual machine 10 can be determined according to the model of this network interface card 30 driver of corresponding this network interface card 30 from the operating system of described request server 1, so that this network interface card transmission related data of follow-up use.

Described shared driving 110 also for storage device 100 designated memory space at described virtual machine 10, and is set up a first window in this designated memory space.For example, described shared driving 110 can carry out designated memory space according to the memory address of setting.Described first window is the window corresponding with the window of the base register of the PCIe of network interface card 30.

Described shared driving 110, is further used in the base register internal memory of the PCIe that according to the memory address of the base register of the PCIe of this network interface card 30, this first window is mapped to this network interface card 30.Be while wanting the PCIe base register of network interface card 30 of access share service device 3 for the ease of the virtual machine 10 in described request server 1 by the mapping between this first window and described base register, only need the first window of the above-mentioned foundation of access.

For example, described shared agency 112 is detecting described virtual machine 10 while utilizing described first window to carry out access action, be the instruction of the internal memory of the base register of this network interface card 30 in share service device 3 described in access by the instruction transformation of the designated memory space of virtual machine 10 described in this access (being first window), and carry out the instruction after this conversion.

Described Sharing Management 114, also sets up a Second Window for the storage device 33 at described share service device 3, and this Second Window is mapped in the designated memory space in described virtual machine 10.

The I/O memory management unit (input/output memory management unitIOMMU) that described Sharing Management 114 is further used for arranging on described share service device 3 and request server 1 carries out address transition for direct internal memory operation (Direct Memory Access, the DMA) instruction of this network interface card 30.

Particularly, described Sharing Management 114 is detecting this network interface card 30 need carry out access to the storage device of the virtual machine 10 in described request server 1 100 time, and the DMA instruction transformation of utilizing the IOMMU on described share service device 3 that this network interface card 30 is sent becomes the DMA instruction of access Second Window.Then, described Sharing Management 114 is utilized the IOMMU instruction for the designated memory space of the virtual machine 10 in access described request server 1 by the DMA instruction transformation of described access Second Window on described request server 1, and carries out the instruction after conversion.

The foundation of described first, second window is that the nontransparent bridge joint (non-transparent bridging NTB) that utilizes PCIe carries out, and the mapping of described first, second window utilizes IOMMU to carry out.

In addition, in other embodiments, in the time that described share service device 3 comprises multiple network interface card 30, described Sharing Management 114 is also for distributing the network interface card 30 of described share service device 3 to described virtual machine 10, and the function of carrying out above-mentioned modules for the network interface card 30 of this distribution is to realize sharing of the network interface card 30 of described request server 1 to this distribution in described share service device 3.

As shown in Figure 3, be the flow chart that the present invention has the preferred embodiments of the shared method of device of quick peripheral component interconnection interface.First, step S2, the access of the configuration space of the PCIe of virtual machine 10 in described request server 1 network interface card 30 to described share service device 3 is obtained in described shared driving 110, and this access is passed to described shared agency 112.Described access is in order to obtain the model of described network interface card 30.

Step S4, described shared agency 112 sends this access to Sharing Management 114 in described share service device 3.

Step S6, described Sharing Management 114 receives this access, go the PCIe configuration space of described network interface card 30 to read the model of this network interface card 30 according to this access, and return to described shared agency 112 by the model of this network interface card 30 reading and by the memory address of the base register of the PCIe of this network interface card 30.

Step S8, the model of this network interface card 30 is fed back to described virtual machine 10 by described shared agency 112.Described virtual machine 10 can be determined according to the model of this network interface card 30 driver of corresponding this network interface card 30 from the operating system of described request server 1.

Step S10, described shared driving 110 is designated memory space in the storage device 100 of described virtual machine 10, and in this designated memory space, set up a first window, and according to the memory address of the base register of the PCIe of this network interface card 30, this first window is mapped in the base register internal memory of PCIe of this network interface card 30.

Step S12, described Sharing Management 114 is set up a Second Window in the storage device 33 of described share service device 3, and this Second Window is mapped in the designated memory space in described virtual machine 10, then, process ends.

Above execution mode is only unrestricted in order to technical scheme of the present invention to be described, although the present invention is had been described in detail with reference to above preferred embodiments, those of ordinary skill in the art should be appreciated that and can modify or be equal to the spirit and scope that replacement should not depart from technical solution of the present invention technical scheme of the present invention.

Claims

1. A device sharing system with a fast peripheral component interconnection standard PCIe interface operates in a plurality of servers, and is characterized in that at least one request server is connected with another sharing server, and the sharing server provides a device with a PCIe interface. and the device with the PCIe interface as the device to be shared, the system includes shared drivers, shared agents and shared management, wherein:

The shared driver is used to acquire the access of the virtual machine in the requesting server to the PCIe configuration space of the device to be shared in the shared server to obtain the model of the device to be shared, and pass the access to the the shared agent described above;

The sharing agent is used to transmit the access to the sharing management in the sharing server;

The sharing management is used to receive the access, and return the model of the device to be shared and the memory address of the base address register BAR of the PCIe of the device to be shared to the shared agent;

The sharing agent is also used to feed back the model of the device to be shared to the virtual machine, and the virtual machine determines the driver corresponding to the device to be shared according to the model of the device to be shared;

The shared driver is also used to set up a first window in the designated storage space of the virtual machine, and map the first window to the PCIe of the device to be shared according to the memory address of the base address register of the PCIe of the device to be shared. The base register memory of ; and

The sharing management is further used to create a second window in the storage device of the sharing server, and map the second window to a specified storage space in the virtual machine.

2. The device sharing system with fast peripheral component interconnection standard PCIe interface as claimed in claim 1, wherein said sharing agent is also used for detecting that said virtual machine utilizes said first window When executing the access action, convert the instruction for accessing the specified storage space of the virtual machine into an instruction for accessing the memory of the base address register of the device to be shared in the shared server, and execute the converted instruction.

3. as claimed in claim 1, there is the device sharing system of fast peripheral component interconnection standard PCIe interface, it is characterized in that, described sharing management is also used for setting the input/output memory on described sharing server and request server The management unit IOMMU performs address conversion for the DMA instruction of the device to be shared.

4. as claimed in claim 3, there is the device sharing system of fast peripheral component interconnection standard PCIe interface, it is characterized in that, described sharing management is also used for:

When detecting that the device to be shared needs to access the storage device of the virtual machine of the requesting server, use the IOMMU on the shared server to convert the DMA command issued by the device to be shared into accessing the second window DMA instructions; and

Using the IOMMU on the requesting server to convert the DMA instruction for accessing the second window into an instruction for accessing the specified storage space of the virtual machine in the requesting server, and execute the converted instruction.

5. as claimed in claim 1, there is the device sharing system of fast peripheral component interconnection standard PCIe interface, it is characterized in that, the establishment of described first and second windows utilizes the non-transparent bridging NTB of PCIe to carry out, and The mapping of the first and second windows is performed by using the IOMMU.

6. A device sharing method with a PCIe interface that utilizes a device sharing system with a fast peripheral component interconnection standard PCIe interface as claimed in claim 1, runs in a plurality of servers, and is characterized in that at least one request Server is connected with another shared server, and described shared server provides the device with PCIe interface and described device with PCIe interface as the device to be shared, and this method comprises the steps:

The shared driver obtains the access of the virtual machine in the request server to the configuration space of the PCIe of the device to be shared in the shared server to obtain the model of the device to be shared, and passes the access to the shared agent;

the sharing agent transmits the access to the sharing management in said sharing server;

The sharing management receives the access, returns the model of the device to be shared and the memory address of the base address register BAR of the PCIe of the device to be shared to the sharing agent;

The sharing agent feeds back the model of the device to be shared to the virtual machine, and the virtual machine determines the driver corresponding to the device to be shared according to the model of the device to be shared;

The shared driver establishes a first window in the designated storage space of the virtual machine, and maps the first window to the base address register of the PCIe of the device to be shared according to the memory address of the base address register of the PCIe of the device to be shared memory; and

The sharing management creates a second window in the storage device of the sharing server, and maps the second window to a specified storage space in the virtual machine.

7. as claimed in claim 6, there is the device sharing method of fast peripheral component interconnection standard PCIe interface, it is characterized in that, this method also comprises:

When the sharing agent detects that the virtual machine uses the first window to perform an access action, it converts the instruction for accessing the specified storage space of the virtual machine into an instruction for accessing the device to be shared in the sharing server. The base register of the memory instruction, and executes the translated instruction.

8. the device sharing method with fast peripheral component interconnection standard PCIe interface as claimed in claim 6, is characterized in that, this method also comprises:

The IOMMU on the sharing server and the requesting server is set to perform address conversion for the DMA command of the device to be shared.

9. the device sharing method with fast peripheral component interconnection standard PCIe interface as claimed in claim 8, is characterized in that, this method also comprises:

When detecting that the device to be shared needs to access the storage device of the virtual machine of the requesting server, use the IOMMU on the shared server to convert the direct memory operation DMA command issued by the device to be shared into an access DMA instructions for the second window; and

10. the device sharing method with fast peripheral component interconnection standard PCIe interface as claimed in claim 6, is characterized in that, when described shared server comprises a plurality of devices to be shared, this method also comprises:

The sharing management assigns the device to be shared in the sharing server to the virtual machine.