WO2017026931A1

WO2017026931A1 - Implementing input/output in a virtualized environment

Info

Publication number: WO2017026931A1
Application number: PCT/SE2015/050867
Authority: WO
Inventors: Christopher Wayne ROSSMITH; John Christensen
Original assignee: Telefonaktiebolaget Lm Ericsson (Publ)
Priority date: 2015-08-11
Filing date: 2015-08-11
Publication date: 2017-02-16

Abstract

The present disclosure relates to an apparatus for and method of implementing input/output in a host in a virtualized environment, the host comprising a kernel space and a user space, the user space comprising a virtual machine, a user space virtual machine manager, a paravirtualized driver and a configuration agent, and the kernel space comprising a virtual function input/output, VFIO, an NIC driver and a kernel space virtual machine manager. The method comprises dividing (21) instructions to be sent to the input/output device into configuration control instructions (22) and data (23), transmitting (24) the configuration control instructions to a configuration agent. At the configuration agent, (25) the configuration control instructions are received (25) translated (26) to a form usable at a Virtual Function input/output, VFIO, and/or Network Interface Card, NIC, driver and transmitting (27) to the configuration control. In the kernel space the configuration instructions are used (28) to configure the input/output device. The method further comprises transferring (29) the data to a direct memory access module and using (30) the direct memory access module to transfer the data directly to the input/output device.

Description

I M PLEMENTING INPUT/OUTPUT IN A VIRTUALIZED ENVIRONM ENT

Technical field

The present invention relates to implementing input/output in a virtualized environment. Background Virtualized and para-virtualized network devices (emulated devices) are used in virtualized environments, to decouple the guest operating system from the physical hardware. This allows multiple emulated devices to share a single physical network device. It also allows live migration of virtual machines from one physical machine to another, without restarting the virtual machine. A disadvantage of virtualized and para-virtualized devices is a significant decrease in network input/output (I/O) performance, in terms of lower throughput, higher latency and higher central processing unit (CPU) usage.

In order to overcome the problem of the reduction in performance caused by emulation, many newer processor architectures have support for Peripheral Component Interconnect (PCI) device pass-through. This capability allows a physical PCI device from the host machine to be assigned directly to a guest virtual machine. The guest operating system drivers can use the device hardware without relying on any driver capabilities from the host operating system, once the device has been setup.

The Single Root Input /Output Virtualization (SR-IOV) specification is a standard for PCI device pass-through, that enables multiple Virtual Machines to share a single physical device. SR-IOV gives a network performance that is close to that of a native host. However there are two problems with this arrangement, namely that the guest operating system must have vendor specific drivers for the network device and that live migration is not possible.

Emulated devices support live migration, but do not have the network input/output performance requested by some network centric applications. PCI device pass-through gives requested performance but does not allow virtual machines to be live migrated.

Summary

Accordingly, in a first aspect of the invention there is provided an apparatus for implementing input/output in a host in a virtualized environment, the host comprising a processor, a memory, an input or output, I/O, device, an Input/output memory management unit, IOMMU , and a Network Interface Card, NIC. The apparatus is configured such that the memory comprises a kernel space and a user space. The user space comprises code for implementing a virtual machine, a user space virtual machine manager and a para- virtualized driver. The kernel space comprises code for implementing a virtual function input/output, VFIO, an NIC driver; and a kernel space virtual machine manager. The user space further comprises a configuration agent for communicating configuration instructions between the para-virtualized driver, and the VFIO and/or NIC driver. The para-virtual ized driver comprises a Direct Memory Access module configured to enable communication of data directly between the virtual machine, and the NIC by means of the IOMMU without passing through the Kernel space.

Preferably the para-virtualized driver is a VirtIO driver.

Preferably the user space virtual machine manager is a Qemu hypervisor.

Preferably the kernel space virtual machine manager is a Kernel Virtual Machine, KVM.

Preferably the configuration agent is a Vhost agent. Preferably data related to configuration instructions is stored in the virtual machine.

Preferably the configuration agent is stateless.

In a second aspect of the present invention there is provided a method of implementing an input/output in a host in a virtualized environment, the host comprising a kernel space and a user space. The user space comprises a virtual machine, a user space virtual machine manager, a para-virtualized driver and a configuration agent. The kernel space comprises a virtual function input/output, VFIO, an NIC driver and a kernel space virtual machine manager. The method comprises dividing instructions to be sent to the input/output device into configuration control instructions and data, and transferring the configuration control instructions to a configuration agent. The method further comprises, at the configuration agent translating the configuration control instructions to a form usable at a Virtual Function input/output, VFIO, and/or Network Interface Card, NIC, driver, and transferring the configuration control instructions to the VFIO and/or NIC driver. The method yet further comprises, in the kernel space, using the configuration instructions to configure the input/output device and transferring the data to a direct memory access module, and using the direct memory access module to transfer the data directly to the input/output device. In a third aspect of the present invention there is provided a computer program, comprising instructions which, when executed on at least one processor, cause the at least one processor to carry out a method according to the second aspect of the invention. In a fourth aspect of the present invention, there is provided a computer program product comprising a computer program according to the third aspect of the present invention.

In a fifth aspect of the present invention, there is provided a carrier containing a computer program product according to the fourth aspect of the present invention, wherein the carrier optionally includes an electrical signal, an optical signal, a radio signal, a magnetic tape or disk, an optical disk or a memory stick.

Brief Description Of The Drawings

Figure 1 is a schematic diagram of a computer architecture according to the prior art;

Figure 2 is a schematic diagram of a computer architecture employing Direct Memory Access according to the prior art;

Figure 3 is a schematic diagram of a para-virtualized architecture according to an embodiment;

Figure 4 is a flow chart of a method of implementing input/output in a virtualized environment according to an embodiment; Figure 5 is a flow chart of a method of handling interrupts according to an embodiment; and

Figure 6 is a schematic diagram showing live migration of a virtual machine.

Detailed description

Embodiments will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments are shown. However, other embodiments in many different forms are possible within the scope of the present disclosure. Rather, the following embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. The present invention seeks to overcome the problems outlined above by providing a mechanism in which the instructions to be sent from a virtual machine to an input/output device are divided into configuration (control) instructions and data. The configuration instructions are translated into a format usable by the input/output device by means of a configuration agent resident in the user space. The data is transferred by means of Direct Memory Access. Direct memory access is a feature of computer systems which allows certain hardware systems to access main system memory independently of the central processing unit. Typically, if DMA is not used, the central processing unit is typically fully occupied during a read/write operation to an input/output device. Figure 1 is a schematic diagram of a typical architecture in which input/output is controlled by the kernel. In the user space (1), applications interact with the kernel (2), which interacts with the central processing unit (CPU) (3), the memory (4) and input/output devices (5). Figure 2 is a schematic of an architecture which uses DMA. With DMA the CPU offloads routine input/output instructions to a DMA module (6), controlled (7) by the CPU. The CPU provides the DMA with the addresses of the bytes to be transferred, the number of bytes and the destination addresses via an address bus (8). The CPU signals the DMA and is then able to carry on with other tasks whilst the DMA takes care of the input/output instructions. Bytes in the memory are obtained and transferred by allowing the DMA temporary control of the data bus (9).

In an embodiment, DMA is incorporated into para-virtual ized drivers located in the virtual machine. This enables the configuration agent to be used solely for configuration (control) instructions and enables the kernel located backend of the configuration agent to be taken out of the data path.

In an embodiment the configuration agent is stateless. Each instruction arriving at the configuration agent is translated with no reference to any previous instruction or other information on the system. All the data related to configuration and control is stored in the virtual machine.

The arrangement allows instructions to be passed to the input/output device without the use of the Kernel space. The delays associated with scheduling of the resources of the kernel are therefore avoided, with a subsequent improvement in performance of the virtual machine. The maintenance of the configuration instructions in the virtual machine and the data in a direct memory access module enables a snapshot of the virtual machine to be taken at any time. This enables live migration of the virtual machine, which is not possible with pass through.

Figure 3 is a schematic diagram of the arrangement of a para-virtualized architecture, according to an embodiment. The host comprises a user space (1 ), a kernel space (2) and at least one input/output device and its associated controlling modules (10). A virtual machine (11) is located in the user space. In an embodiment, the virtual machine operates in a virtualized environment. It comprises a para- virtualized driver (12) and a user space virtual machine manager (13). The para-virtualized drivers comprise direct memory access modules (14), which enable data to be transferred directly to a network interface card (19) and hence to an input/output device without passing through the kernel space (2). The user space further comprises a configuration agent (15), through which configuration (control) instructions are passed, translated and then transferred to the Virtual Function Input/Output (18) and the Network Interface Card driver (17), both located in the kernel space (2). The configuration instructions are then passed to Input/output memory management unit (IOMMU) (20) and the Network Interface Card, NIC (19), and hence to the input/output device (5). The kernel space comprises a Virtual Function input/output (VFIO) (18), a Network Interface Card (NIC) driver (17) and a kernel space virtual machine manager (16).

Figure 4 is a flowchart illustrating the steps for providing instructions from the virtual machine to an input/output device. The first step is to divide (21) instructions into configuration instructions (control) (22) and data (23). The configuration instructions are then transmitted (24)) to a configuration agent. On receipt (25) by the configuration agent, the instructions are translated (26) for use by a virtual function input/output (VFIO) driver or a Network Interface Card (NIC) driver in Kernel space. The translated instructions are then transmitted (27) to the VFIO or NIC and hence transmitted to the Input/output memory management unit (IOMMU) and the Network Interface Card, NIC, where they are used for device configuration (28). Data (23) is transferred (29) to a Direct memory Access, DMA, module in the user space. The DMA is then used to transfer (30) the data directly to the input/output device.

Figure 5 is a flowchart illustrating the steps of sending an interrupt from an input/output device to the virtual machine according to an embodiment. The first step comprises the creation of the interrupt at the input/output device (31). The interrupt is then transmitted (32) to the kernel space virtual machine manager, transferred (33) to the user space virtual machine manager and finally (34) to the paravirtualised driver, where it is executed.

Figure 6 is a diagram illustrating the live migration of a virtual machine. The process begins with stopping (35) of the virtual machine on the original host, followed by the saving (36) of the virtual machine state, including the device states. The virtual machine state is then transferred to the new host (37). The virtual machine is then started (38) on the new host. An input/output device is then started with the new state (39).

In an embodiment the paravirtualized driver is a virtIO driver. Virtio is an abstraction for a set of common emulated devices in a paravirtuaiized hypervisor. This design allows the hypervisor to export a common set of emulated devices and make them available through a common application programming interface (API).

In an embodiment, the user space virtual machine manager is a Quick Emulator, Qemu, hypervisor. Qemu is an open source hosted hypervisor which performs hardware virtualisation. It provides a number of device models which enables it to run a variety of unmodified operating systems. In an embodiment, the kernel space virtual machine manager is a Kernel Virtual Manager, KVM. KVM is a virtualization infrastructure for the Linux kernel that turns it into a hypervisor. KVM and Qemu can be used together in order to run virtual machines at near-native speed. In a further embodiment therefore the user space virtual machine manager comprises a Qemu hypervisor and the kernel space virtual machine manager comprises a KVM hypervisor. In an embodiment, the configuration agent is a Vhost user agent.

In a further embodiment, the paravirtualised driver is a virtIO driver, the user space virtual machine manager is a Qemu hypervisor, the kernel space virtual machine manager is a KVM and the configuration agent is a Vhost user agent. In this embodiment, Qemu interfaces the Vhost backend through a Vhost character device. The Vhost backend runs as a user space process "Vhost agent" in the host. It receives configurations from Qemu and passes these configurations to the Linux VFIO driver (IOMMU and I RQs configuration) and the NIC driver (Virt-queue descriptors and driver configuration).

The Vhost Agent only handles configuration. Data are transferred, directly between hardware and guests, using DMA, with the IOMM U doing address translations, removing the Vhost Agent from the data plane. Hardware interrupts are sent to the guests through KVM irqfds setup with the VFIO drive.

The Vhost backend retains the configuration parts to keep live migration support. The VirtIO virt-queues are unmodified, for the same reason, but are now shared with the network hardware to enable DMA in the data path, between guests and the network hardware. This effectively removes the Vhost backend from the data path.

The person skilled in the art will appreciate that each of these software modules may be replaced with an equivalent which performs the same function. The invention is not limited to any specific software or hardware.

The foregoing embodiments of the present invention have been presented for the purposes of illustration and description. These descriptions and embodiments are not intended to limit the invention to the precise form disclosed, and many modifications and variations are possible in light of the above disclosure. The embodiments were chosen and described in order to best explain the principle of the invention and its practical applications to thereby enable others skilled in the art to best utilize the invention in its various embodiments and with various modifications as are suited to the particular use contemplated.

Claims

1. An apparatus for implementing input/output in a host in a virtualized environment, the host comprising a processor (3), a memory (4), an input or output, I/O, device (5), an Input/output memory management unit, IOMMU (20), and a Network Interface Card, NIC (19), the apparatus being configured such that:

the memory comprises a kernel space (2) and a user space (1);

wherein the user space comprises code for implementing:

a virtual machine (11);

a user space virtual machine manager (13);

a paravirtualized driver (12); and

wherein the kernel space comprises code for implementing:

a virtual function input/output, VFIO (18);

an NIC driver (17); and

a kernel space virtual machine manager (16);

wherein the user space further comprises a configuration agent (15) for communicating configuration instructions between the paravirtualized driver (12), and the VFIO (18) and/or NIC driver (17); and the paravirtualized driver (12) comprises a Direct Memory Access module (14) configured to enable communication of data directly between the virtual machine (11), and the NIC (19) and/or the IOMMU (20) without passing through the Kernel space.

2. An apparatus as claimed in claim 1 , wherein the paravirtualized driver is a VirtIO driver.

3. An apparatus as claimed in claim 1 or claim 2, wherein the user space virtual machine manager is a Qemu hypervisor.

4. An apparatus as claimed in any preceding claim, wherein the kernel space virtual machine manager is a Kernel Virtual Machine, KVM.

5. An apparatus as claimed in any preceding claim, wherein the configuration agent is a Vhost agent.

6. An apparatus as claimed in any preceding claim, wherein data related to configuration instructions is stored in the virtual machine

7. An apparatus as claimed in claim 6, wherein the configuration agent is stateless.

8. A method of implementing input/output in a host in a virtualized environment, the host comprising a kernel space and a user space, the user space comprising a virtual machine, a user space virtual machine manager, a paravirtualized driver and a configuration agent, and the kernel space comprising a virtual function input/output, VFIO, an NIC driver and a kernel space virtual machine manager, the method comprising:

dividing (21 ) instructions to be sent to the input/output device into configuration control instructions (22) and data (23);

transmitting (24) the configuration control instructions to a configuration agent;

at the configuration agent:

receiving (25) the configuration control instructions;

translating (26) the configuration control instructions to a form usable at a Virtual Function input/output, VFIO, and/or Network Interface Card, NIC, driver;

transmitting (27) the configuration control instructions to the VFIO and/or NIC driver;

in the kernel space:

using (28) the configuration instructions to configure the input/output device;

and

transferring (29) the data to a direct memory access module; and using (30) the direct memory access module to transfer the data directly to the input/output device.

9. A method as claimed in claim 8, wherein the paravirtualized driver is a VirtIO driver.

10. A method as claimed in claim 8 or claim 9, wherein the user space virtual machine manager is a Qemu hypervisor.

1 1 . A method as claimed in any of claims 8 to 10, wherein the kernel space virtual machine manager is a Kernel Virtual Machine, KVM.

12. A method as claimed in any of claims 8 to 1 1 , wherein the configuration agent is a Vhost agent.

13. A method as claimed in any of claims 8 to 12, wherein data related to configuration instructions is stored in the virtual machine

14. A method as claimed in claim 13, wherein the configuration agent is stateless.

15. A computer program, comprising instructions which, when executed on at least one processor, cause the at least one processor to carry out a method according to any one of claims 8 to 14.

16. A computer program product comprising a computer program as claimed in claim 15.

17. A carrier containing the computer program product of claim 16, wherein the carrier optionally includes an electrical signal, an optical signal, a radio signal, a magnetic tape or disk, an optical disk or a memory stick.