CN117453352A

CN117453352A - Equipment straight-through method under Xen

Info

Publication number: CN117453352A
Application number: CN202311769161.2A
Authority: CN
Inventors: 张云飞; 韩乃平; 黄磊; 崔建英
Original assignee: Kirin Software Co Ltd
Current assignee: Kirin Software Co Ltd
Priority date: 2023-12-21
Filing date: 2023-12-21
Publication date: 2024-01-26
Anticipated expiration: 2043-12-21
Also published as: CN117453352B

Abstract

The application relates to a device pass-through method under Xen, which comprises the steps of preparing a device tree of a primary edition of a client, preparing a configuration file of the client, distributing static continuous memory with the size of 2-order pages for the client and returning mfn, creating Stage2 mapping, obtaining page frame number gfn of the client, initializing construction parameters of the client by a user layer based on gfn, constructing a device tree of a final edition of the client based on the device tree of the primary edition, the configuration file, gfn and the construction parameters, initializing a memory layout of an operating system of the client, wherein the memory of the operating system comprises a kernel, the device tree of the final edition of the client, a kernel module and a memory file system, mapping gfn to the user layer to obtain gvaddr, and copying the kernel, the device tree, the kernel module and the memory file system of the client into physical memories corresponding to gfn based on gvaddr and the memory layout.

Description

Equipment straight-through method under Xen

Technical Field

The application relates to the field of equipment through, in particular to an equipment through method under Xen.

Background

In the field of virtualization, device pass-through (Passthrough) is an important virtualization technology that allows a guest (VM) to directly access a hardware device on a physical host, rather than through a virtualization layer (Hypervisor). This may provide client performance that is closer to physical machine performance and, in some scenarios, increases flexibility of the virtualized environment. An administrator may assign physical devices to particular clients through a virtualization management tool or a command line tool. This hands over the device from the control of the host operating system to the client. At the end of the client lifecycle or under the administrator's operation, the device may be released back to the host operating system control. This typically involves a reset and reassignment of the device. One of the main advantages of device pass-through is that it can provide client performance that is close to local performance, as clients can directly access hardware resources without the need for intermediation through the virtualization layer. Device pass-through is very useful for workloads requiring specific hardware resources and application scenarios with high performance requirements. In systems that support device pass-through, it is often necessary to enable IOMMU. The IOMMU is a hardware device whose main function is to manage and map Direct Memory Access (DMA) of the device. DMA allows external devices (e.g., network adapters, graphics cards, etc.) to directly access system memory to increase data transfer speeds. The IOMMU is operative to create a mapping between system memory and external devices to provide better memory management and security, which provides a virtualized address space for clients, isolating clients and other clients from access to the devices.

Current device pass-through functionality requires that both host hardware and the virtualization platform support this functionality. The primary virtualization platform, such as Xen, KVM, VMware, generally provides corresponding technical support. In addition to software support, this function requires hardware support. As described above, software is basically based on IOMMU hardware to implement device pass-through functionality. The IOMMU, however, is not supported on all platforms as a high level hardware. Particularly with embedded platforms where hardware resources are scarce, it is quite common that the hardware resources in embedded systems are limited, which may include limited processing power, memory capacity, storage space, and other resources. Such resource tension may be caused for a variety of reasons, including power consumption requirements, cost constraints, size limitations, and application scenarios. Based on this situation, the virtualized device-through technology is made difficult to apply to the floor in an embedded scenario. The embedded scenario is a field with extremely high requirements on performance and real-time performance, so that a device pass-through virtualization technology without hardware IOMMU support is urgently needed in the scenario.

Disclosure of Invention

In order to solve the problem that the embedded device cannot use the device through technology in the virtualization due to the lack of IOMMU hardware, the invention provides a device through method under Xen, which adopts the following technical scheme:

a device pass-through method under Xen, comprising:

step S101, starting a Xen device direct connection function and preparing a device tree of a client;

step S102, starting to construct a client through a user layer tool, and preparing a configuration file of the client;

step S103, the user layer allocates static continuous memory with the size of 2 order pages for the client by driving the hypercall calling partner system and returns a starting physical address page frame number mfn;

step S104, creating Stage2 mapping, and obtaining page frame number gfn of the client based on the initial physical address page frame number mfn obtained in step S103, so as to make gfn equal to mfn;

step S105, the user layer initializes the construction parameters of the client based on the page frame number gfn obtained in step S104;

step S106, constructing a final version of the device tree of the client based on the device tree of the primary version prepared in step S101, the configuration file obtained in step S102 and the construction parameters obtained in step S105;

step S107, initializing the memory layout of the client operating system based on the construction parameters obtained in the step S105, wherein the operating system memory comprises a kernel, a device tree of the final version of the client obtained in the step S106, a kernel module and a memory file system, and the kernel module is a program which is dynamically loaded by the operating system and is not directly compiled to the kernel;

step S108, mapping the gfn to a user layer by using a mmap function based on the construction parameters obtained in the step S105 to obtain gvaddr, wherein gvaddr is the address of the user layer;

in step S109, the kernel, the device tree, the kernel module, and the memory file system of the client are copied to the physical memory corresponding to gfn based on the gvaddr and the memory layout.

In another possible implementation manner, the step S101 starts a device pass-through function of Xen, and prepares a device tree of the client, including:

reserving equipment tree nodes of the direct equipment needing to realize the direct function;

the device tree nodes of the pass-through device are added to the device tree of the client.

In another possible implementation manner, the step S102 starts to build the client through the user layer tool, and prepares a configuration file of the client, including:

creating a client through an xl create guard.cfg command, the guard.cfg being a configuration file of the client;

setting device register address space mapping through a user layer parameter iomem= [ "start, num [ @ gfn ]", ], wherein a start parameter represents a starting physical address of a peripheral device, a num parameter represents the number of pages to be mapped, gfn represents a starting memory address to be mapped to a client, and a page frame number of the starting physical address corresponding to the start is equal to gfn;

the configuration file is modified, and the interrupt of the pass-through device is passed through to the client.

In another possible implementation manner, the step S104 creates a Stage2 map, and obtains the page frame number gfn of the client based on the starting physical address page frame number mfn obtained in step S103, and makes gfn equal to mfn, including:

finding page table p2m_domain of Stage2 in domain data structure of client;

and (3) circularly inserting gfn equal to mfn page tables with the number of 2 order into the p2m_domain to establish Stage2 mapping and obtain page frame numbers gfn of clients.

In another possible implementation, the user layer initializes build parameters of the client based on the gfn, including the client's RAM space size, RAM start address, GIC version, CPU type, and associated hardware resources.

In summary, the present application has the following beneficial effects:

enabling Xen to support device pass-through functionality on chips without IOMMU hardware. Because of the limitations of embedded hardware, we cannot modify the chip hardware. However, by this means, the Xen virtualized device pass-through function can be applied to most embedded devices without adding additional overhead to the system. The performance and the real-time performance of the client are greatly enhanced.

Drawings

Fig. 1 is a schematic flow chart of a device through method under Xen in an embodiment of the present application.

FIG. 2 is a schematic diagram of a device pass-through with IOMMU participation under virtualization in an embodiment of the present application.

Fig. 3 is a schematic structural diagram of device pass-through without IOMMU participation under virtualization in an embodiment of the present application.

Detailed Description

The present application is described in further detail below in conjunction with figures 1-3.

Modifications of the embodiments which do not creatively contribute to the invention may be made by those skilled in the art after reading the present specification, but are protected by patent laws only within the scope of claims of the present application.

For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

In addition, the term "and/or" herein is merely an association relationship describing an association object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone. In this context, unless otherwise specified, the term "/" generally indicates that the associated object is an "or" relationship.

Embodiments of the present application are described in further detail below with reference to the drawings attached hereto.

As shown in fig. 1, an embodiment of the present application provides a device pass-through method under Xen, where the method is implemented by using an APM64 hardware architecture with a Xen virtualization platform, and specifically includes:

step S101, a Xen device pass-through function is started, and a device tree of the client is prepared.

For the embodiment of the present application, step S101 specifically includes:

in the embodiment of the application, a user informs Xen to start the pass-through function of a certain device through the device tree, modifies the device tree of Xen, and adds an 'Xen' keyword into the device tree node of the device with the pass-through function. Thus Xen will retain control of the device at start-up rather than giving control of this device to dom0.

The device tree nodes of the pass-through device are added to the device tree of the client, and in the embodiment of the application, the reserved device tree nodes are added to the device tree of the client, so that the client definitely controls the pass-through device to be owned by the client.

Step S102, the client is built through the user layer tool, and a configuration file of the client is prepared.

For the embodiment of the present application, step S102 specifically includes:

creating a client by an xl create guard.cfg command, the guard.cfg being the client's profile, in this embodiment the user layer tool starts building a client by creating a client using the "xl create guard.cfg" command on the dom0 Linux system, the "guard.cfg" being the client's profile. There is a need to prepare the client with the configuration files required for the pass-through function.

Setting device register address space mapping by user layer parameter iomem= [ "start, num [ @ gfn ]", ], wherein the start parameter represents the starting physical address of the peripheral, the num parameter represents the number of pages to be mapped, gfn represents the starting memory address to be mapped to the client, wherein the page frame number of the starting physical address corresponding to start is equal to gfn, e.g. the address of the starting memory address (start) is 0x28d00000, gfn must be 0x28d00, and then Xen will perform Stage2 mapping of the device address according to the input parameter, eventually making IPA of the device address equal to PA. The DMA is ensured to correctly find the address space of the device.

Modifying the configuration file, and directly passing the interrupt of the direct equipment to the client, specifically, setting irqs= [ irq1, irq2, irq3 … … irqn ] parameters in the configuration file of the client, wherein the numbers in brackets are interrupt numbers which need to be passed through to the client, and directly informing the client after the interrupt of the equipment which needs to be passed through occurs by passing the interrupt of the direct equipment to the client, so as to complete corresponding interrupt processing, such as starting data transfer of DMA or collecting data of DMA transfer to an upper layer.

In step S103, the user layer allocates static continuous memory of 2 order pages for the client by driving the hypercall call partner system and returns the initial physical address page frame number mfn.

For the embodiment of the present application, 2 order can be understood as the power N of 2, xen is responsible for allocating and managing physical memory for clients, each DomU (client) has its independent address space, and a continuous memory allocator is used to allocate static continuous memory of 2 order page size for clients. Static memory is to ensure that clients are actually allocated before startup, rather than being allocated again by copy-on-write means after startup. The continuous memory provides a DMA buffer for DMA to read and write directly. Therefore, before creating a client, static continuous memory needs to be allocated to the client through Xen's partner system first, specifically, the user layer specifies the parameter "mem=size" in the client's configuration file, where "size" is the memory that the client runs at a specified size. This instructs Xen to allocate a piece of static memory directly from its heap space, rather than dynamically expanding the running memory for the client by means of copy-on-write after the client is started. The Xen partner system is different from Linux, and can allocate managed memory to 2 order pages, wherein the order value is 20 at maximum, namely, 4G memory space is allocated for the client at maximum. On this static continuous memory we can allocate DMA memory buffers directly.

Step S104, create Stage2 map and get the client page frame number gfn based on the starting physical address page frame number mfn obtained in step S103, making gfn equal to mfn.

For the embodiment of the application, creating Stage2 map establishes IPA to PA memory map for client memory space, and in Xen's traditional scheme (IOMMU support) as shown in fig. 2, the MMU address map of the CPU is divided into two stages Stage1 and Stage2.Stage1 converts the virtual address VA of the CPU into an intermediate physical address IPA based on the operating system page table, and Stage2 converts IPA into a real physical address PA based on the Xen page table. Xen itself is a set of fully virtualized platforms that the operating system considers itself to be running in a real piece of hardware that is not aware of the presence of IPA. The PA it considers to the operating system is actually the IPA in the virtualized environment. The CPU starts DMA through DMA engine (DMA controller driver) to complete one data transfer. Taking the data transmission direction as an example from the memory to the device (such as a network card, a display card and the like), the CPU tells the DMA about the physical memory address PA (PA is actually virtualized IPA herein above), the device physical address, the data size and the like, and the DMA completes one data transmission according to the configurations. When the DMA accesses the memory, the same page table is already configured by Xen in the IOMMU, i.e. the IPA- > PA of the IOMMU is the same as the IPA- > PA of MMU Stage2, so the DMA can access the correct physical address.

The problem with Xen is that its memory is dynamically allocated and the address space in which the client runs is generally fixed, so normally IPA is not equal to PA. The memory access of the DMA must convert IPA to PA to find the correct data, i.e., IPA is not equal to PA in the prior art.

In the embodiment of the present application, the step 2 map is created, and the page frame number gfn of the client is obtained, which specifically includes:

the page table p2m_domain of Stage2 is found in the domain data structure of the client, and according to the obtained initial physical address page frame number mfn of the static continuous memory, the size is 2 order pages, gfn is modified to be mfn, page table items of mfn to gfn are circularly built, and finally the virtualized IPA is equal to PA, so that page tables of which the number is 2 order are circularly inserted gfn to be equal to mfn in the p2m_domain, stage2 mapping is built, the effect of page frame number gfn of the client is obtained, and as shown in FIG. 3, DMA addresses IPA configured by DMA engines are directly used for addressing in the memory under the condition of no IOMMU. Since the IPA is equal to PA and the page table entry of 2 order pages is inserted into the two-level memory mapping page table entry, that is, the two-level mapping of IPA equal to PA is established and the size is the size of register space, when DMA is used, the operating system uses the DMA memory buffer and the through device register address as the source destination address to start DMA transmission, the source destination address of the operating system for DMA is the physical address PA considered by the operating system, but is actually the intermediate physical address IPA under the virtualization condition, and since the IPA is equal to PA, the final PA address of DMA under the virtualization condition is finally provided, and the DMA can find the correct data in the memory by using PA.

In step S105, the user layer initializes the client build parameters based on the page frame number gfn obtained in step S104.

For the embodiment of the present application, compared to the conventional client, the memory address of the client in the embodiment of the present application starts from 0x40000000, and the running address space is dynamically random, so that in the implementation of the present application, the client needs to be built based on the dynamic address, and since the above steps S101 to S104 create the condition for implementing the pass-through function for the client capable of implementing the device pass-through function, and the building parameters include the RAM space, GIC version, CPU type and related hardware resources of the client, the building parameters of the client can be initialized based on gfn, so that the hardware environment can be virtualized for the client, so that the client capable of implementing the device pass-through can be conveniently built based on the hardware environment, and how to build the client capable of implementing the device pass-through based on the hardware environment is as follows:

step S106, constructing a final device tree of the client based on the device tree of the primary version prepared in step S101, the configuration file obtained in step S102, and the construction parameters obtained in step S105.

For the embodiment of the application, the backup tree node is built for the client organization according to the hardware environment described by the building parameters, wherein the starting address and the size of the memory node are built according to gfn and the designated size.

Step S107, initializing the memory layout of the client operating system based on the construction parameters obtained in step S105.

The operating system memory includes a kernel, a device tree of the final version of the client obtained in step S106, a kernel module, and a memory file system, where the kernel module is a program that is dynamically loaded by the operating system, rather than directly compiled to the kernel. The construction parameters provided for step S107 by step S105 enable step S107 to initialize the memory layout based on the construction parameters.

Step S108, mapping gfn to the user layer by using mmap function based on the construction parameters obtained in step S105, and obtaining gvaddr which is the address of the user layer.

In step S109, the kernel, device tree, kernel module, and memory file system of the client are copied to the physical memory corresponding to gfn based on gvaddr and memory layout.

For the embodiment of the present application, the construction of the device tree is completed in step S106, and the construction of the kernel, the kernel module and the memory file system is the same as the conventional construction method, which is not repeated here.

The user layer initializes the memory layout of the kernel, the device tree, the kernel module, the memory file system and tells Xen. The user layer uses mmap function to map gfn to address space of own process, and the user layer address is gvaddr. The user can copy the kernel, the device tree, the kernel module and the memory file system into the physical memory corresponding to gfn according to the initialized memory layout through the physical memory distributed by the gvaddr operation partner system, so that the effect of constructing the client with the function of directly connecting the device is achieved, and the client can realize the device direct connection under the condition of no IOMMU device.

The foregoing is only a partial embodiment of the present application and it should be noted that, for a person skilled in the art, several improvements and modifications can be made without departing from the principle of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A method for device pass-through under Xen, comprising:

step S101, starting a Xen device direct connection function, and preparing a device tree of a primary version of a client;

2. The Xen device pass-through method according to claim 1, wherein the step S101, starting the Xen device pass-through function, prepares a device tree of the client, comprises:

3. The method according to claim 1, wherein the step S102, starting to build the client through the user layer tool, prepares the configuration file of the client, includes:

4. The device pass-through method under Xen according to claim 1, wherein the creating step S104 creates a Stage2 map, and obtains a page frame number gfn of the client based on the start physical address page frame number mfn obtained in step S103, and makes gfn equal to mfn, includes:

finding page table p2m_domain of Stage2 in domain data structure of client;

5. The Xen-based device pass-through method of claim 1 wherein the user layer initializes the client build parameters including the client RAM space size, RAM start address, GIC version, CPU type and associated hardware resources based on the gfn.