CN116841731A

CN116841731A - FPGA (field programmable Gate array) virtualized resource scheduling system and method

Info

Publication number: CN116841731A
Application number: CN202310616927.7A
Authority: CN
Inventors: 李玉成; 李志刚; 路雪松
Original assignee: CETC 52 Research Institute
Current assignee: CETC 52 Research Institute
Priority date: 2023-05-29
Filing date: 2023-05-29
Publication date: 2023-10-03

Abstract

The invention belongs to the technical field of computers, and particularly relates to an FPGA (field programmable gate array) virtualized resource scheduling system and method. An FPGA virtualized resource scheduling system comprises a resource scheduling management module and a plurality of resource processing nodes, wherein each resource processing node comprises an FPGA virtualized module and an upper controller module which are in communication connection, and the upper controller module is in communication connection with the resource scheduling management module; the FPGA virtualization module comprises a static area and one or more dynamic reconfigurable areas which are formed by dividing a physical FPGA by adopting a local dynamic reconfiguration technology, wherein each dynamic reconfigurable area is used for deploying and realizing an FPGA resource scheduling task; the upper controller module is used for supervising the FPGA resource use condition of the dynamic reconfigurable area in the FPGA virtualization module; the resource scheduling management module is used for receiving resource scheduling requirements and scheduling and managing the PFGA resources of the dynamic reconfigurable areas in all nodes according to the FPGA resource use conditions of the dynamic reconfigurable areas in each FPGA virtualization module.

Description

FPGA (field programmable Gate array) virtualized resource scheduling system and method

Technical Field

The invention belongs to the technical field of computers, and particularly relates to an FPGA (field programmable gate array) virtualized resource scheduling system and method.

Background

With the current society, the trend of informatization, digitalization, networking and intellectualization is deepened continuously, the data calculation amount and the information processing scale are in explosive growth, and higher requirements are put on the calculation and control capability of a platform, and a satellite-borne computer is taken as an example, and is an embedded system and is usually composed of a microprocessor, a storage module, a communication module, a collection module and the like, and is a brain of a satellite and mainly responsible for tasks such as on-board data management, communication management, energy management, attitude and orbit management, load management and the like. As satellite space tasks become increasingly complex, they also place increasing demands on the performance of on-board computers. Meanwhile, on the other hand, with the slow progress of the semiconductor process and the thermal problem caused by the increase of the chip density, the number of integrated transistors of a single chip is further reduced, so that the computational power growth speed of the single chip is difficult to adapt to the current complex production environment and the requirement of mass data processing. How to increase the resource utilization of chips is a major challenge at present.

For the traditional spaceborne computer, the computing power is sealed in the independent spaceborne computer, the resources of each computer cannot be shared with each other, the resource waste is generated, and the existing closed type spaceborne computing system with independent functions cannot meet the intelligent computing requirement of a spacecraft. Therefore, it is necessary to improve the single-point computing capability, and adopt a cloud computing architecture, and a plurality of heterogeneous computers mutually cooperate to complete a target computing task, share (rare) computing resources, improve computing efficiency, balance computing load, improve computing capability of a satellite-borne computing system, and meet the requirement of intelligent computing.

Therefore, for high-performance computing equipment such as FPGA, dynamic loading and resource management of device functions driven by tasks are realized by means of a virtualization technology. Virtualization techniques may virtualize more virtual platforms on one physical platform, and each of these virtual platforms may be used as separate parts. Virtualization has great advantages over direct use of physical platforms in terms of efficient utilization of resources, dynamic deployment, and high reliability.

FPGAs have gained increasing attention in both cloud computing and edge computing due to their hardware flexibility, superior computing throughput, and low power consumption. Although FPGAs offer great advantages over CPUs and GPUs, these advantages come from a tradeoff of design and usability. In conventional FPGA development models, users typically model application scenarios using Hardware Description Language (HDL), and then map the hardware model onto the FPGA through a specific FPGA development tool, ultimately generating an FPGA image that can be run. The main disadvantage of this development mode is that the FPGA can only be developed and used by a single user, irrespective of the application scenario, the kind of product of the FPGA, etc. For example, for an application that requires little resources and does not need to run continuously, most of the FPGA hardware resources will be idle most of the time. Clearly, it is difficult to fully utilize the FPGA in a space-time domain.

In order to improve the development efficiency of the FPGA, better utilize the logic resources of the FPGA, facilitate the large-scale deployment and application of the FPGA, the FPGA needs to be subjected to a certain degree of logic abstraction, so that a top-level user does not need to pay much attention to the implementation mode and detail of the hardware logic of the FPGA. Thus, FPGA resource virtualization techniques are presented. The FPGA virtualization technology breaks the limitation of time and space dimensions, so that a user can easily schedule and use various resources of a plurality of FPGAs at different times.

Currently mainstream FPGA virtualization technologies include FPGAOverlay technology, partially reconfigurable and virtualized manager technology.

The FPGA Overlay technology provides a more familiar programming architecture and interface for upper users by adding an Overlay layer, and the users can only need to be concerned with upper application implementation without worrying about specific hardware circuit implementation, thereby realizing abstraction and virtualization of hardware resources at the bottom layer of the FPGA.

The Overlay technology is mainly used for abstracting and virtualizing the bottom hardware resources, provides a more familiar programming architecture and interface for the upper-layer user, greatly simplifies the use of the hardware resources by the upper-layer user, and plays a role in abstracting the hardware resources. However, the FPGA virtualization implemented by the Overlay technology has an obvious defect that multiple tenants cannot use FPGA resources simultaneously, so that the FPGA virtualization implemented by the Overlay technology cannot fully use and schedule FPGA resources.

The partial reconfiguration means that one or more areas can be divided into the FPGA, and the areas are programmed and configured independently in the running process of the FPGA so as to change the logic of the circuits in the areas, but the normal running of other circuits of the FPGA is not affected.

The partial reconfigurability enables the FPGA to directly carry out multi-task switching by hardware in two dimensions of time and space, wherein the vFPGA is essentially one or more FPGA areas which can be dynamically reconfigurated, and the vFPGA areas can belong to one user together or belong to a plurality of users respectively and run the same or different applications. When one of the vFPGAs is dynamically reconfigured, the operation of the other vFPGAs is not affected.

By using the partial reconfigurable technology, the FPGA can be divided into a plurality of sub-areas, and used as a virtual FPGA for single or multiple users, and meanwhile, a part of logic resources are reserved as non-reconfigurable areas for realizing necessary infrastructure, such as memory management and network communication. The FPGA virtualization realized by using the partial reconfigurable technology and the virtualization management technology can realize multi-tenant sharing of FPGA resources and can improve the resource utilization rate of the FPGA, but the technology cannot abstract hardware resources like an Overlay technology so as to simplify the application development difficulty of users.

Disclosure of Invention

The invention aims to provide an FPGA virtualized resource scheduling system and method, which realize the concurrent use of single FPGA resources by multiple users through FPGA virtualization, abstract and manage FPGA hardware resources, automatically complete task deployment by an automatic deployment UI interface, design a resource scheduling strategy aiming at the condition of multiple nodes, automatically schedule user tasks to the operation of proper nodes, reduce the complexity of the use of the FPGA resources by the users, and effectively improve the flexibility of the use of the FPGA resources and the utilization rate of the resources.

In order to achieve the above purpose, the invention adopts the following technical scheme:

the first aspect of the embodiment of the invention provides an FPGA virtualized resource scheduling system, which comprises a resource scheduling management module and a plurality of resource processing nodes, wherein each resource processing node comprises an FPGA virtualized module and an upper controller module which are in communication connection, and the upper controller module is in communication connection with the resource scheduling management module;

the FPGA virtualization module comprises a static area and one or more dynamic reconfigurable areas which are formed by dividing a physical FPGA by adopting a local dynamic reconfiguration technology, wherein each dynamic reconfigurable area is used for deploying and realizing an FPGA resource scheduling task;

The upper controller module is used for supervising the FPGA resource use condition of the dynamic reconfigurable area in the FPGA virtualization module;

the resource scheduling management module is used for receiving resource scheduling requirements and scheduling and managing the PFGA resources of the dynamic reconfigurable areas in all nodes according to the FPGA resource use conditions of the dynamic reconfigurable areas in each FPGA virtualization module.

As a preferred scheme, each dynamic reconfigurable area comprises a vFPGA for deploying and realizing the FPGA resource scheduling task and a vFPGA controller for managing the vFPGA; each vFPGA is connected with the system AXI-Internetwork through an AXI4 Master interface and an AXI4 Slave interface;

the vFPGA controller comprises a clutch, a memory management unit, a DMA module and an AXI4-AXIS bridge; the clutch is connected with the vFPGA through an AXI4 interface and an AXIS interface; the memory management unit is respectively connected with the clutch and the AXI-Interconnect through an AXI4 interface; the DMA module is respectively connected with the clutch and the AXI-Interconnect through an AXI4 interface and is connected with the memory management unit; the AXI4-AXIS bridge is respectively connected with the clutch and the AXI-Interconnect through an AXIS interface;

the clutch is used for isolating signal connection between the vFPGA and the static logic;

The memory management unit maps the virtual address space to a corresponding physical address through a page table;

the DMA module is used for realizing direct memory access;

and the AXI4-AXIS bridge is used for realizing data transmission in the vFPGA controller and data transmission among the vFPGAs.

As a preferred scheme, the FPGA virtualization module further comprises a PICE protocol stack unit, wherein the PICE protocol stack unit is used for maintaining a PICE protocol stack and is also used for communicating the vFPGA in the FPGA virtualization sub-module with the upper controller module;

the PICE protocol stack unit comprises an application layer, a transaction layer, a data link layer and a physical layer which are connected in sequence;

the application layer is connected with the upper controller module through a PCIe interface, and converts the acquired FPGA resource scheduling task requirement into data and sends the data to the transaction layer;

the transaction layer receives data from the application layer and encapsulates the data into a data packet to be sent to the data link layer, and can also receive a data message from the data link layer and forward the data message to the application layer;

the data link layer receives the data message from the transaction layer, adds a Sequence Number prefix and a CRC suffix, and uses ACK/NAK to ensure the reliability of data message transmission;

the physical layer is used for receiving and forwarding various data packets and also used for creating and decoding special sequences of FPGA links used for synchronizing and managing each dynamic reconfigurable area.

As a preferred scheme, each upper controller submodule comprises a vFPGA controller driver, a system manager, a system monitor, an FPGA hardware manager and a hardware resource pool;

the vFPGA controller is driven and used for operating the vFPGA controller;

the system manager is used for communicating with the resource scheduling management module, is scheduled and managed by the resource scheduling management module and is responsible for creating and destroying the vFPGA;

the system monitor is used for periodically monitoring the working state of the FPGA, the running condition of each vFPGA, the use condition of the virtualized resources of the FPGA and the flow statistics condition of the PCIE protocol stack module, and permanently recording the monitored data information into the database;

the hardware resource pool comprises a vFPGA resource pool and an FPGA memory pool and is used for managing and recording the use of FPGA hardware resources;

and the FPGA hardware manager is used for communicating related operations with the FPGA virtualization module and implementing the allocated hardware resources on specific FPGA hardware through driving.

As a preferred scheme, the system manager performs network communication with the resource scheduling management module, and transmits the FPGA virtualized resource information of the computing node to the resource scheduling management module;

receiving a scheduling request of a resource scheduling management module, making corresponding processing according to different requests, and delivering the processing to various downstream managers for implementation;

And receiving a resource information monitoring request of the resource scheduling management module, and uploading information detected by the system monitor to the resource scheduling management module.

As a preferred scheme, the resource scheduling management module further comprises an automatic deployment UI;

the automatic deployment UI is used for receiving the FPGA resource scheduling requirement and carrying out network communication with the resource scheduling management module;

the resource scheduling management module comprises an external interface, a node management unit and a resource scheduling unit;

the external interface is used for providing an automatic deployment interface of the FPGA resource scheduling task according to the FPGA resource scheduling task requirement acquired by the automatic deployment UI;

the node management unit is used for monitoring the FPGA resource use condition of the dynamic reconfigurable area in each node;

and the resource scheduling unit is used for scheduling the received FPGA resource scheduling task to a proper dynamic reconfigurable area for operation according to the FPGA resource use condition of the dynamic reconfigurable area in each node.

A second aspect of the embodiment of the present invention provides a method for scheduling FPGA virtualized resources, including:

the resource scheduling management module receives the FPGA resource scheduling task and deploys the FPGA resource scheduling task to a proper dynamic reconfigurable area for operation according to the requirements of the FPGA resource scheduling task and the occupation condition of the FPGA resources of each node;

And the dynamic reconfigurable area for deploying the FPGA resource scheduling task is used for creating a vFPGA for running the FPGA resource scheduling task according to the requirement of the FPGA resource scheduling task.

As a preferred scheme, the step of acquiring the occupation condition of the FPGA resources of each node by the resource scheduling management module comprises the following steps:

after the resource scheduling management module operates, the node management unit distributes service through multicast, and the service carries IP and port information;

the system manager of each upper node controller receives the multicast message and establishes network connection with the IP and the port in the multicast;

after the connection is established successfully, the system manager of each upper node controller actively initiates a registration request, and after the node management unit receives the registration request, the node management unit takes in the resources of the node to manage so as to complete registration;

after registration, the system manager of each upper node controller reports the FPGA resource use condition period of each node to the node management unit.

As a preferred solution, the step of creating a vffpga for running the FPGA resource scheduling task according to the requirements of the FPGA resource scheduling task in the dynamically reconfigurable area for deploying the FPGA resource scheduling task includes:

starting a clutch in a vFPGA controller of the dynamic reconfigurable area, resetting the reconfigurable module, downloading a corresponding bit stream for reconfiguration, configuring a memory management unit, and configuring an IO_LUT;

And resetting the reconfigurable module and closing the clutch to finish the creation work of the vFPGA so as to load the received FPGA resource scheduling task into the vFPGA for operation.

As a preferred solution, the process of configuring the memory management unit for the dynamically reconfigurable area is as follows:

s1, judging whether the mapped address space is smaller than a required mapping space N, if not, ending the configuration flow, and if so, executing a step S2;

s2, taking out a memory resource from the FPGA memory pool, and adding a mapping record to a memory management unit of the vFPGA by driving;

s3, repeating the steps S1-S2 until the mapped address space is greater than or equal to the required mapping space N, and ending the configuration flow.

The beneficial effects of the invention are as follows:

1. according to the invention, the concurrent use of single FPGA resources by multiple users is realized through the FPGA virtualization method, the problem that the FPGA can only be developed and used by a single user is solved, and the purpose of improving the utilization rate of the FPGA resources is achieved. The FPGA resources can be fully utilized, so long as the residual FPGA resources can meet the operation requirements of user tasks, the user tasks can be operated in the vFPGA, the problem that only one user task is operated on the FPGA, a large number of FPGA resources are in idle states, and the resource utilization rate is low is solved.

2. The upper controller module can respond to the FPGA resource state request of the resource scheduling management module and respond the resource information recorded in the database to the resource scheduling management service. Meanwhile, the upper controller module supports the management of FPGA resources, responds to the FPGA resource request of the resource scheduling management module, and schedules user tasks to a proper position for operation if the resources meet the resource scheduling management service requirements.

3. The method has the advantages that the use difficulty of FPGA resources is reduced, the FPGA is used as a whole in the traditional method, the FPGA program is loaded into the FPGA through the simulator to run, the user only needs to establish network connection with an upper controller, and the operation such as resource application, FPGA task creation and the like is carried out through a request response mode, so that the use threshold of the FPGA is reduced.

4. The whole chain from resource virtualization to resource management to resource scheduling is developed by using C/C++ self-grinding, K8S is not required to be deployed, the resource occupation is small, the deployment is simple, the weight is reduced, and the flexibility is high.

Drawings

In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic structural diagram of an FPGA virtualized resource scheduling system according to a first embodiment.

Fig. 2 is a schematic diagram of the structure of an FPGA virtualization module.

Fig. 3 is a schematic diagram of the architecture of a vffpga controller.

Fig. 4 is a schematic structural diagram of a PCIE protocol stack unit.

Fig. 5 is a schematic structural diagram of an upper controller module.

FIG. 6 is a flowchart of the operation of the system manager.

Fig. 7 is a schematic diagram of the structure of the resource scheduling management module.

Fig. 8 is a flow chart for creating a vffpga in a dynamically reconfigurable area.

Detailed Description

The following specific examples are presented to illustrate the present invention, and those skilled in the art will readily appreciate the additional advantages and capabilities of the present invention as disclosed herein. The invention may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present invention. It should be noted that the following embodiments and features in the embodiments may be combined with each other without conflict.

Embodiment one:

referring to fig. 1, the present embodiment provides an FPGA virtualized resource scheduling system, including a resource scheduling management module and a plurality of resource processing nodes, where each resource processing node includes an FPGA virtualized module and an upper controller module that are communicatively connected, and the upper controller module is communicatively connected with the resource scheduling management module;

The FPGA virtualization module comprises a static area and one or more dynamic reconfigurable areas which are formed by dividing a physical FPGA by adopting a local dynamic reconfiguration technology, wherein each dynamic reconfigurable area is used for deploying and realizing the FPGA resource scheduling task. The FPGA virtualization module is divided into a static area and a dynamic reconfigurable area, the hardware resources of the whole FPGA comprise dsp, pcie, lut, ff, bram, pblock and the like, and specifically, the quantity of the hardware resources of each FPGA virtualization module is determined by actual hardware, and other hardware resources except for some hardware resources used by a static area curing program (wherein the static area program occupies a small proportion of resources and occupies less than 5% of the total resources of the FPGA) are used in the dynamic area.

The static area is an area which is divided on the FGPA and runs the curing program, the area can not be dynamically reconfigured, and the mcs file corresponding to the static area can be cured in the flash in the actual application process. The dynamic reconfigurable area comprises Microblaze (embedded soft core), HWIAP IP core and Pcie interface. Software programming is carried out on Microblaze through SDK to realize the local dynamic configuration function of FPGA; the HWIAP IP core is used for realizing the read-write operation of the FPGA to the configuration space through the AXI bus, and a user finishes the operation of writing bit streams into the vFPGA through the HWIAP IP core when creating the FPGA virtualization task. The internal structure of the FPGA virtualization module is shown with reference to fig. 2.

and the resource scheduling management module is autonomously realized by using C/C++, is used for receiving resource scheduling requirements, and performs scheduling management on PFGA resources of the dynamic reconfigurable areas in all nodes according to the FPGA resource use conditions of the dynamic reconfigurable areas in each FPGA virtualization module.

Further, each dynamic reconfigurable area comprises a vFPGA for deploying and realizing the FPGA resource scheduling task and a vFPGA controller for managing the vFPGA; each vFPGA is connected with the system AXI-Internetwork through an AXI4 Master interface and an AXI4 Slave interface;

referring to fig. 3, the vfga controller includes a clutch, a memory management unit, a DMA module, an AXI4-AXIs bridge; the clutch is connected with the vFPGA through an AXI4 interface and an AXIS interface; the memory management unit is respectively connected with the clutch and the AXI-Interconnect through an AXI4 interface; the DMA module is respectively connected with the clutch and the AXI-Interconnect through an AXI4 interface and is connected with the memory management unit; the AXI4-AXIS bridge is respectively connected with the clutch and the AXI-Interconnect through an AXIS interface;

the clutch is used for isolating signal connection between the vFPGA and the static logic, and the interface mainly comprises AXI4, AXIS and clock signals;

the DMA module is used for realizing direct memory access;

Specific:

the vFPGA accesses the memory through an AXI4 interface, and the AXIS interface is provided with one group and two groups and is used for data transmission between modules. And the data transmission between the upper controller and the FPGA is completed through a PCIE protocol stack, and the data transmission between two vFPGAs in the FPGA virtualization module is directly realized through AXI4-Interconnect in the module.

The data exchange between the vFPGA is carried out by using an AXIS interface, and the system exchange structure is realized by adopting AXI4, and related components are required to be designed to complete protocol conversion. When an AXIS packet is sent, the DMA module queries the written destination address from the IO LUT via the TDEST signal value, thereby writing the AXIS packet to the address via the AXI4 interface. For the receiving interface, the AXI4-AXIs bridge is required to stream AXI4 write operations to the AXIs protocol, and determine the value of TDEST from the address offset written.

The vFPGA can directly use an AXI4 protocol for read-write access to the memory access, but in order to ensure the isolation between memories of the vFPGA, a memory management unit is required to access for management. Each vFPGA is provided with a set of independent virtual address space, the virtual address space of each vFPGA is mapped to different physical addresses through a memory management unit, and the address spaces which can be accessed by different vFPGAs are mutually exclusive on the physical addresses. The address mapping table in the memory management unit is configured by PS before the vFPGA is deployed, the address range with the size of 4MB is designed to be one page in consideration of the resource cost of the memory management unit, and the memory management unit records the offset from the virtual address to the physical address of each page.

Further, the FPGA virtualization module further comprises a PICE protocol stack unit, wherein the PICE protocol stack unit is used for maintaining the PICE protocol stack and is also used for communicating the vFPGA in the FPGA virtualization sub-module with the upper controller module. The PICE protocol stack unit is mainly controlled by an upper layer controller, after the initialization of the vFPGA is completed, the PICE protocol stack unit configures PCIE related resources for the vFPGA, creates a required FIFO, and the vFPGA can receive and SEND related data streams through an AXIS interface to complete SEND/RECV operation;

referring to fig. 4, the pice protocol stack unit is implemented in layers, which include a plurality of layers, namely an application layer, a transaction layer, a data link layer, and a physical layer, from top to bottom. Each layer of the PICE protocol stack unit is realized through hardware logic, and data messages are generated by an application layer during transmission and finally transmitted out through a transaction layer, a data link layer and a physical layer; the receiving end is the opposite step, the data first passes through the physical layer, then is sent upwards to the data link layer, the transaction layer and finally reaches the application layer. The PICE protocol stack unit is a PCI bus.

The application layer is connected with the upper controller module through a PCIe interface, and converts the acquired FPGA resource scheduling task requirement into data and sends the data to the transaction layer. PCI spec defines 256 bytes of configuration space, and PCIe bus almost completely reserves PCI bus configuration space for PCI device compatibility. And extends the configuration space to 4KB for supporting new functions in some PCIe buses, such as Capability, power management, MSI, etc. PCIe expands the configuration space to 4KB, and the original CF8/CFC access still has access to the first 256 bytes of all PCIe configuration space, but no remaining space is accessed. PCIe introduces an enhanced configuration space access mechanism that allows access to the configuration space just like memory by mapping it to MMIO space, thus allowing access to the full 4KB configuration space.

The transaction layer receives data from the application layer and encapsulates the data into data packets to be sent to the data link layer, and can also receive data messages from the data link layer and forward the data messages to the application layer. The transaction layer defines transactions used by the PCIE protocol stack module, most of which are PCI bus compatible in PCIE Spec, specifying four types of requests (requests): memory, IO, configuration and Messages. The first three are inherited from PCI/PCI-X buses, and the fourth message is a newly added type of PCIe. The transaction layer receives data from the core layer and encapsulates it TLP (Transaction Layer Packet) to the data link layer. Alternatively, the transaction layer may receive data packets from the data link layer and forward the data packets to the core layer.

And the data link layer receives the data message from the transaction layer, adds a Sequence Number prefix and a CRC suffix, and uses ACK/NAK to ensure the reliability of data message transmission. It also defines a variety of DLLP (Data Link Layer Pakcet), DLLP results from the data link layer ending at the data link layer.

The physical layer is used for receiving and forwarding various data packets and also used for creating and decoding special sequences of FPGA links used for synchronizing and managing each dynamic reconfigurable area. The physical layer is the lowest layer of the PCIe bus, interconnecting PCIe devices together. The physical layer also implements the link training and initialization functions, which are accomplished by LTSSM.

Further, the FPGA virtualization module needs an upper controller module to manage hardware resources, and communicates with a resource scheduling management module through the upper controller module, and in the design, the upper controller module program runs in FT 2000/4.

Referring to fig. 5, each upper layer controller sub-module includes a vfga controller driver, a system manager, a system monitor, an FPGA hardware manager, a hardware resource pool.

The vFPGA controller is driven and used for operating the vFPGA controller. Because the whole life cycle of the vFPGA is controlled by the upper controller submodule, and a Linux system is operated on the FT2000/4 processor at the end of the upper controller submodule. In order to facilitate the upper controller submodule to operate the bottom hardware, a vFPGA controller driver is designed.

For the vFPGA controller, the relevant information of the register is shown in table 1, and the driver mainly needs to operate three parts: clutch state, mapping table of the io_lut and memory mapping table of the memory management unit MMU.

Table 1 fpga controller register table

The driving implementation of the device adopts character device driving implementation based on a device tree. For each vFPGA controller device, an associated character device file is exposed to the user, who can control the underlying controller hardware through the operation/dev/vFPGA_n device file link. The user communicates with the device driver by way of a system call from ioctl.

The calling mode parameters are shown in table 2:

table 2vFPGA controller driving interface

For the DECOUPL_ON and DECOUPL_OFF, the system call needs to be completed by matching with an interrupt service routine. When the user initiates these two commands, the driver modifies the value of decuppler_ctrl, respectively, and then the driver blocks to wait for a completion amount provided by the interrupt service routine corresponding to the interrupt triggered after the completion of the relevant operation by the DECOUPLER. The other command orders perform read-write operation on the register in a non-blocking mode.

The system manager is used for communicating with the resource scheduling management module, is scheduled and managed by the resource scheduling management module and is responsible for creating and destroying the vFPGA. The main work of the system manager is to receive and process related operation requests of the resource scheduling management service, and the requests are realized on specific hardware resources of the current FPGA according to the current resource condition. In addition, the system manager may also receive a request from the resource scheduling management module to retrieve the monitoring results of the system monitor from the database and return the results.

Further, referring to fig. 6, the workflow of the system manager is: the system manager performs network communication with the resource scheduling management module, and transmits the FPGA virtualized resource information of the computing node to the resource scheduling management module;

The system monitor is used for periodically monitoring the working state of the FPGA, the running condition of each vFPGA, the use condition of the virtualized resources of the FPGA and the flow statistics condition of the PCIE protocol stack module, and permanently recording the monitored data information into the database.

The hardware resource pool comprises a vFPGA resource pool and an FPGA memory pool and is used for managing and recording the use of FPGA hardware resources. The FPGA hardware resources comprise reconfigurable blocks (such as CLB/Block RAM/DSP, etc.), resources in PCIE Stack and FPGA physical memory resources. The management of the resources is relatively simple, and only the possession of each task for the two resources needs to be recorded, so that the same task can only possess one reconfigurable area. For FPGA physical memory resources, since the bottom MMU uses a size of 4M as one page, in the FPGA physical memory resource pool, all physical memories need to be managed in a 4M size range. If the physical memory size of the FPGA in the system is 8GB, the number of the memory resources divided by the FPGA is 8GB/4M, namely 2048.

When a task is deployed, resources are taken out of the free resources in the three pools to deploy and run the task. Firstly, the task is deployed into a reconfigurable area according to the reconfigurable area corresponding to the bit stream file of the task, and then corresponding memory is allocated to the task according to the requirement of the task, which is required to be allocated according to the memory requirement in the task description. If a task needs 4G of memory, i.e., the virtual address space of the FPGA is 0-4GB, 1024 resources need to be fetched from the free area in the FPGA memory pool. When a task is offloaded, resources allocated to the task are reclaimed to free areas of the resource pools.

And the FPGA hardware manager is used for communicating related operations with the FPGA virtualization module and implementing the allocated hardware resources on specific FPGA hardware through driving. These related operations are the configuration of the reconfigurable area, the control of the clutch during the reconfiguration, the configuration of the MMU, the io_lut, etc.

The database can be selected according to actual requirements, and the use of sqlite for data persistence records of other components in the upper controller module is suggested here, because sqlite is a server-free lightweight database, and has small resource occupation and easy use.

Further, the resource scheduling management module further comprises an automatic deployment UI;

the automatic deployment UI is used for receiving the FPGA resource scheduling requirement, carrying out network communication with the resource scheduling management module, realizing parameter input by a user through a UI interface, and calling a corresponding restfulApi interface by the UI client according to parameters filled by the user to realize automatic deployment of the task.

Referring to fig. 7, the resource scheduling management module includes an external interface, a node management unit, and a resource scheduling unit;

and the external interface is used for providing an FPGA resource scheduling task automatic deployment interface according to the FPGA resource scheduling task requirements acquired by the automatic deployment UI. The external interface mainly provides a RestfrulAPI service, and the automatic deployment of tasks can be realized by calling the RestfrulApi interface.

And the node management unit is used for monitoring the FPGA resource use condition of the dynamic reconfigurable area in each node.

Specific:

the resource scheduling management module mainly provides resource management service for users, particularly the use scene of multiple nodes, the users do not need to care about which node the tasks should be deployed to and the occupation condition of the resources of each node, and the users only need to send the tasks and the resource information required by the tasks to the resource scheduling management module through the graphical UI, so that the automatic deployment of the tasks can be completed. The user does not feel the internal scheduling in the whole process, and the FPGA resource scheduling task deployment experience can be greatly improved. The resource scheduling management module is deployed on any CPU mainboard operating system in the same local area network.

After the resource scheduling management software module operates, the node management unit firstly distributes service through multicast, the service carries IP and port information, the system manager of each upper controller submodule establishes network connection with the IP and port in the multicast after receiving the multicast information, each system manager initiatively initiates a registration request to the node management unit after successful connection establishment, the request information comprises the resource type, the resource state, the resource use condition and the like of the node, the node management unit receives the registration request and then brings the resource of the node into management, and after successful registration, the system manager periodically reports the FPGA resource use condition to the node management unit, so that the resource scheduling management module can master the resource use condition and the state of each node, and once a user initiates a task deployment request, the user task can be scheduled to a proper node to operate according to a scheduling strategy.

When a user creates an FPGA task, a resource scheduling unit firstly traverses each node to find out the node meeting the resource requirement, if no node meeting the requirement exists, the task deployment fails, if only one node meets the requirement, the task is deployed to the node for operation, and if a plurality of nodes meet the requirement, the task is scheduled to a proper node for operation according to a scheduling strategy. Taking the resource maximum utilization policy as an example, 500 Block RAMs are needed for the user task, and assuming that two nodes meet the requirement, one node is provided with 600 Block RAMs and the other node is provided with 2000 Block RAMs, the user task is scheduled to the node with 600 Block RAMs, so that the maximization of the resource utilization of a single node can be ensured to the greatest extent. The scheduling strategy is custom designed according to actual demands, the scheduling strategy is realized in a dynamic library mode, and a developer can realize replacement of the scheduling strategy only by realizing the corresponding strategy according to interface requirements.

Embodiment two:

the embodiment provides a method for scheduling FPGA virtualized resources, which comprises the following steps:

Further, the step of the resource scheduling management module obtaining the occupation condition of the FPGA resources of each node comprises the following steps:

Further, referring to fig. 8, the step of creating a vfg for running FPGA resource scheduling tasks for the dynamically reconfigurable area includes:

Still further, referring to fig. 8, the process of configuring the memory management unit for the dynamically reconfigurable area is as follows:

The above examples are merely illustrative of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention, and various modifications and improvements made by those skilled in the art to the technical solution of the present invention should fall within the protection scope of the present invention without departing from the design spirit of the present invention.

Claims

1. The FPGA virtualized resource scheduling system is characterized by comprising a resource scheduling management module and a plurality of resource processing nodes, wherein each resource processing node comprises an FPGA virtualized module and an upper controller module which are in communication connection, and the upper controller module is in communication connection with the resource scheduling management module;

2. The FPGA virtualized resource scheduling system of claim 1 wherein each dynamically reconfigurable area comprises a vffpga for deploying and implementing FPGA resource scheduling tasks, and a vffpga controller for managing the vffpga; each vFPGA is connected with the system AXI-Internetwork through an AXI4 Master interface and an AXI4 Slave interface;

the DMA module is used for realizing direct memory access;

3. The FPGA virtualized resource scheduling system of claim 1 wherein the FPGA virtualized module further comprises a PICE protocol stack unit, the PICE protocol stack unit being configured to maintain a PICE protocol stack and being further configured to communicate with the upper controller module by the FPGA in the FPGA virtualized sub-module;

4. The FPGA virtualized resource scheduling system of claim 1, wherein each upper layer controller sub-module comprises a vfg controller driver, a system manager, a system monitor, an FPGA hardware manager, a hardware resource pool;

the vFPGA controller is driven and used for operating the vFPGA controller;

5. The system for scheduling FPGA virtualized resources according to claim 4, wherein the system manager is in network communication with the resource scheduling management module, and transmits the FPGA virtualized resource information of the computing node to the resource scheduling management module;

6. The FPGA virtualized resource scheduling system of claim 1, wherein the resource scheduling management module further comprises an automated deployment UI;

7. An FPGA virtualized resource scheduling method based on the FPGA virtualized resource scheduling system of any of claims 4-6, comprising:

the dynamic reconfigurable area for deploying the FPGA resource scheduling task is used for creating a vFPGA for running the FPGA resource scheduling task according to the requirement of the FPGA resource scheduling task.

8. The method for scheduling FPGA virtualized resources according to claim 7, wherein the step of the resource scheduling management module obtaining occupancy of FPGA resources of each node comprises:

9. The method of claim 7, wherein the step of creating a vffpga for running the FPGA resource scheduling task in accordance with the requirements of the FPGA resource scheduling task in the dynamically reconfigurable area for deploying the FPGA resource scheduling task comprises:

10. The method for scheduling FPGA virtualized resources according to claim 9, wherein the process of configuring the memory management unit for the dynamically reconfigurable area is: