WO2021223744A1 - 实现热迁移的方法、芯片、板卡和存储介质 - Google Patents

实现热迁移的方法、芯片、板卡和存储介质 Download PDF

Info

Publication number
WO2021223744A1
WO2021223744A1 PCT/CN2021/092199 CN2021092199W WO2021223744A1 WO 2021223744 A1 WO2021223744 A1 WO 2021223744A1 CN 2021092199 W CN2021092199 W CN 2021092199W WO 2021223744 A1 WO2021223744 A1 WO 2021223744A1
Authority
WO
WIPO (PCT)
Prior art keywords
virtual
hardware
data
migrated
specific virtual
Prior art date
Application number
PCT/CN2021/092199
Other languages
English (en)
French (fr)
Inventor
鲁海波
孟小甫
Original Assignee
中科寒武纪科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中科寒武纪科技股份有限公司 filed Critical 中科寒武纪科技股份有限公司
Priority to US17/923,677 priority Critical patent/US20230185595A1/en
Priority to EP21800406.7A priority patent/EP4148568A1/en
Publication of WO2021223744A1 publication Critical patent/WO2021223744A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/485—Task life-cycle, e.g. stopping, restarting, resuming execution
    • G06F9/4856—Task life-cycle, e.g. stopping, restarting, resuming execution resumption being on a different machine, e.g. task migration, virtual machine migration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing

Definitions

  • This disclosure generally relates to the field of computers. More specifically, the present disclosure relates to a method, chip, board, and computer-readable storage medium for realizing thermal migration.
  • Live migration is also called live migration and live migration, that is, the virtual machine saves (SAVE)/restores (LOAD) procedures to completely save the running state of the entire virtual machine, and migrate from one physical server to another. On a physical server. After recovery, the virtual machine still runs smoothly, and the user will not notice any difference.
  • SAVE virtual machine saves
  • LAD restores
  • the solution of the present disclosure provides a method, a chip, a board, and a computer-readable storage medium for realizing thermal migration.
  • a system for realizing hot migration which is connected to the kernel space, and the system includes hardware and read-only memory.
  • Hardware virtualization is a plurality of virtual hardware
  • the read-only memory stores firmware
  • the firmware includes physical functions to: receive a hot migration start request from the kernel space, and the hot migration start request specifies specific virtual hardware.
  • the specific virtual hardware is one of the plurality of virtual hardware; sending an instruction to be migrated to the kernel space, the instruction to be migrated records data to be migrated, and the data to be migrated is associated with the specific virtual hardware;
  • the specific virtual hardware fetches the data to be migrated in batches and sends it to the kernel space; and after the data to be migrated is sent, sends an end signal to the kernel space.
  • a system for realizing hot migration which is connected to the kernel space, and the system includes hardware and read-only memory.
  • Hardware virtualization is multiple virtual hardware.
  • the read-only memory stores firmware, which includes physical functions and multiple virtual functions.
  • the physical function is used to receive data to be migrated, the data to be migrated is associated with specific virtual hardware, the specific virtual hardware is one of the multiple virtual hardware, and the multiple virtual functions are managed by the physical function.
  • the virtual hardware corresponds to a virtual function, and the physical function sends the data to be migrated to the specific virtual hardware through the virtual function. After the data to be migrated is sent, the physical function sends an end signal to the kernel space.
  • an integrated circuit device including the aforementioned system, and a board card, including the aforementioned integrated circuit device.
  • a method for performing hot migration to save a path to a system includes hardware and is virtualized into multiple virtual hardware.
  • the method includes: receiving a hot migration start request, where the hot migration start request specifies a specific virtual hardware, and the specific virtual hardware is one of the multiple virtual hardware; idle a specific virtual function, and the specific virtual function corresponds to The specific virtual hardware; send the data to be migrated in batches from the specific virtual hardware; and send an end signal after the data to be migrated is sent.
  • a method for performing a hot migration recovery path on a system including hardware, which is virtualized into multiple virtual hardware.
  • the method includes: receiving data to be migrated, the hot migration data corresponding to specific virtual hardware, and the specific virtual hardware is one of the multiple virtual hardware; idle a specific virtual function, the specific virtual function corresponds to all The specific virtual hardware; sending the data to be migrated to the specific virtual hardware through a specific virtual function; and sending an end signal after the data to be migrated is sent.
  • a computer-readable storage medium on which is stored hot-migrated computer program code, and when the computer program code is run by a processor, the foregoing method is executed.
  • This disclosure can realize the hot migration of virtualized device information from the source server to the destination server.
  • FIG. 1 is a schematic diagram showing an artificial intelligence chip framework of an embodiment of the present disclosure
  • FIG. 2 is a schematic diagram showing the internal structure of a computing device according to an embodiment of the disclosure.
  • FIG. 3 is a flowchart of the migration saving path of the embodiment of the present disclosure.
  • FIG. 4 is a schematic diagram showing the migration and preservation path of the source server in the embodiment of the present disclosure.
  • FIG. 5 is a flowchart of a migration restoration path in an embodiment of the present disclosure
  • FIG. 6 is a schematic diagram showing the migration and recovery path of the destination server in the embodiment of the present disclosure.
  • FIG. 7 is a structural diagram showing an integrated circuit device of an embodiment of the present disclosure.
  • Fig. 8 is a schematic diagram showing a board of an embodiment of the present disclosure.
  • the present disclosure relates to a framework using virtualization technology, which is applied to an application specific integrated circuit, for example, a machine learning device used for a neural network, and the machine learning device may be a convolutional neural network accelerator.
  • an application specific integrated circuit for example, a machine learning device used for a neural network
  • the machine learning device may be a convolutional neural network accelerator.
  • the following will illustrate with an example of artificial intelligence chip.
  • FIG. 1 is a framework diagram of artificial intelligence chip virtualization.
  • the framework 100 includes a user space 102, a kernel space 104, and a system on chip 106, which are separated by a dotted line on the figure.
  • the user space 102 is the running space of the user program. It only performs simple operations and cannot directly call system resources. It must pass through the system interface to issue instructions to the kernel.
  • the kernel space 104 is the space where the kernel code runs. Any command can be executed and all the resources of the system can be called.
  • the system-on-chip 106 is each module of the artificial intelligence chip, which cooperates with the user space 102 through the kernel space 104.
  • the hardware of the user space 102 is collectively referred to as a device or device, and the hardware of the system-on-chip 106 is collectively referred to as a device or unit as a distinction.
  • This arrangement is only for describing the technology of this embodiment more clearly, and does not impose any limitation on the technology of this disclosure.
  • this embodiment is illustrated by virtualizing one component into four virtual components, but this disclosure does not limit the number of virtual components.
  • the user space 102 is controlled by the hardware monitor tool 108 before virtualization is executed, and the information of the system-on-chip 106 is obtained by calling the interface.
  • the hardware monitor tool 108 can not only collect the information of the system-on-chip 106, but also obtain the overhead of the system-on-chip 106 resources by the upper-level software in real time, so that the user can grasp the detailed information and status of the current system-on-chip 106 in real time.
  • the detailed information and status can be: Dozens of data such as hardware device model, firmware version number, driver version number, device utilization, storage device overhead status, board power consumption and board peak power consumption, and fast peripheral component interconnection (PCIe). Based on the different versions and usage scenarios of the hardware monitor tool 108, the monitored information content and quantity will vary.
  • the user virtual machine 110 is an abstraction and simulation of the real computing environment.
  • the system will allocate a set of data structures to manage the state of the user virtual machine 110. Its data structure includes a full set of registers, physical memory usage, virtual device status, and so on.
  • the physical space of the user space 102 in this embodiment is virtualized into four virtual spaces 112, 114, 116, and 118. These four virtual spaces 112, 114, 116, and 118 are independent of each other and can be equipped with different guest operating systems.
  • guest operating system 1, guest operating system 2, guest operating system 3, and guest operating system 4 the guest operating systems can be Windows, Linux, Unix, iOS, Android, etc., and each guest operating system is separately Run different applications.
  • the user virtual machine 110 is implemented by a fast emulator (QEMU).
  • QEMU is an open source virtualization software written in C language. It virtualizes interfaces through dynamic binary conversion and provides a series of hardware models to make guest operating system 1, guest operating system 2, guest operating system 3, and guest operating system 4 They all think that they have direct access to the system-on-chip 106.
  • User space 102 includes processors, memories, I/O devices, etc.
  • QEMU can virtualize the processors in user space 102 into four virtual processors, and virtualize the memory into four virtual memories, as well as I/O devices. The virtualization is four virtual I/O devices.
  • Each guest operating system occupies a portion of the resources of the user space 102, for example, each occupies a quarter, that is, they can access a virtual processor, a virtual memory, and a virtual I/O device to execute the guest operating system's resources. Task. Through this mode, guest operating system 1, guest operating system 2, guest operating system 3, and guest operating system 4 can operate independently.
  • the kernel space 104 carries the kernel virtual machine 120 and the chip driver 122.
  • the kernel virtual machine 120 is matched with QEMU, and is mainly responsible for the virtualization of the kernel space 104 and the system on chip 106, so that each guest operating system can obtain its own address space when accessing the system on chip 106.
  • the space on the system-on-chip 106 mapped to the guest operating system is actually a virtual component mapped to this process.
  • the kernel virtual machine 120 includes a physical function driver, which is a driver that specifically manages the global functions of the SR-IOV device, and generally requires a higher authority than an ordinary virtual machine to operate it.
  • the physical function driver includes all the functions of the traditional driver, and the user space 102 can access the I/O resources of the system-on-chip 106.
  • QEMU uses the system call interface provided by the kernel virtual machine 120 to set the kernel.
  • QEMU uses the virtualization function of the kernel virtual machine 120 to provide hardware for its own virtual machine. Virtualization acceleration to improve the performance of virtual machines. From the perspective of the kernel virtual machine 120, when a user cannot directly interact with the kernel space 104, the management tool of the user space 102 needs to be used, and therefore, the tool that runs in the user space 102, QEMU, needs to be used.
  • the chip driver 122 is used to drive the physical function (PF) 126.
  • PF physical function
  • the user space 102 is not accessed by the hardware monitor tool 108 via the chip driver 122 to access the system-on-chip 106. Therefore, the guest operating system 1, the client The operating system 2, the guest operating system 3, and the guest operating system 4 are respectively configured with a user-side kernel space 124 for loading the chip driver 122 so that each guest operating system can still drive the system-on-chip 106.
  • the system-on-chip 106 implements virtualization through single root I/O virtualization (SR-IOV) technology.
  • SR-IOV technology is based on soft The hardware is combined to realize the virtualization of the components of the system-on-chip 106.
  • SR-IOV technology is a virtualization solution that allows efficient sharing of PCIe resources between virtual machines.
  • SR-IOV technology enables a single PCIe resource to be shared by multiple virtual components of the system-on-chip 106, providing dedicated resources for these virtual components . In this way, each virtual component has its own corresponding uniquely accessible resource.
  • the system-on-chip 106 of this embodiment includes hardware and firmware.
  • the hardware includes a read-only memory ROM (not shown in the figure) for storing firmware
  • the firmware includes a physical function 126 for supporting or cooperating with the PCIe function of SR-IOV.
  • the physical function 126 has the right to fully configure PCIe resources.
  • the physical function 126 will virtualize multiple virtual functions (VF) 128, in this embodiment there are four virtual functions 128.
  • the virtual function 128 is a lightweight PCIe function that is managed by the physical function 126 and can share PCIe physical resources with the physical function 126 and other virtual functions 128 associated with the same physical function 126.
  • the virtual function 128 is only allowed to control the resources that the physical function 126 allocates to itself.
  • each virtual function 128 can access its own PCIe configuration space through its own bus, device, and function number.
  • Each virtual function 128 has a memory space for mapping its register set. The virtual function 128 driver operates on the register set to enable its function, and directly assigns it to the corresponding user virtual machine 110. Although it is virtual, the user virtual machine 110 is made to think that it is an actual PCIe device.
  • the hardware of the system-on-chip 106 also includes a computing device 130, a video codec device 132, a JPEG codec device 134, a storage device 136, and PCIe 138.
  • the computing device 130 is an intelligent processing unit (IPU), which is used to perform neural network convolution calculation;
  • the video codec device 132 is used to code and decode video data;
  • the JPEG codec device 134 It is used to encode and decode static pictures using the JPEG algorithm;
  • the storage device 136 can be a dynamic random access memory (DRAM) to store data;
  • PCIe 138 is the aforementioned PCIe.
  • PCIe 138 will virtualize Into four virtual interfaces 140, the virtual function 128 and the virtual interface 140 have a one-to-one correspondence, that is, the first virtual function is connected to the first virtual interface, the second virtual function is connected to the second virtual interface, and so on.
  • the computing device 130 is virtualized into four virtual computing devices 142
  • the video codec device 132 is virtualized into four virtual video codec devices 144
  • the JPEG codec device 134 is virtualized into four virtual JPEGs
  • the codec device 146 virtualizes the storage device 136 into four virtual storage devices 148.
  • Each guest operating system is configured with a set of virtual kits.
  • Each set of virtual kits includes a user virtual machine 110, a virtual interface 140, a virtual function 128, a virtual computing device 142, a virtual video codec device 144, and a virtual JPEG Codec device 146 and a virtual storage device 148.
  • Each group of virtual suites run independently without affecting each other, and are used to perform the tasks delivered by the corresponding guest operating system to ensure that each guest operating system can access the configured virtual computing through the configured virtual interface 140 and virtual function 128
  • the device 142, the virtual video codec device 144, the virtual JPEG codec device 146, and the virtual storage device 148 The device 142, the virtual video codec device 144, the virtual JPEG codec device 146, and the virtual storage device 148.
  • each guest operating system responds to different tasks when performing tasks, and the hardware that needs to be accessed may also be different. For example, if a task is to perform matrix convolution calculations, the guest operating system will pass the configuration Virtual interface 140 and virtual function 128 to access the configured virtual computing device 142; if a task is to perform video encoding and decoding, the guest operating system will access the configured virtual video through the configured virtual interface 140 and virtual function 128 Encoding and decoding device 144; if a task is to perform JPEG encoding and decoding, the guest operating system will access the configured virtual JPEG encoding and decoding device 146 through the configured virtual interface 140 and virtual function 128; if a certain task is to read Or write data, the guest operating system will access the configured virtual storage device 148 through the configured virtual interface 140 and the virtual function 128.
  • FIG. 2 shows a schematic diagram of the internal structure of a computing device 130.
  • the computing device 130 has a total of sixteen processing unit cores (processing unit core 0 to processing unit core 15) for performing matrix calculation tasks, and every four processing unit cores form a processing unit group, that is, a cluster.
  • processing unit core 0 to processing unit core 3 constitute a first cluster 202
  • processing unit core 4 to processing unit core 7 constitute a second cluster 204
  • processing unit core 8 to processing unit core 11 constitute a third cluster 206.
  • the processing unit core 12 to the processing unit core 15 constitute a fourth cluster 208.
  • the computing device 130 basically uses a cluster as a unit to perform computing tasks.
  • the computing device 130 also includes a storage unit core 210 and a shared storage unit 212.
  • the storage unit core 210 is mainly used to control data exchange and serves as a communication channel between the computing device 130 and the storage device 136.
  • the shared storage unit 212 is used to temporarily store the calculated intermediate values of these clusters 202, 204, 206, and 208.
  • the storage unit core 210 is divided into four virtual storage unit cores, and the shared storage unit 212 is also divided into four virtual shared storage units.
  • Each virtual computing device 142 is configured with a virtual storage unit core, a virtual shared storage unit, and a cluster to support the tasks of a specific guest operating system. Similarly, each virtual computing device 142 operates independently and does not affect each other.
  • the number of clusters of the computing device 130 should be at least the same as the number of virtual computing devices 142 to ensure that each virtual computing device 142 can be configured with one cluster.
  • the cluster is appropriately configured to the virtual computing device 142 to increase the flexibility of hardware configuration.
  • the video codec device 132 of this embodiment includes six video codec units.
  • the video codec device 132 can flexibly deploy the video codec unit as a unit according to the number of virtual components and required resources.
  • the video codec device 132 is virtualized into four virtual video codec devices 144. Assuming that the first virtual video codec device and the second virtual video codec device require more video codec resources, two of them can be configured respectively.
  • the video codec unit is provided to the first virtual video codec device and the second virtual video codec device, and one video codec unit is respectively allocated to the other virtual video codec devices 144.
  • the video codec device 132 is virtualized into three virtual video codec devices 144. If any virtual video codec device does not require more video codec resources, two video codecs can be configured respectively. Unit for each virtual video codec device 144.
  • the number of video codec units should be at least the same as the number of virtual video codec devices 144 to ensure that each virtual video codec device 144 can be configured with one video codec unit.
  • the number of video codec units is more than that of virtual video codec devices
  • the video codec unit can be appropriately configured to the virtual video codec device 144 according to actual needs, so as to increase the flexibility of hardware configuration.
  • the JPEG codec device 134 of this embodiment includes six JPEG codec units.
  • the JPEG encoding and decoding device 134 can flexibly deploy the JPEG encoding and decoding unit as a unit according to the number of virtual components and the required resources.
  • the deployment method is the same as that of the video encoding and decoding device 132, so it will not be repeated.
  • the storage device 136 can adopt a non-uniform memory access (NUMA) architecture and include multiple DDR channels.
  • NUMA non-uniform memory access
  • the storage device 136 can flexibly take the DDR channel as the unit according to the number of virtual components and the required resources.
  • the deployment method is the same as that of the computing device 130, the video codec device 132, and the JPEG codec device 134, so it will not be repeated here.
  • the hot migration solution of the present disclosure is divided into two stages: the first stage is to package and send the configuration and data on the source server to the destination server, that is, the migration save path; the second stage is to combine these groups The state and data are placed in the corresponding place of the destination server, that is, the migration recovery path.
  • This hot migration solution is to completely save the running state of the entire virtual machine and quickly restore it to the original hardware platform or even different hardware platforms.
  • the source server and the destination server can be on the same hardware platform or on different hardware platforms. After recovery, the virtual machine still runs smoothly, and the user will not notice any difference.
  • the aforementioned two stages will be explained separately.
  • Figure 3 shows an embodiment of the present disclosure, which is a flow chart of the migration save path.
  • the source server of this embodiment may be the system disclosed in Figure 1, and Figure 4 shows the source server with the architecture of Figure 1 performing Schematic diagram of the migration save path.
  • the driver, firmware, hardware information, context information and its status information related to the specific virtual hardware on the system-on-chip 106 are packaged and sent from the source server.
  • the "status information" "Can include virtual function driver status information, firmware and hardware status information, state machine, register, hardware internal state context state information, software state machine, variables, constant runtime context state information, etc.
  • the virtualization management software initiates a migration request to the simulated virtual machine QEMU 402.
  • the virtualization management software of this embodiment is Libvirt 401.
  • Libvirt 401 is an open source application programming interface (API), background program, and management tool used to manage the virtualization platform. It can be used to manage the virtualization technology of QEMU 402 .
  • API application programming interface
  • Libvirt 401 can start hot migration to ensure the normal operation of virtual machine services.
  • step 302 QEMU 402 notifies the physical function driver 403 to start migration, that is, QEMU 402 initiates a hot migration start request.
  • This embodiment provides a model to manage the entire migration saving path process.
  • the model is a virtual machine learning unit QEMU object model (VMLU QOM), where the virtual machine learning unit refers to the virtualized artificial intelligence on-chip shown in FIG. 1 System, and the QEMU object model simulates the relationship between the processor of a specific architecture and the device and the bus.
  • VMLU QOM virtual machine learning unit QEMU object model
  • VMLU QOM 404 simulates a processor model based on the characteristics of the artificial intelligence system-on-chip in Figure 1. In terms of the relationship between the simulation device and the bus, since the system-on-chip 106 communicates with the kernel space 104 through PCIe 138, the relationship between the device connection to the bus and the bus connection device is simulated by VMLU QOM 404.
  • Memory mapping I/O Memory mapping I/O, MMIO
  • I/O devices are placed in memory space instead of I/O space. From the perspective of the processor in the user space 102, the system access to other devices after the memory mapped I/O is the same as the access to the memory, which simplifies the difficulty of programming and the complexity of the interface.
  • VMLU QOM 404 in QEMU 402 initializes the hot migration start request, and sends the hot migration start request to the physical function driver 403.
  • the physical function driver 403 notifies the virtual function driver 405 to start migration.
  • the virtual function driver 405 is stored in the virtual machine kernel space.
  • the hardware of the system-on-chip 106 looks like a piece of memory in the virtual function driver 405, and the virtual function driver 405 maps the memory to the memory.
  • the read and write operations in the /O space (that is, the system on chip 106) will be captured and managed by VMLU QOM 404.
  • the VMLU QOM 404 can return a value that should be returned according to the needs of the virtual function driver 405, so that the virtual function driver 405 and the physical function driver 403 are synchronized.
  • the VMLU QOM 404 obtains the migration status of the physical function driver 403 by calling the interface of the physical function driver 403.
  • VMLU QOM 404 returns the status of the physical function 406 to the virtual function driver 405.
  • VMLU QOM 404 changes the physical function
  • the state of the driver 403 ready for hot migration is transferred to the virtual function driver 405.
  • step 304 the virtual function driver 405 suspends execution of tasks from the processor in the user space 102.
  • the method adopted in this embodiment is that the virtual function driver 405 does not return the control command to the guest operating system of the user space 102. The guest operating system will continue to wait without receiving the control command, and will not issue the next task to the virtual function.
  • the driver 405 is used to suspend the execution of tasks on the user space side.
  • step 305 the virtual function driver 405 notifies the physical function driver 403 to prepare for migration. After the execution of the instruction from the user space 102 is suspended, the virtual function driver 405 notifies the physical function driver 403 that the user space 102 is ready, and there will be no interference in issuing instructions during the hot migration.
  • the physical function driver 403 notifies the physical function 406 to prepare for migration.
  • the physical function driver 403 sends a hot migration start request to the physical function 406.
  • the hot migration start request specifies the specific virtual hardware 408 for hot migration.
  • the specific virtual hardware 408 is one of the multiple virtual hardware of the system-on-chip 106, that is, The hot migration start request records the hot migration for the specific virtual hardware 408.
  • the hot migration start request is for a specific virtual function 407 and its corresponding specific virtual hardware 408.
  • the specific virtual hardware 408 may be a specific virtual computing device, such as the virtual computing device 142 in FIG. 1, and the data to be migrated includes calculated intermediate values stored in the virtual shared storage unit and data stored in the virtual storage unit core.
  • the specific virtual hardware 408 may also be the specific virtual storage device 148 in FIG. 1, and the data to be migrated includes data stored in the specific virtual storage device 148.
  • the specific virtual hardware 408 may also be a virtual video codec device 144 or a virtual JPEG codec device 146, and the data to be migrated includes corresponding codec information.
  • the physical function 406 uploads to the physical function driver 403 data including the driver of the specific virtual function 407, firmware, and information of the specific virtual hardware 408, context information and its status information.
  • the physical function 406 sends the instruction to be migrated to the physical function driver 403 of the kernel space 104.
  • the instruction to be migrated records the data to be migrated related to the specific virtual hardware 408, so that the physical function driver 403 knows how much data needs to be migrated. .
  • the physical function 406 responds to the hot migration start request and idles the specific virtual function 407 so that it does not communicate with the user space 102 temporarily, but other virtual functions and their corresponding virtual hardware operate as usual.
  • the physical function 406 takes out the data to be migrated from the specific virtual hardware 408 in batches, and sends it to the physical function driver 403. After the data to be migrated is sent, the physical function 406 sends an end signal to the physical function driver 403.
  • the VMLU QOM 404 obtains the data to be migrated from the physical function driver 403.
  • the physical function driver 403 in the kernel space first sends the instructions to be migrated to the VMLU QOM 404 in the user virtual machine, and then sends the data to be migrated to the VMLU QOM 404.
  • VMLU QOM 404 After VMLU QOM 404 receives the instruction to be migrated, it calls the read function to obtain the data to be migrated from the physical function driver 403.
  • step 309 VMLU QOM 404 transmits the data to be migrated to Libvirt 401.
  • step 310 when the data to be migrated is sent, the physical function 406 releases the resources of the specific virtual hardware 408 and the specific virtual function 407, the VMLU QOM 404 sends an end signal to the virtual function driver 405, and the virtual function driver 405 sends a control signal To the interface 409 of the virtual function driver, the guest operating system continues to issue tasks.
  • this embodiment implements the migration saving path. While the foregoing process is executed, for non-specific virtual functions and hardware, tasks from the user space 102 are still executed without being affected.
  • FIG. 5 is a flowchart showing the migration recovery path
  • FIG. 6 shows a schematic diagram of the migration recovery path in the environment of FIG. 1.
  • this embodiment is to migrate the data to be migrated to the destination server after the migration save path is completed in the embodiment of FIG. 3 and FIG. 4.
  • Libvirt 601 initiates a request to import the data to be migrated to QEMU 602.
  • QEMU 602 receives the data to be migrated sent in the previous embodiment from off-chip, and initiates a hot migration start request.
  • the off-chip refers to the source server, and the source server and the destination server may be on the same hardware platform or on different hardware platforms.
  • step 502 the VMLU QOM 604 sends the data to be migrated to the physical function driver 603.
  • VMLU QOM 604 After VMLU QOM 604 receives the data to be migrated, it responds to the hot migration start request, calls the write function, and sends the data to be migrated to the physical function driver 603.
  • step 503 the physical function 606 receives the data to be migrated.
  • VMLU QOM 604 sends the data to be migrated to the physical function driver 603, and the physical function driver 603 sends the data to be migrated to the physical function 606.
  • step 504 the configuration, data and its context are restored for the specific virtual function 607 and the specific virtual hardware 608.
  • the physical function 606 idles the specific virtual function 607 so that it does not communicate with the user space 102 temporarily, but other virtual functions operate as usual. After the specific virtual function 607 is idle, the physical function 606 sends the data to be migrated to the specific virtual hardware 608 through the specific virtual function 607.
  • the specific virtual hardware 608 may be the virtual computing device, the specific virtual storage device, the virtual video codec device, or the virtual JPEG codec device of FIG. 1.
  • the data to be migrated includes drivers, firmware and hardware information, context information, and status information related to the specific virtual hardware 608.
  • the specific virtual function 607 and the specific virtual hardware 608 and the specific virtual function 407 and the specific virtual hardware 408 have exactly the same environment and data.
  • step 505 the physical function 606 reports to the physical function driver 603 that the migration has been completed. After the data to be migrated is sent, the physical function 606 sends an end signal to the physical function driver 603 of the kernel space 104.
  • step 506 the physical function driver 603 notifies VMLU QOM 604 that the hot migration has been completed, that is, the physical function driver 603 sends an end signal to QEMU 602.
  • step 507 the VMLU QOM 604 changes state to notify the virtual function driver 605 that the hot migration has been completed.
  • VMLU QOM 604 responds to the end signal to notify the virtual function driver 605 that the hot migration has been completed, and at the same time changes the state of the base address register to point to the specific virtual function 607 and specific virtual hardware 608 of the destination server.
  • step 508 the virtual function driver 605 sends a control signal to the interface 609 of the virtual function driver to re-execute the tasks of the guest operating system.
  • step 509 the interface 609 of the virtual function driver notifies the virtual function driver 605 to re-execute the tasks of the guest operating system.
  • the virtual function driver 605 again receives tasks from the processor in the user space 102 through the interface 609 of the virtual function driver. These tasks no longer access the specific virtual hardware 408 of the source server, but instead access the specific virtual hardware 608 of the destination server.
  • step 510 VMLU QOM 604 notifies Libvirt 601 that the hot migration is completed, and Libvirt 601 cleans up the hardware resources allocated on the source server. This completes the migration recovery path.
  • the present disclosure can realize the hot migration of the virtualized application specific integrated circuit.
  • FIG. 7 is a structural diagram of an integrated circuit device 700 according to an embodiment of the present disclosure.
  • the integrated circuit device 700 is the artificial intelligence system-on-chip in the foregoing multiple embodiments, and includes a computing device 702, which may be the computing device 130 in FIG.
  • the integrated circuit device 700 further includes a universal interconnection interface 704 and other processing devices 706.
  • the other processing device 706 may be one or more types of general-purpose and/or special-purpose processors such as a central processing unit, a graphics processor, and an artificial intelligence processor. The number of processors is not limited but determined according to actual needs. .
  • the other processing device 706 serves as an interface between the computing device 702 and external data and control, performs basic control including but not limited to data transfer, and completes the starting and stopping of the computing device 702. The other processing device 706 can also cooperate with the computing device 702 to complete computing tasks.
  • the universal interconnect interface 704 can be used to transfer data and control commands between the computing device 702 and other processing devices 706.
  • the computing device 702 may obtain required input data from other processing devices 706 via the universal interconnect interface 704, and write the input data to the on-chip storage unit of the computing device 702.
  • the computing device 702 can obtain control instructions from other processing devices 706 via the universal interconnect interface 704, and write them into the on-chip control buffer of the computing device 702.
  • the universal interconnection interface 704 can also read the data in the storage module of the computing device 702 and transmit it to other processing devices 706.
  • the integrated circuit device 700 further includes a storage device 708, which can be connected to the computing device 702 and other processing devices 706, respectively.
  • the storage device 708 is used to store data of the computing device 702 and other processing devices 706, and is particularly suitable for data that cannot be fully stored in the internal storage of the computing device 702 or other processing devices 706 for the data required for calculation.
  • the integrated circuit device 700 can be used as a system-on-chip (SOC) for mobile phones, robots, drones, video capture and other equipment, thereby effectively reducing the core area of the control part, increasing processing speed and reducing overall power consumption .
  • the universal interconnection interface 704 of the integrated circuit device 700 is connected to certain components of the device. Some components here can be, for example, a camera, a monitor, a mouse, a keyboard, a network card or a wifi interface.
  • This disclosure also discloses a chip or integrated circuit chip, which includes the integrated circuit device 700.
  • the present disclosure also discloses a chip packaging structure, which includes the above-mentioned chip.
  • the board card 800 may also include other supporting components.
  • the supporting components include a storage device 804, an interface device 806 and a control device 808.
  • the storage device 804 is connected to the chip 802 in the chip package structure through a bus 814 for storing data.
  • the storage device 804 may include multiple sets of storage units 810.
  • the interface device 806 is electrically connected to the chip 802 in the chip packaging structure.
  • the interface device 806 is used to implement data transmission between the chip 802 and an external device 812 (for example, a server or a computer).
  • the interface device 806 is a standard PCIe interface, and the data to be processed is transferred from the server to the chip 802 through the standard PCIe interface to realize data transfer.
  • the calculation result of the chip 802 is also transmitted back to the external device 812 by the interface device 806.
  • the control device 808 is electrically connected to the chip 802 to monitor the state of the chip 802. Specifically, the chip 802 and the control device 808 may be electrically connected through an SPI interface.
  • the control device 808 may include a single-chip microcomputer ("MCU", Micro Controller Unit).
  • electronic equipment or devices can include data processing devices, robots, computers, printers, scanners, tablets, smart terminals, mobile phones, driving recorders, navigators, sensors, cameras, servers, cloud servers, and cameras , Cameras, projectors, watches, earphones, mobile storage, wearable devices, vehicles, household appliances, and/or medical equipment.
  • the transportation means include airplanes, ships, and/or vehicles; the household appliances include TVs, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, and range hoods; the medical equipment includes nuclear magnetic resonance, B-ultrasound and/or electrocardiograph.
  • Another embodiment of the present disclosure is a computer-readable storage medium on which is stored hot-migrated computer program code.
  • the computer program code is run by a processor, the method described in FIG. 3 or FIG. 5 is executed.
  • This disclosure can realize the hot migration of the virtual hardware driver, firmware and hardware information, context information and its status information in the source server to the destination server, especially the virtual computing device and virtual video codec device of the hot migration application specific integrated circuit. , Virtual JPEG codec device, etc.
  • a system for realizing hot migration connected to the kernel space, including: hardware, virtualized into multiple virtual hardware; and read-only memory, storing firmware, the firmware including: physical functions, used to: The hot migration start request of the kernel space, the hot migration start request specifies specific virtual hardware, and the specific virtual hardware is one of the multiple virtual hardware; sending a migration instruction to the kernel space, and the migration Instruct to record the data to be migrated, the data to be migrated is associated with the specific virtual hardware; the data to be migrated are taken out from the specific virtual hardware in batches and sent to the kernel space; and when the data to be migrated is sent After that, an end signal is sent to the kernel space.
  • the system according to A1 wherein the hardware includes a computing device, the specific virtual hardware is a specific virtual computing device, and the computing device includes: an intelligent processing device for performing convolution calculations of a neural network; and
  • the shared storage unit is used to temporarily store the calculated intermediate value of the convolution calculation, the shared storage unit is virtualized into a plurality of virtual shared storage units, and the specific virtual computing device corresponds to one virtual shared storage unit; wherein, the The data to be migrated includes the calculated intermediate value stored in the virtual shared storage unit.
  • the computing device further includes: a storage unit core, virtualized into a plurality of virtual storage unit cores, and the specific virtual computing device is configured with a virtual storage unit core; wherein, the waiting The migration data includes data stored in the virtual storage unit core.
  • Clause A4 The system according to A1, wherein the hardware includes a storage device, virtualized into multiple virtual storage devices, the specific virtual hardware is a specific virtual storage device, and the data to be migrated includes storage in the specific virtual storage device. Storage device data.
  • Clause A5. The system according to A1, wherein the hardware includes a video codec device, and the specific virtual hardware is a specific virtual video codec device.
  • firmware further includes a plurality of virtual functions managed by the physical function, the specific virtual hardware corresponds to a virtual function, and the physical function responds to the hot migration start request, Idle the virtual function.
  • Clause A9 The system according to A1, wherein the data to be migrated includes one of driver, firmware and hardware information, context information, and status information.
  • the kernel space carries a physical function driver
  • the user space includes a user-side kernel space
  • the user-side kernel space carries a virtual function driver
  • the physical function driver The program receives the hot migration start request from the user virtual machine and sends it to the virtual function driver, and the virtual function driver suspends execution of tasks from the user space.
  • a system for realizing hot migration connected to the kernel space, including: hardware, virtualized into multiple virtual hardware; and read-only memory, storing firmware, the firmware including: physical functions to receive data to be migrated , The data to be migrated is associated with specific virtual hardware, the specific virtual hardware is one of the multiple virtual hardware; and the multiple virtual functions are managed by the physical function, and the specific virtual hardware corresponds to a virtual function
  • the physical function sends the data to be migrated to the specific virtual hardware through the virtual function; when the data to be migrated is sent, the physical function sends an end signal to the kernel space.
  • Clause A18 The system according to A17, wherein the hardware includes a computing device, the specific virtual hardware is a specific virtual computing device, and the computing device includes: an intelligent processing device for performing convolution calculations of a neural network; and
  • the shared storage unit is used to temporarily store the calculated intermediate value of the convolution calculation, the shared storage unit is virtualized into a plurality of virtual shared storage units, and the specific virtual computing device corresponds to one virtual shared storage unit; wherein, the The virtual function stores the calculated intermediate value in the data to be migrated to the virtual shared storage unit.
  • the computing device further includes: a storage unit core, virtualized into multiple virtual storage unit cores, and the specific virtual computing device is configured with one virtual storage unit core; wherein the virtual The function stores the corresponding data in the data to be migrated to the virtual storage unit core.
  • Clause A20 The system according to A17, wherein the hardware includes a storage device, virtualized into a plurality of virtual storage devices, the specific virtual hardware is a specific virtual storage device, and the virtual function converts the data in the data to be migrated The corresponding data is stored in the virtual storage device.
  • Clause A28 The system according to A27, wherein the user space includes a user-side kernel space, the user-side kernel space carries a virtual function driver, and the user virtual machine informs the virtual function in response to the end signal
  • the virtual function driver receives a task from the user space, and the task controls the specific virtual hardware.
  • a method for saving a path for hot migration of a system comprising: receiving a hot migration start request, the hot migration start request specifying specific virtual hardware
  • the specific virtual hardware is one of the multiple virtual hardware; the specific virtual function is idle, and the specific virtual function corresponds to the specific virtual hardware; the data to be migrated is sent in batches from the specific virtual hardware; and when After the data to be migrated is sent, an end signal is sent.
  • a method for performing a hot migration recovery path for a system including hardware, virtualized into multiple virtual hardware, the method comprising: receiving data to be migrated, the hot migration data corresponding to specific virtual hardware,
  • the specific virtual hardware is one of the multiple virtual hardware;
  • the specific virtual function is idle, and the specific virtual function corresponds to the specific virtual hardware;
  • the data to be migrated is sent to the specific virtual hardware through the specific virtual function ;
  • an end signal is sent.
  • Clause A35 The method according to A34, wherein the hardware is one of a computing device, a storage device, a video codec device, and a JPEG codec device of an artificial intelligence system-on-chip.
  • Clause A36 A computer-readable storage medium on which is stored the computer program code for hot migration. When the computer program code is run by a processor, the method described in any one of clauses A 32-35 is executed.

Abstract

提供一种实现热迁移的方法、计算装置及可读存储介质，计算装置包括在集成电路装置中，集成电路装置包括通用互联接口和其他处理装置。计算装置与其他处理装置进行交互，共同完成用户指定的计算操作。集成电路装置还可以包括存储装置，存储装置分别与计算装置和其他处理装置连接，用于计算装置和其他处理装置的数据存储。

Description

实现热迁移的方法、芯片、板卡和存储介质
相关申请的交叉引用
本申请要求于2020年5月7日申请的，申请号为202010379648X，名称为“实现热迁移的方法、芯片、板卡和存储介质”的中国专利申请的优先权，在此将其全文引入作为参考。
技术领域
本披露一般地涉及计算机领域。更具体地，本披露涉及一种实现热迁移的方法、芯片、板卡及计算机可读存储介质。
背景技术
热迁移(Live Migration)又称为动态迁移、实时迁移，即虚拟机通过保存(SAVE)/恢复(LOAD)的程序，将整个虚拟机的运行状态完整保存下来，从一台物理服务器迁移到另一台物理服务器上。恢复以后，虚拟机仍旧平滑运行，用户不会察觉到任何差异。
在人工智能领域中，由于专用集成电路(ASIC)的复杂度高，以至于无法完全实现热迁移。随着云计算的发达，在云端采用人工智能芯片的服务器均需要利用虚拟化，以保障用户的独立运算空间，而热迁移可以提供稳定的服务，因此如何在人工智能芯片上实现热迁移是现有技术中需要解决的问题。
发明内容
为了至少部分地解决背景技术中提到的技术问题，本披露的方案提供了一种实现热迁移的方法、芯片、板卡及计算机可读存储介质。
根据本披露的第一方面，提供一种实现热迁移的系统，连接至内核空间，所述系统包括硬件及只读存储器。硬件虚拟化为多个虚拟硬件，只读存储器存储有固件，该固件包括物理功能，用以：接收来自所述内核空间的热迁移启动请求，所述热迁移启动请求指定特定虚拟硬件，所述特定虚拟硬件为所述多个虚拟硬件其中之一；发送待迁移指令至所述内核空间，所述待迁移指令记录待迁移数据，所述待迁移数据关联至所述特定虚拟硬件；自所述特定虚拟硬件分批取出所述待迁移数据并发送至所述内核空间；以及当所述待迁移数据发送完毕后，发送结束信号至所述内核空间。
根据本披露的第二方面，提供一种实现热迁移的系统，连接至内核空间，所述系统包括硬件及只读存储器。硬件虚拟化为多个虚拟硬件。只读存储器存储有固件，该固件包括：物理功能及多个虚拟功能。物理功能用以接收待迁移数据，所述待迁移数据关联至特定虚拟硬件，所述特定虚拟硬件为所述多个虚拟硬件其中之一，多个虚拟功能受所述物理功能管理，所述特定虚拟硬件对应一个虚拟功能，物理功能通过所述虚拟功能发送所述待迁移数据至所述特定虚拟硬件。当所述待迁移数据发送完毕后，所述物理功能发送结束信号至所述内核空间。
根据本披露的第三方面，提供一种集成电路装置，包括前述的系统，亦提供一种板卡，包括前述的集成电路装置。
根据本披露的第四方面，提供一种对系统进行热迁移保存路径的方法，所述系统包括硬件，虚拟化为多个虚拟硬件。所述方法包括：接收热迁移启动请求，所述热迁移启动请求指定特定虚拟硬件，所述特定虚拟硬件为所述多个虚拟硬件其中之一；闲置特定虚拟功能，所述特定虚拟功能对应至所述特定虚拟硬件；自所述特定虚拟硬件分批发送待迁移数据；以及当所述待迁移数据发送完毕后，发送结束信号。
根据本披露的第五方面，提供一种对系统进行热迁移恢复路径的方法，所述系统包括硬件，虚拟化为多个虚拟硬件。所述方法包括：接收待迁移数据，所述热迁移数据对应至特定虚拟硬件，所述特定虚拟硬件为所述多个虚拟硬件其中之一；闲置特定虚拟功能，所述特定虚拟功能对应至所述特定虚拟硬件；通过特定虚拟功能发送所述待迁移数据至所述特定虚拟硬件；以及当所述待迁移数据发送完毕后，发送结束信号。
根据本披露的第六方面，提供一种计算机可读存储介质，其上存储有热迁移的计算机程序代码，当所述计算机程序代码由处理器运行时，执行前述的方法。
本披露可以实现将虚拟化设备的信息从源服务器热迁移至目的服务器上。
附图说明
通过参考附图阅读下文的详细描述，本披露示例性实施方式的说明以及其他目的、特征和优点将变得易于理解。在附图中，以示例性而非限制性的方式示出了本披露的若干实施方式，并且相同或对应的标号表示相同或对应的部分，其中：
图1是示出本披露实施例的人工智能芯片框架的示意图；
图2是示出本披露实施例的计算装置的内部结构示意图；
图3是本披露实施例的迁移保存路径的流程图；
图4是示出本披露实施例的源服务器端进行迁移保存路径的示意图；
图5是本披露实施例的迁移恢复路径的流程图；
图6是示出本披露实施例的目的服务器端进行迁移恢复路径的示意图；
图7是示出本披露实施例的集成电路装置的结构图；以及
图8是示出本披露实施例的板卡的示意图。
具体实施方式
下面将结合本披露实施例中的附图，对本披露实施例中的技术方案进行清楚、完整地描述。所描述的实施例是本披露一部分实施例，而不是全部的实施例。基于本披露中的实施例，本领域技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本披露保护的范围。
应当理解，本披露的权利要求、说明书及附图中的术语“第一”、“第二”、“第三”和“第四”等是用于区别不同对象，而不是用于描述特定顺序。本披露的说明书和权利要求书中使用的术语“包括”和“包含”指示所描述特征、整体、步骤、操作、元素和/或组件的存在，但并不排除一个或多个其它特征、整体、步骤、操作、元素、组件和/或其集合的存在或添加。
还应当理解，在此本披露说明书中所使用的术语仅仅是出于描述特定实施例的目的，而并不意在限定本披露。如在本披露说明书和权利要求书中所使用的那样，除非上下文清楚地指明其它情况，否则单数形式的“一”、“一个”及“该”意在包括复数形式。还应当进一步理解，在本披露说明书和权利要求书中使用的术语“和/或”是 指相关联列出的项中的一个或多个的任何组合以及所有可能组合，并且包括这些组合。
如在本说明书和权利要求书中所使用的那样，术语“如果”可以依据上下文被解释为“当...时”或“一旦”或“响应于确定”或“响应于检测到”。
下面结合附图来详细描述本披露的具体实施方式。
本披露涉及一种采用虚拟化技术的框架，应用在专用集成电路上，例如用于神经网络的机器学习设备上，该机器学习设备可以是卷积神经网络加速器。以下将以人工智能芯片示例说明。
图1是人工智能芯片虚拟化的框架图，该框架100包括用户空间102、内核空间104及片上系统106，在图上以虚线区隔开。用户空间102为用户程序的运行空间，只执行简单的运算，不能直接调用系统资源，必须通过系统接口，才能向内核发出指令。内核空间104是内核代码运行的空间，可以执行任意命令，调用系统的一切资源。片上系统106为人工智能芯片的各模块，通过内核空间104与用户空间102进行协作。
在此实施例中，用户空间102的硬件统一称之为器或设备，片上系统106的硬件统一称之为装置或单元，作为区分。这样的安排只为了更清楚的描述此实施例的技术，并非对本披露的技术做任何的限制。
除非另行强调，此实施例以将一个部件虚拟化为四个虚拟部件来示例说明，但本披露不限制虚拟部件的数量。
用户空间102在未运行虚拟化前，是由硬件监测器工具108所控制，通过调用接口获取片上系统106的信息。硬件监测器工具108不仅可以采集片上系统106的信息，还可以实时获取上层软件对片上系统106资源的开销，为用户实时掌握当前片上系统106的详细信息和状态，这些详细信息和状态可以是：硬件设备型号、固件版本号、驱动版本号、设备利用率、存储装置开销状态、板卡功耗和板卡峰值功耗、快速外设组件互连(PCIe)等数十种数据。基于硬件监测器工具108的版本及使用场景的不同，所监测的信息内容及数量会有所差异。
在系统启动虚拟化后，用户空间102的操作改由用户虚拟机110接管，用户虚拟机110是对真实计算环境的抽象和模拟，系统会分配一套数据结构来管理用户虚拟机110的状态，其数据结构包括全套寄存器、物理内存的使用情况、虚拟设备的状态等等。此实施例的用户空间102的物理空间虚拟化为四个虚拟空间112、114、116、118，这四个虚拟空间112、114、116、118独立互不影响，可分别搭载不同的客户操作系统，如图中所示的客户操作系统1、客户操作系统2、客户操作系统3及客户操作系统4，客户操作系统可以是Windows、Linux、Unix、iOS、安卓等，每个客户操作系统上分别运行不同的应用程序。
在本披露的环境里，用户虚拟机110是以快速仿真器(QEMU)来实现。QEMU是一个用C语言编写的开源虚拟化软件，通过动态二进制转换将接口虚拟化，并提供一系列的硬件模型，使得客户操作系统1、客户操作系统2、客户操作系统3、客户操作系统4都认为自己直接访问片上系统106。用户空间102包括处理器、存储器、I/O设备等，QEMU可以将用户空间102的处理器虚拟化为四个虚拟处理器，并将存储器虚拟化为四个虚拟存储器，亦将I/O设备的虚拟化为四个虚拟I/O设备。每个客户操作系统各占用一部分用户空间102的资源，例如各占四分之一，也就是分别能访问一个虚拟处理器、一个虚拟存储器及一个虚拟I/O设备，以执行该客户操作系统的任务。 通过这种模式，客户操作系统1、客户操作系统2、客户操作系统3、客户操作系统4就能独立运作。
内核空间104载有内核虚拟机120及芯片驱动程序122。内核虚拟机120与QEMU搭配，主要负责内核空间104及片上系统106的虚拟化，使得每个客户操作系统在访问片上系统106时都能获得自己的地址空间。更详细来说，映射给客户操作系统的片上系统106上的空间实际上是映射给这个进程的虚拟部件。
内核虚拟机120包括物理功能驱动程序，是一个专门管理SR-IOV设备全局功能的驱动程序，一般需要具有比普通虚拟机更高的权限才能对其进行操作。物理功能驱动程序包含了所有传统驱动程序的功能，用户空间102能够访问片上系统106的I/O资源。
从用户虚拟机110的角度来看，在虚拟机运行期间，QEMU通过内核虚拟机120提供的系统调用接口进行内核设置，QEMU使用了内核虚拟机120的虚拟化功能，为自己的虚拟机提供硬件虚拟化加速以提高虚拟机的性能。从内核虚拟机120的角度来看，当用户无法直接跟内核空间104交互时，需要借助用户空间102的管理工具，因此需要借助QEMU这个运行在用户空间102的工具。
芯片驱动程序122用以驱动物理功能(physical function，PF)126，在虚拟机运行期间，用户空间102不由硬件监测器工具108经芯片驱动程序122来访问片上系统106，因此客户操作系统1、客户操作系统2、客户操作系统3、客户操作系统4分别配置有用户端内核空间124，用以载入芯片驱动程序122，使得各客户操作系统依然可以驱动片上系统106。
片上系统106是通过单根I/O虚拟化(single root I/O virtualization，SR-IOV)技术来执行虚拟化的，更详细来说，在本披露的环境里，SR-IOV技术是以软硬件结合实现的，使得片上系统106的各部件虚拟化。SR-IOV技术是一种虚拟化解决方案，允许在虚拟机之间高效共享PCIe资源，SR-IOV技术使得单个PCIe资源可由片上系统106的多个虚拟部件共享，为这些虚拟部件提供专用的资源。这样，每个虚拟部件都有自己对应的唯一可访问的资源。
此实施例的片上系统106包含硬件和固件。硬件包括只读存储器ROM(未显示于图中)，用以存储固件，而固件包括物理功能126，用于支持或协作SR-IOV的PCIe功能，物理功能126拥有完全配置PCIe资源的权力。在实施SR-IOV技术时，物理功能126会虚拟化出多个虚拟功能(virtual function，VF)128，在此实施例中为四个虚拟功能128。虚拟功能128是一种轻量级PCIe功能，受物理功能126管理，可与物理功能126以及与同一物理功能126关联的其他虚拟功能128共享PCIe物理资源。虚拟功能128仅允许控制物理功能126配置给自己的资源。
一旦在物理功能126中启用了SR-IOV，各个虚拟功能128就可以通过自身的总线、设备和功能编号去访问自己的PCIe配置空间。每个虚拟功能128都具有一个内存空间，用于映射其寄存器集。虚拟功能128驱动程序对寄存器集进行操作以启用其功能，并直接指定给对应的用户虚拟机110。虽然是虚拟的，但会让用户虚拟机110认为是实际存在的PCIe设备。
片上系统106的硬件还包括计算装置130、视频编解码装置132、JPEG编解码装置134、存储装置136及PCIe 138。在此实施例中，计算装置130为智能处理装置 (intelligent processing unit，IPU)，用以执行神经网络的卷积计算；视频编解码装置132用以对视频数据进行编解码；JPEG编解码装置134用以对采用JPEG算法的静态图片进行编解码；存储装置136可以为动态随机存取存储器(DRAM)，用以存储数据；PCIe 138即为前述的PCIe，在虚拟机运行期间，PCIe 138会虚拟化为四个虚拟接口140，虚拟功能128与虚拟接口140为一对一对应关系，也就是第一虚拟功能对接第一虚拟接口，第二虚拟功能对接第二虚拟接口，以此类推。
通过SR-IOV技术，计算装置130虚拟化为四个虚拟计算装置142、将视频编解码装置132虚拟化为四个虚拟视频编解码装置144、将JPEG编解码装置134虚拟化为四个虚拟JPEG编解码装置146、将存储装置136虚拟化为四个虚拟存储装置148。
每个客户操作系统分别配置一组虚拟套件，每组虚拟套件包括一个用户虚拟机110、一个虚拟接口140、一个虚拟功能128、一个虚拟计算装置142、一个虚拟视频编解码装置144、一个虚拟JPEG编解码装置146及一个虚拟存储装置148。每组虚拟套件各自独立运行互不影响，用来执行相对应的客户操作系统所交付的任务，以确定每个客户操作系统能通过所配置的虚拟接口140及虚拟功能128访问所配置的虚拟计算装置142、虚拟视频编解码装置144、虚拟JPEG编解码装置146及虚拟存储装置148。
更详细来说，每个客户操作系统在执行任务时，响应任务的不同，所需访问的硬件可能也不同，例如：某个任务是进行矩阵卷积计算，则该客户操作系统会通过所配置的虚拟接口140及虚拟功能128访问所配置的虚拟计算装置142；如某个任务是进行视频编解码，则该客户操作系统会通过所配置的虚拟接口140及虚拟功能128访问所配置的虚拟视频编解码装置144；如某个任务是进行JPEG编解码，则该客户操作系统会通过所配置的虚拟接口140及虚拟功能128访问所配置的虚拟JPEG编解码装置146；如某个任务是读取或写入数据，则该客户操作系统会通过所配置的虚拟接口140及虚拟功能128访问所配置的虚拟存储装置148。
图2示出了一种计算装置130的内部结构示意图。计算装置130共有十六个处理单元核(处理单元核0至处理单元核15)，用于执行矩阵计算任务，每四个处理单元核组成一个处理单元组，也就是集群(cluster)。更详细来说，处理单元核0至处理单元核3组成第一集群202，处理单元核4至处理单元核7组成第二集群204，处理单元核8至处理单元核11组成第三集群206，处理单元核12至处理单元核15组成第四集群208。计算装置130基本上是以集群为单元执行计算任务。
计算装置130还包括存储单元核210及共享存储单元212。存储单元核210主要用于控制数据交换，作为计算装置130与存储装置136沟通的渠道。共享存储单元212用以暂存这些集群202、204、206、208的计算中间值。在虚拟化运行期间，存储单元核210会切分为四个虚拟存储单元核，共享存储单元212也同样会切分为四个虚拟共享存储单元。
每个虚拟计算装置142分别配置一个虚拟存储单元核、一个虚拟共享存储单元以及一个集群，以支撑特定客户操作系统的任务。同样地，每个虚拟计算装置142各自独立运作，互不影响。
计算装置130的集群数量至少应与虚拟计算装置142的数量相同，以确保每个虚拟计算装置142能配置一个集群，当集群数量多于虚拟计算装置142的数量时，便可 以根据实际需求，将集群适当地配置给虚拟计算装置142，以增加硬件配置弹性。
此实施例的视频编解码装置132包括六个视频编解码单元。视频编解码装置132可以弹性地根据虚拟部件的数量以及所需资源，以视频编解码单元为单位做弹性的调配。例如：视频编解码装置132虚拟化为四个虚拟视频编解码装置144，假设第一虚拟视频编解码装置及第二虚拟视频编解码装置需要较多的视频编解码资源，则可以分别配置两个视频编解码单元给第一虚拟视频编解码装置及第二虚拟视频编解码装置，而分别配置一个视频编解码单元给其他虚拟视频编解码装置144。又例如：视频编解码装置132虚拟化为三个虚拟视频编解码装置144，在任一个虚拟视频编解码装置均不需要较多的视频编解码资源的情况下，则可以分别配置两个视频编解码单元给每个虚拟视频编解码装置144。
视频编解码单元的数量至少应与虚拟视频编解码装置144的数量相同，以确保每个虚拟视频编解码装置144能配置一个视频编解码单元，当视频编解码单元数量多于虚拟视频编解码装置144的数量时，便可以根据实际需求，将视频编解码单元适当地配置给虚拟视频编解码装置144，以增加硬件配置弹性。
同样地，此实施例的JPEG编解码装置134包括六个JPEG编解码单元。JPEG编解码装置134可以弹性地根据虚拟部件的数量以及所需资源，以JPEG编解码单元为单位做弹性的调配，其调配方式与视频编解码装置132无异，故不赘述。
存储装置136可以采用非统一内存存取(non-uniform memory access，NUMA)架构，包括多个DDR通道，存储装置136可以弹性地根据虚拟部件的数量以及所需资源，以DDR通道为单位做弹性的调配，其调配方式与计算装置130、视频编解码装置132及JPEG编解码装置134无异，故不赘述。
基于前述示例性的框架，本披露的热迁移方案分为两阶段：第一阶段是将源服务器上的组态及数据打包并发送至目的服务器，即迁移保存路径；第二阶段是将这些组态及数据放置到目的服务器的相应处，即迁移恢复路径。此热迁移方案是将整个虚拟机的运行状态完整保存下来，同时快速恢复到原有硬件平台甚至是不同硬件平台上，换言之，源服务器和目的服务器可以在同一个硬件平台或不同硬件平台上。恢复以后，虚拟机仍旧平滑运行，用户不会察觉到任何差异。前述两阶段将分别说明之。
图3是示出本披露的一个实施例，其为迁移保存路径的流程图，此实施例的源服务器可以是图1所揭示的系统，图4则示出具有图1架构的源服务器端进行迁移保存路径的示意图。此实施例在用户空间102仍在运行时将片上系统106上的特定虚拟硬件相关的驱动程序、固件和硬件信息、上下文信息及其状态信息打包好自源服务器端送出，所述的“状态信息”可以包括虚拟功能的驱动程序的状态信息、固件和硬件的状态信息、状态机、寄存器、硬件内部状态的上下文状态信息、软件的状态机、变量、常量运行时的上下文状态信息等。
在步骤301中，虚拟化管理软件向仿真虚拟机QEMU 402发起迁移请求。此实施例的虚拟化管理软件是Libvirt 401，Libvirt 401是一种用于管理虚拟化平台的开源应用程序编程接口(API)、后台程序和管理工具，它可以用来管理QEMU 402的虚拟化技术。当片上系统106出现故障、资源分配不均(如负载过重、负载过轻)等情况时，Libvirt 401便可启动热迁移来保证虚拟机业务的正常运行。
在步骤302中，QEMU 402通知物理功能驱动程序403启动迁移，即QEMU 402 初始化热迁移启动请求。此实施例提供一种模型来管理整个迁移保存路径的过程，该模型为虚拟机器学习单元QEMU对象模型(VMLU QOM)，其中虚拟机器学习单元指的是针对图1所示的虚拟化人工智能片上系统，而QEMU对象模型则是模拟特定架构的处理器以及装置与总线的关系。
在模拟特定架构的处理器方面，VMLU QOM 404针对图1的人工智能片上系统的特性模拟出处理器模型。在模拟装置与总线的关系方面，由于片上系统106会通过PCIe 138与内核空间104联系，这种装置连接总线、总线连接装置的关系，由VMLU QOM 404模拟。
至于用户空间102和物理功能驱动程序403之间的互动，便是通过VMLU QOM404的内存映射I/O(Memory mapping I/O，MMIO)实现的。内存映射I/O是PCI规范的一部分，I/O设备被放置在内存空间而不是I/O空间。从用户空间102的处理器的角度来看，内存映射I/O后系统访问其他设备是与访问内存相同的，简化了程序设计的难度和接口的复杂性。
在此步骤中，QEMU 402中的VMLU QOM 404初始化热迁移启动请求，并将热迁移启动请求发送至物理功能驱动程序403。
在步骤303中，物理功能驱动程序403通知虚拟功能驱动程序405启动迁移。虚拟功能驱动程序405存储在虚拟机内核空间中，基于VMLU QOM 404的内存映射I/O，片上系统106的硬件在虚拟功能驱动程序405看来像一块内存，虚拟功能驱动程序405对内存映射I/O空间(即片上系统106)的读和写操作会被VMLU QOM 404捕捉并管理。对于读操作，VMLU QOM 404可以根据虚拟功能驱动程序405的需要返回应返回的值，使得虚拟功能驱动程序405和物理功能驱动程序403之间同步。
VMLU QOM 404通过调用物理功能驱动程序403的接口获取物理功能驱动程序403的迁移状态。当虚拟功能驱动程序405欲读取VMLU QOM 404的内存映射I/O空间时，VMLU QOM 404便将物理功能406的状态返回给虚拟功能驱动程序405，在此步骤中，VMLU QOM 404将物理功能驱动程序403准备进行热迁移的状态传递至虚拟功能驱动程序405。
在步骤304中，虚拟功能驱动程序405暂停执行来自用户空间102的处理器的任务。此实施例所采取的方式是，虚拟功能驱动程序405不返回控制指令至用户空间102的客户操作系统，客户操作系统未收到控制指令，便会继续等待，不会发出下一个任务给虚拟功能驱动程序405，以暂停执行用户空间端的任务。
在步骤305中，虚拟功能驱动程序405通知物理功能驱动程序403准备进行迁移。当暂停执行来自用户空间102的指令后，虚拟功能驱动程序405通知物理功能驱动程序403用户空间102已准备就绪，在热迁移期间不会有指令下发干扰。
在步骤306中，物理功能驱动程序403通知物理功能406准备进行迁移。物理功能驱动程序403发送热迁移启动请求至物理功能406，所述热迁移启动请求指定特定虚拟硬件408进行热迁移，特定虚拟硬件408为片上系统106多个虚拟硬件其中之一，亦即，所述热迁移启动请求记载针对特定虚拟硬件408进行热迁移。为说明方便，在此假设热迁移启动请求是针对特定虚拟功能407及其相对应的特定虚拟硬件408。
特定虚拟硬件408可以是特定的虚拟计算装置，例如图1中的虚拟计算装置142，则待迁移数据包括存储在虚拟共享存储单元的计算中间值，以及存储在虚拟存储单元 核的数据。特定虚拟硬件408也可以是图1的特定虚拟存储装置148，则待迁移数据包括存储在特定虚拟存储装置148的数据。特定虚拟硬件408还可以是虚拟视频编解码装置144或是虚拟JPEG编解码装置146，则待迁移数据包括相应的编解码信息。
在步骤307中，物理功能406向物理功能驱动程序403上传包括特定虚拟功能407的驱动程序、固件和特定虚拟硬件408的信息、上下文信息及其状态信息的数据。首先，物理功能406发送待迁移指令至内核空间104的物理功能驱动程序403，所述待迁移指令记录了与特定虚拟硬件408有关的待迁移数据，使物理功能驱动程序403知悉有多少数据需要迁移。接着，物理功能406响应热迁移启动请求，闲置特定虚拟功能407，使其暂时勿与用户空间102进行沟通，但其他的虚拟功能及其相对应的虚拟硬件则照常运行。在闲置特定虚拟功能407后，物理功能406自特定虚拟硬件408分批取出待迁移数据，并发送至物理功能驱动程序403。当待迁移数据发送完毕后，物理功能406发送结束信号至物理功能驱动程序403。
在步骤308，VMLU QOM 404从物理功能驱动程序403获取待迁移数据。内核空间的物理功能驱动程序403先将待迁移指令发送至用户虚拟机里的VMLU QOM 404，再将待迁移数据发送至VMLU QOM 404。VMLU QOM 404接收到待迁移指令后，调用读取函数，从物理功能驱动程序403获取待迁移数据。
在步骤309中，VMLU QOM 404将待迁移数据传输到Libvirt 401。
在步骤310中，当待迁移数据发送完毕后，物理功能406释放特定虚拟硬件408及特定虚拟功能407的资源，VMLU QOM 404发送结束信号至虚拟功能驱动程序405，虚拟功能驱动程序405发送控制信号至虚拟功能驱动程序的接口409，客户操作系统接续下发任务。
通过上述的流程，此实施例实现迁移保存路径，在执行前述流程的同时，对于非特定虚拟功能及硬件来说，仍旧执行来自用户空间102的任务，不受影响。
本披露的另一个实施例是迁移恢复路径，此实施例的目的服务器亦为图1的系统，与源服务器具有同样的环境。图5是示出迁移恢复路径的流程图，图6则示出迁移恢复路径在图1环境中的示意图。更详细来说，此实施例是在图3及图4的实施例完成迁移保存路径后，接着将待迁移数据迁移到目的服务器上。
在步骤501中，Libvirt 601向QEMU 602发起待迁移数据导入的请求。QEMU 602自片外接收在前一个实施例中发出的待迁移数据，并初始化热迁移启动请求。所述片外指的是源服务器，源服务器与目的服务器可能在同一个硬件平台或不同硬件平台上。
在步骤502中，VMLU QOM 604向物理功能驱动程序603发送待迁移数据。VMLU QOM 604接收到待迁移数据后，响应热迁移启动请求，调用写入函数，将待迁移数据发送至物理功能驱动程序603。
在步骤503中，物理功能606接收待迁移数据。在前一个步骤中，VMLU QOM 604将待迁移数据发送至物理功能驱动程序603，物理功能驱动程序603再将待迁移数据发送至物理功能606。
在步骤504中，针对特定虚拟功能607及特定虚拟硬件608恢复组态、数据及其上下文。
首先，物理功能606闲置特定虚拟功能607，使其暂时勿与用户空间102进行沟通，但其他的虚拟功能则照常运行。在闲置特定虚拟功能607后，物理功能606通过 特定虚拟功能607发送待迁移数据至特定虚拟硬件608。
同样地，特定虚拟硬件608可以是图1的虚拟计算装置、特定虚拟存储装置、虚拟视频编解码装置或是虚拟JPEG编解码装置。待迁移数据包括与特定虚拟硬件608相关的驱动程序、固件和硬件信息、上下文信息及状态信息等。回复后，特定虚拟功能607及特定虚拟硬件608与特定虚拟功能407及特定虚拟硬件408具有完全相同的环境及数据。
在步骤505中，物理功能606向物理功能驱动程序603报告迁移已经完成。当待迁移数据发送完毕后，物理功能606发送结束信号至内核空间104的物理功能驱动程序603。
在步骤506中，物理功能驱动程序603通知VMLU QOM 604热迁移已经完成，亦即物理功能驱动程序603发送结束信号至QEMU 602。
在步骤507中，VMLU QOM 604改变状态以便通知虚拟功能驱动程序605热迁移已经完成。VMLU QOM 604响应结束信号，通知虚拟功能驱动程序605热迁移已完成，同时改变基地址寄存器的状态，指向目的服务器的特定虚拟功能607及特定虚拟硬件608。
在步骤508中，虚拟功能驱动程序605向虚拟功能驱动程序的接口609发送控制信号，以重新接续执行客户操作系统的任务。
在步骤509中，虚拟功能驱动程序的接口609通知虚拟功能驱动程序605重新接续执行客户操作系统的任务。虚拟功能驱动程序605重新通过虚拟功能驱动程序的接口609接收来自用户空间102的处理器的任务，这些任务不再访问源服务器的特定虚拟硬件408，而改以访问目的服务器的特定虚拟硬件608。
在步骤510中，VMLU QOM 604通知Libvirt 601热迁移完成，Libvirt 601清理源服务器上所分配的硬件资源。至此完成迁移恢复路径。
结合前述迁移保存路径及迁移恢复路径的实施例，本披露得以实现虚拟化专用集成电路的热迁移。
图7是示出根据本披露实施例的一种集成电路装置700的结构图。如图7所示，集成电路装置700即是前述多个实施例中的人工智能片上系统，其包括计算装置702，计算装置702可以是图1的计算装置130。另外，集成电路装置700还包括通用互联接口704和其他处理装置706。
其他处理装置706可以是中央处理器、图形处理器、人工智能处理器等通用和/或专用处理器中的一种或多种类型的处理器，其数目不做限制而是依实际需要来确定。其他处理装置706作为计算装置702与外部数据和控制的接口，执行包括但不限于数据搬运，完成对计算装置702的开启、停止等的基本控制。其他处理装置706也可以和计算装置702协作共同完成运算任务。
通用互联接口704可以用于在计算装置702与其他处理装置706间传输数据和控制指令。例如，计算装置702可以经由通用互联接口704从其他处理装置706中获取所需的输入数据，写入计算装置702片上的存储单元。进一步，计算装置702可以经由通用互联接口704从其他处理装置706中获取控制指令，写入计算装置702片上的控制缓存。替代地或可选地，通用互联接口704也可以读取计算装置702的存储模块中的数据并传输给其他处理装置706。
集成电路装置700还包括存储装置708，其可以分别与计算装置702和其他处理装置706连接。存储装置708用于保存计算装置702和其他处理装置706的数据，尤其适用于所需要运算的数据在计算装置702或其他处理装置706的内部存储中无法全部保存的数据。
根据应用场景的不同，集成电路装置700可以作为手机、机器人、无人机、视频采集等设备的片上系统(SOC)，从而有效地降低控制部分的核心面积，提高处理速度并降低整体的功耗。在此情况时，集成电路装置700的通用互联接口704与设备的某些部件相连接。此处的某些部件可以例如是摄像头，显示器，鼠标，键盘，网卡或wifi接口。
本披露还揭露一种芯片或集成电路芯片，其包括了集成电路装置700。本披露还揭露一种芯片封装结构，其包括了上述芯片。
本披露另一个实施例是一种板卡，其包括了上述芯片封装结构。参阅图8，板卡800除了包括多个上述芯片802以外，还可以包括其他的配套部件，该配套部件包括存储器件804、接口装置806和控制器件808。
存储器件804与芯片封装结构内的芯片802通过总线814连接，用于存储数据。存储器件804可以包括多组存储单元810。
接口装置806与所述芯片封装结构内的芯片802电连接。所述接口装置806用于实现芯片802与外部设备812(例如服务器或计算机)之间的数据传输。在此实施例中，接口装置806为标准PCIe接口，待处理的数据由服务器通过标准PCIe接口传递至芯片802，实现数据转移。芯片802的计算结果亦由接口装置806传送回外部设备812。
控制器件808与芯片802电连接，以便对芯片802的状态进行监控。具体地，芯片802与控制器件808可以通过SPI接口电连接。控制器件808可以包括单片机(“MCU”，Micro Controller Unit)。
本披露的另一个实施例是一种电子设备或装置，其包括了上述板卡800。根据不同的应用场景，电子设备或装置可以包括数据处理装置、机器人、电脑、打印机、扫描仪、平板电脑、智能终端、手机、行车记录仪、导航仪、传感器、摄像头、服务器、云端服务器、相机、摄像机、投影仪、手表、耳机、移动存储、可穿戴设备、交通工具、家用电器、和/或医疗设备。所述交通工具包括飞机、轮船和/或车辆；所述家用电器包括电视、空调、微波炉、冰箱、电饭煲、加湿器、洗衣机、电灯、燃气灶、油烟机；所述医疗设备包括核磁共振仪、B超仪和/或心电图仪。
本披露另一个实施例为一种计算机可读存储介质，其上存储有热迁移的计算机程序代码，当所述计算机程序代码由处理器运行时，执行如图3或图5所述的方法。
本披露可以实现将源服务器中的虚拟硬件的驱动程序、固件和硬件信息、上下文信息及其状态信息热迁移到目的服务器上，特别是热迁移专用集成电路的虚拟计算装置、虚拟视频编解码装置、虚拟JPEG编解码装置等。
依据以下条款可更好地理解前述内容：
条款A1、一种实现热迁移的系统，连接至内核空间，包括：硬件，虚拟化为多个虚拟硬件；以及只读存储器，存储有固件，该固件包括：物理功能，用以：接收来自所述内核空间的热迁移启动请求，所述热迁移启动请求指定特定虚拟硬件，所述特 定虚拟硬件为所述多个虚拟硬件其中之一；发送待迁移指令至所述内核空间，所述待迁移指令记录待迁移数据，所述待迁移数据关联至所述特定虚拟硬件；自所述特定虚拟硬件分批取出所述待迁移数据并发送至所述内核空间；以及当所述待迁移数据发送完毕后，发送结束信号至所述内核空间。
条款A2、根据A1所述的系统，其中所述硬件包括计算装置，所述特定虚拟硬件为特定虚拟计算装置，所述计算装置包括：智能处理装置，用以执行神经网络的卷积计算；以及共享存储单元，用以暂存所述卷积计算的计算中间值，所述共享存储单元虚拟化为多个虚拟共享存储单元，所述特定虚拟计算装置对应一个虚拟共享存储单元；其中，所述待迁移数据包括存储在所述虚拟共享存储单元的计算中间值。
条款A3、根据A2所述的系统，其中所述计算装置还包括：存储单元核，虚拟化为多个虚拟存储单元核，所述特定虚拟计算装置配置一个虚拟存储单元核；其中，所述待迁移数据包括存储在所述虚拟存储单元核的数据。
条款A4、根据A1所述的系统，其中所述硬件包括存储装置，虚拟化为多个虚拟存储装置，所述特定虚拟硬件为特定虚拟存储装置，所述待迁移数据包括存储在所述特定虚拟存储装置的数据。
条款A5、根据A1所述的系统，其中所述硬件包括视频编解码装置，所述特定虚拟硬件为特定虚拟视频编解码装置。
条款A6、根据A1所述的系统，其中所述硬件包括JPEG编解码装置，所述特定虚拟硬件为特定虚拟JPEG编解码装置。
条款A7、根据A1所述的系统，其中所述固件还包括多个虚拟功能，受所述物理功能管理，所述特定虚拟硬件对应一个虚拟功能，所述物理功能响应所述热迁移启动请求，闲置所述虚拟功能。
条款A8、根据A1所述的系统，其中当所述待迁移数据发送完毕后，所述物理功能释放所述硬件及所述虚拟功能。
条款A9、根据A1所述的系统，其中所述待迁移数据包括驱动程序、固件和硬件信息、上下文信息及状态信息其中之一。
条款A10、根据A1所述的系统，其中所述内核空间连接至用户空间，所述用户空间载有用户虚拟机，所述用户虚拟机初始化所述热迁移启动请求。
条款A11、根据A10所述的系统，其中所述用户虚拟机为快速仿真器。
条款A12、根据A10所述的系统，其中所述内核空间将所述待迁移指令发送至所述用户虚拟机，并将所述待迁移数据发送至所述用户虚拟机。
条款A13、根据A10所述的系统，其中所述内核空间载有物理功能驱动程序，所述用户空间包括用户端内核空间，所述用户端内核空间载有虚拟功能驱动程序，所述物理功能驱动程序接收来自所述用户虚拟机的所述热迁移启动请求并发送至所述虚拟功能驱动程序，所述虚拟功能驱动程序暂停执行来自所述用户空间的任务。
条款A14、根据A13所述的系统，其中所述虚拟功能驱动程序不返回控制指令至所述用户空间，以暂停执行所述任务。
条款A15、根据A13所述的系统，其中所述内核空间发送所述结束信号至所述虚拟功能驱动程序，所述虚拟功能驱动程序接续执行所述任务。
条款A16、根据A13所述的系统，其中当暂停执行来自所述用户空间的任务后， 所述虚拟功能驱动程序通知所述物理功能驱动程序，所述物理功能驱动程序发送所述热迁移启动请求至所述物理功能。
条款A17、一种实现热迁移的系统，连接至内核空间，包括：硬件，虚拟化为多个虚拟硬件；以及只读存储器，存储有固件，该固件包括：物理功能，用以接收待迁移数据，所述待迁移数据关联至特定虚拟硬件，所述特定虚拟硬件为所述多个虚拟硬件其中之一；以及多个虚拟功能，受所述物理功能管理，所述特定虚拟硬件对应一个虚拟功能；其中，所述物理功能通过所述虚拟功能发送所述待迁移数据至所述特定虚拟硬件；当所述待迁移数据发送完毕后，所述物理功能发送结束信号至所述内核空间。
条款A18、根据A17所述的系统，其中所述硬件包括计算装置，所述特定虚拟硬件为特定虚拟计算装置，所述计算装置包括：智能处理装置，用以执行神经网络的卷积计算；以及共享存储单元，用以暂存所述卷积计算的计算中间值，所述共享存储单元虚拟化为多个虚拟共享存储单元，所述特定虚拟计算装置对应一个虚拟共享存储单元；其中，所述虚拟功能将所述待迁移数据中的计算中间值存储至所述虚拟共享存储单元。
条款A19、根据A18所述的系统，其中所述计算装置还包括：存储单元核，虚拟化为多个虚拟存储单元核，所述特定虚拟计算装置配置一个虚拟存储单元核；其中，所述虚拟功能将所述待迁移数据中的相应数据存储至所述虚拟存储单元核。
条款A20、根据A17所述的系统，其中所述硬件包括存储装置，虚拟化为多个虚拟存储装置，所述特定虚拟硬件为特定虚拟存储装置，所述虚拟功能将所述待迁移数据中的相应数据存储至所述虚拟存储装置。
条款A21、根据A17所述的系统，其中所述硬件包括视频编解码装置，所述特定虚拟硬件为特定虚拟视频编解码装置。
条款A22、根据A17所述的系统，其中所述硬件包括JPEG编解码装置，所述特定虚拟硬件为特定虚拟JPEG编解码装置。
条款A23、根据A17所述的系统，其中所述待迁移数据包括驱动程序、固件和硬件信息、上下文信息及状态信息其中之一。
条款A24、根据A17所述的系统，其中所述内核空间连接至用户空间，所述用户空间载有用户虚拟机，所述用户虚拟机自片外接收所述待迁移数据并初始化热迁移启动请求。
条款A25、根据A24所述的系统，其中所述用户虚拟机为快速仿真器。
条款A26、根据A24所述的系统，其中所述内核空间载有物理功能驱动程序，用以自所述用户虚拟机接收所述待迁移数据，并响应所述热迁移启动请求，将所述待迁移数据发送至所述物理功能。
条款A27、根据A26所述的系统，其中所述物理功能驱动程序发送所述结束信号至所述用户虚拟机。
条款A28、根据A27所述的系统，其中所述用户空间包括用户端内核空间，所述用户端内核空间载有虚拟功能驱动程序，所述用户虚拟机响应所述结束信号，通知所述虚拟功能驱动程序所述热迁移已完成，所述虚拟功能驱动程序接收来自所述用户空间的任务，所述任务控制所述特定虚拟硬件。
条款A29、根据A28所述的系统，其中所述用户虚拟机响应所述结束信号，改变基地址寄存器的状态，所述基地址寄存器指向所述特定虚拟硬件。
条款A30、一种集成电路装置，包括条款A1-29的任意一项所述的系统。
条款A31、一种板卡，包括条款A30所述的集成电路装置。
条款A32、一种对系统进行热迁移保存路径的方法，所述系统包括硬件，虚拟化为多个虚拟硬件，所述方法包括：接收热迁移启动请求，所述热迁移启动请求指定特定虚拟硬件，所述特定虚拟硬件为所述多个虚拟硬件其中之一；闲置特定虚拟功能，所述特定虚拟功能对应至所述特定虚拟硬件；自所述特定虚拟硬件分批发送待迁移数据；以及当所述待迁移数据发送完毕后，发送结束信号。
条款A33、根据A32所述的方法，其中所述硬件为人工智能片上系统的计算装置、存储装置、视频编解码装置及JPEG编解码装置其中之一。
条款A34、一种对系统进行热迁移恢复路径的方法，所述系统包括硬件，虚拟化为多个虚拟硬件，所述方法包括：接收待迁移数据，所述热迁移数据对应至特定虚拟硬件，所述特定虚拟硬件为所述多个虚拟硬件其中之一；闲置特定虚拟功能，所述特定虚拟功能对应至所述特定虚拟硬件；通过特定虚拟功能发送所述待迁移数据至所述特定虚拟硬件；以及当所述待迁移数据发送完毕后，发送结束信号。
条款A35、根据A34所述的方法，其中所述硬件为人工智能片上系统的计算装置、存储装置、视频编解码装置及JPEG编解码装置其中之一。
条款A36、一种计算机可读存储介质，其上存储有热迁移的计算机程序代码，当所述计算机程序代码由处理器运行时，执行条款A 32-35的任意一项所述的方法。

Claims (36)

  1. 一种实现热迁移的系统，连接至内核空间，包括：
    硬件，虚拟化为多个虚拟硬件；以及
    只读存储器，存储有固件，该固件包括：
    物理功能，用以：
    接收来自所述内核空间的热迁移启动请求，所述热迁移启动请求指定特定虚拟硬件，所述特定虚拟硬件为所述多个虚拟硬件其中之一；
    发送待迁移指令至所述内核空间，所述待迁移指令记录待迁移数据，所述待迁移数据关联至所述特定虚拟硬件；
    自所述特定虚拟硬件分批取出所述待迁移数据并发送至所述内核空间；以及
    当所述待迁移数据发送完毕后，发送结束信号至所述内核空间。
  2. 根据权利要求1所述的系统，其中所述硬件包括计算装置，所述特定虚拟硬件为特定虚拟计算装置，所述计算装置包括：
    智能处理装置，用以执行神经网络的卷积计算；以及
    共享存储单元，用以暂存所述卷积计算的计算中间值，所述共享存储单元虚拟化为多个虚拟共享存储单元，所述特定虚拟计算装置对应一个虚拟共享存储单元；
    其中，所述待迁移数据包括存储在所述虚拟共享存储单元的计算中间值。
  3. 根据权利要求2所述的系统，其中所述计算装置还包括：
    存储单元核，虚拟化为多个虚拟存储单元核，所述特定虚拟计算装置配置一个虚拟存储单元核；
    其中，所述待迁移数据包括存储在所述虚拟存储单元核的数据。
  4. 根据权利要求1所述的系统，其中所述硬件包括存储装置，虚拟化为多个虚拟存储装置，所述特定虚拟硬件为特定虚拟存储装置，所述待迁移数据包括存储在所述特定虚拟存储装置的数据。
  5. 根据权利要求1所述的系统，其中所述硬件包括视频编解码装置，所述特定虚拟硬件为特定虚拟视频编解码装置。
  6. 根据权利要求1所述的系统，其中所述硬件包括JPEG编解码装置，所述特定虚拟硬件为特定虚拟JPEG编解码装置。
  7. 根据权利要求1所述的系统，其中所述固件还包括多个虚拟功能，受所述物理功能管理，所述特定虚拟硬件对应一个虚拟功能，所述物理功能响应所述热迁移启动请求，闲置所述虚拟功能。
  8. 根据权利要求1所述的系统，其中当所述待迁移数据发送完毕后，所述物理功能释放所述硬件及所述虚拟功能。
  9. 根据权利要求1所述的系统，其中所述待迁移数据包括驱动程序、固件和硬件信息、上下文信息及状态信息其中之一。
  10. 根据权利要求1所述的系统，其中所述内核空间连接至用户空间，所述用户空间载有用户虚拟机，所述用户虚拟机初始化所述热迁移启动请求。
  11. 根据权利要求10所述的系统，其中所述用户虚拟机为快速仿真器。
  12. 根据权利要求10所述的系统，其中所述内核空间将所述待迁移指令发送至 所述用户虚拟机，并将所述待迁移数据发送至所述用户虚拟机。
  13. 根据权利要求10所述的系统，其中所述内核空间载有物理功能驱动程序，所述用户空间包括用户端内核空间，所述用户端内核空间载有虚拟功能驱动程序，所述物理功能驱动程序接收来自所述用户虚拟机的所述热迁移启动请求并发送至所述虚拟功能驱动程序，所述虚拟功能驱动程序暂停执行来自所述用户空间的任务。
  14. 根据权利要求13所述的系统，其中所述虚拟功能驱动程序不返回控制指令至所述用户空间，以暂停执行所述任务。
  15. 根据权利要求13所述的系统，其中所述内核空间发送所述结束信号至所述虚拟功能驱动程序，所述虚拟功能驱动程序接续执行所述任务。
  16. 根据权利要求13所述的系统，其中当暂停执行来自所述用户空间的任务后，所述虚拟功能驱动程序通知所述物理功能驱动程序，所述物理功能驱动程序发送所述热迁移启动请求至所述物理功能。
  17. 一种实现热迁移的系统，连接至内核空间，包括：
    硬件，虚拟化为多个虚拟硬件；以及
    只读存储器，存储有固件，该固件包括：
    物理功能，用以接收待迁移数据，所述待迁移数据关联至特定虚拟硬件，所述特定虚拟硬件为所述多个虚拟硬件其中之一；以及
    多个虚拟功能，受所述物理功能管理，所述特定虚拟硬件对应一个虚拟功能；
    其中，所述物理功能通过所述虚拟功能发送所述待迁移数据至所述特定虚拟硬件；当所述待迁移数据发送完毕后，所述物理功能发送结束信号至所述内核空间。
  18. 根据权利要求17所述的系统，其中所述硬件包括计算装置，所述特定虚拟硬件为特定虚拟计算装置，所述计算装置包括：
    智能处理装置，用以执行神经网络的卷积计算；以及
    共享存储单元，用以暂存所述卷积计算的计算中间值，所述共享存储单元虚拟化为多个虚拟共享存储单元，所述特定虚拟计算装置对应一个虚拟共享存储单元；
    其中，所述虚拟功能将所述待迁移数据中的计算中间值存储至所述虚拟共享存储单元。
  19. 根据权利要求18所述的系统，其中所述计算装置还包括：
    存储单元核，虚拟化为多个虚拟存储单元核，所述特定虚拟计算装置配置一个虚拟存储单元核；
    其中，所述虚拟功能将所述待迁移数据中的相应数据存储至所述虚拟存储单元核。
  20. 根据权利要求17所述的系统，其中所述硬件包括存储装置，虚拟化为多个虚拟存储装置，所述特定虚拟硬件为特定虚拟存储装置，所述虚拟功能将所述待迁移数据中的相应数据存储至所述虚拟存储装置。
  21. 根据权利要求17所述的系统，其中所述硬件包括视频编解码装置，所述特定虚拟硬件为特定虚拟视频编解码装置。
  22. 根据权利要求17所述的系统，其中所述硬件包括JPEG编解码装置，所述特定虚拟硬件为特定虚拟JPEG编解码装置。
  23. 根据权利要求17所述的系统，其中所述待迁移数据包括驱动程序、固件和硬件信息、上下文信息及状态信息其中之一。
  24. 根据权利要求17所述的系统，其中所述内核空间连接至用户空间，所述用户空间载有用户虚拟机，所述用户虚拟机自片外接收所述待迁移数据并初始化热迁移启动请求。
  25. 根据权利要求24所述的系统，其中所述用户虚拟机为快速仿真器。
  26. 根据权利要求24所述的系统，其中所述内核空间载有物理功能驱动程序，用以自所述用户虚拟机接收所述待迁移数据，并响应所述热迁移启动请求，将所述待迁移数据发送至所述物理功能。
  27. 根据权利要求26所述的系统，其中所述物理功能驱动程序发送所述结束信号至所述用户虚拟机。
  28. 根据权利要求27所述的系统，其中所述用户空间包括用户端内核空间，所述用户端内核空间载有虚拟功能驱动程序，所述用户虚拟机响应所述结束信号，通知所述虚拟功能驱动程序所述热迁移已完成，所述虚拟功能驱动程序接收来自所述用户空间的任务，所述任务控制所述特定虚拟硬件。
  29. 根据权利要求28所述的系统，其中所述用户虚拟机响应所述结束信号，改变基地址寄存器的状态，所述基地址寄存器指向所述特定虚拟硬件。
  30. 一种集成电路装置，包括根据权利要求1-29的任意一项所述的系统。
  31. 一种板卡，包括根据权利要求30所述的集成电路装置。
  32. 一种对系统进行热迁移保存路径的方法，所述系统包括硬件，虚拟化为多个虚拟硬件，所述方法包括：
    接收热迁移启动请求，所述热迁移启动请求指定特定虚拟硬件，所述特定虚拟硬件为所述多个虚拟硬件其中之一；
    闲置特定虚拟功能，所述特定虚拟功能对应至所述特定虚拟硬件；
    自所述特定虚拟硬件分批发送待迁移数据；以及
    当所述待迁移数据发送完毕后，发送结束信号。
  33. 根据权利要求32所述的方法，其中所述硬件为人工智能片上系统的计算装置、存储装置、视频编解码装置及JPEG编解码装置其中之一。
  34. 一种对系统进行热迁移恢复路径的方法，所述系统包括硬件，虚拟化为多个虚拟硬件，所述方法包括：
    接收待迁移数据，所述热迁移数据对应至特定虚拟硬件，所述特定虚拟硬件为所述多个虚拟硬件其中之一；
    闲置特定虚拟功能，所述特定虚拟功能对应至所述特定虚拟硬件；
    通过特定虚拟功能发送所述待迁移数据至所述特定虚拟硬件；以及
    当所述待迁移数据发送完毕后，发送结束信号。
  35. 根据权利要求34所述的方法，其中所述硬件为人工智能片上系统的计算装置、存储装置、视频编解码装置及JPEG编解码装置其中之一。
  36. 一种计算机可读存储介质，其上存储有热迁移的计算机程序代码，当所述计算机程序代码由处理器运行时，执行权利要求32-35的任意一项所述的方法。
PCT/CN2021/092199 2020-05-07 2021-05-07 实现热迁移的方法、芯片、板卡和存储介质 WO2021223744A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US17/923,677 US20230185595A1 (en) 2020-05-07 2021-05-07 Method for realizing live migration, chip, board, and storage medium
EP21800406.7A EP4148568A1 (en) 2020-05-07 2021-05-07 Method for realizing live migration, chip, board, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010379648.XA CN113626131B (zh) 2020-05-07 2020-05-07 实现热迁移的方法、芯片、板卡和存储介质
CN202010379648.X 2020-05-07

Publications (1)

Publication Number Publication Date
WO2021223744A1 true WO2021223744A1 (zh) 2021-11-11

Family

ID=78377071

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/092199 WO2021223744A1 (zh) 2020-05-07 2021-05-07 实现热迁移的方法、芯片、板卡和存储介质

Country Status (4)

Country Link
US (1) US20230185595A1 (zh)
EP (1) EP4148568A1 (zh)
CN (1) CN113626131B (zh)
WO (1) WO2021223744A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114500718A (zh) * 2021-12-27 2022-05-13 天翼云科技有限公司 一种云手机管理方法及装置
WO2023179388A1 (zh) * 2022-03-25 2023-09-28 阿里巴巴（中国）有限公司 一种虚拟机实例的热迁移方法

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101398770A (zh) * 2007-09-30 2009-04-01 赛门铁克公司 迁移一个或多个虚拟机的系统和方法
CN102262557A (zh) * 2010-05-25 2011-11-30 运软网络科技(上海)有限公司 通过总线架构构建虚拟机监控器的方法及性能服务框架
US20120266171A1 (en) * 2011-04-14 2012-10-18 Sungkyunkwan University Apparatus and method for carrying out a migration process of a virtual machine in a device
US8880470B2 (en) * 2009-09-14 2014-11-04 Vmware, Inc. Method and system for performing live migration of persistent data of a virtual machine
CN108121574A (zh) * 2016-11-29 2018-06-05 阿里巴巴集团控股有限公司 一种内核模块的热升级方法、装置及虚拟机系统
US20190129744A1 (en) * 2016-06-29 2019-05-02 Altera Corporation Live migration of hardware accelerated applications

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7904914B2 (en) * 2008-09-30 2011-03-08 Microsoft Corporation On-the-fly replacement of physical hardware with emulation
US9054917B2 (en) * 2012-03-08 2015-06-09 Empire Technology Development Llc Secure migration of virtual machines
CN107533478A (zh) * 2015-07-31 2018-01-02 慧与发展有限责任合伙企业 计算机系统的迁移
CN108037980B (zh) * 2018-01-02 2023-03-14 腾讯科技（深圳）有限公司 控制执行虚拟机热迁移的方法、装置和可读存储介质
US10684887B2 (en) * 2018-05-25 2020-06-16 Vmware, Inc. Live migration of a virtualized compute accelerator workload

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101398770A (zh) * 2007-09-30 2009-04-01 赛门铁克公司 迁移一个或多个虚拟机的系统和方法
US8880470B2 (en) * 2009-09-14 2014-11-04 Vmware, Inc. Method and system for performing live migration of persistent data of a virtual machine
CN102262557A (zh) * 2010-05-25 2011-11-30 运软网络科技(上海)有限公司 通过总线架构构建虚拟机监控器的方法及性能服务框架
US20120266171A1 (en) * 2011-04-14 2012-10-18 Sungkyunkwan University Apparatus and method for carrying out a migration process of a virtual machine in a device
US20190129744A1 (en) * 2016-06-29 2019-05-02 Altera Corporation Live migration of hardware accelerated applications
CN108121574A (zh) * 2016-11-29 2018-06-05 阿里巴巴集团控股有限公司 一种内核模块的热升级方法、装置及虚拟机系统

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114500718A (zh) * 2021-12-27 2022-05-13 天翼云科技有限公司 一种云手机管理方法及装置
CN114500718B (zh) * 2021-12-27 2023-11-03 天翼云科技有限公司 一种云手机管理方法及装置
WO2023179388A1 (zh) * 2022-03-25 2023-09-28 阿里巴巴（中国）有限公司 一种虚拟机实例的热迁移方法

Also Published As

Publication number Publication date
CN113626131B (zh) 2023-03-28
EP4148568A1 (en) 2023-03-15
CN113626131A (zh) 2021-11-09
US20230185595A1 (en) 2023-06-15

Similar Documents

Publication Publication Date Title
US20210232528A1 (en) Configurable device interface
US9798565B2 (en) Data processing system and method having an operating system that communicates with an accelerator independently of a hypervisor
US8065441B2 (en) Method and apparatus for supporting universal serial bus devices in a virtualized environment
WO2021223744A1 (zh) 实现热迁移的方法、芯片、板卡和存储介质
US10140214B2 (en) Hypervisor translation bypass by host IOMMU with virtual machine migration support
US20180217859A1 (en) Technologies for duplicating virtual machine states
CN111309649B (zh) 一种数据传输和任务处理方法、装置及设备
CN113326226A (zh) 一种虚拟化的方法、装置、板卡及计算机可读存储介质
CN113568734A (zh) 基于多核处理器的虚拟化方法、系统、多核处理器和电子设备
CN112433823A (zh) 动态虚拟化物理卡的设备及方法
WO2022062510A1 (zh) 实现热迁移的设备及方法
CN114281467A (zh) 实现热迁移的系统方法、装置和存储介质
CN115202808A (zh) 一种用于虚拟化环境中片上系统的dma方法及系统
CN113326091A (zh) 一种虚拟化的方法、设备、板卡及计算机可读存储介质
CN113326118A (zh) 基于多核处理器的虚拟化方法、系统、多核处理器和电子设备
US11216211B1 (en) Emulated block storage for devices
CN111340202B (zh) 运算方法、装置及相关产品
WO2021170055A1 (zh) 一种虚拟化的方法、设备、板卡及计算机可读存储介质
CN114281468A (zh) 实现热迁移的装置、相关方法及其可读存储介质
WO2021170054A1 (zh) 一种虚拟化的方法、设备、板卡及计算机可读存储介质
CN114281749A (zh) 实现序列化及反序列化树数据的装置、方法和存储介质
CN114328366A (zh) 实现序列化及反序列化嵌套数据的方法、装置和存储介质
CN114281750A (zh) 实现序列化及反序列化逻辑指针的方法、装置和存储介质
CN113326110A (zh) 一种片上系统及板卡
CN114816648A (zh) 一种计算装置和计算方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21800406

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2021800406

Country of ref document: EP

Effective date: 20221207