CN102402462B - Techniques for load balancing GPU enabled virtual machines - Google Patents

Abstract

The present invention discloses techniques for load balancing GPU enabled virtual machines. Exemplary techniques for balancing 3D graphical processor unit use among virtual machines are herein disclosed. In an exemplary embodiment, a virtualization platform can load an instance of a graphics rendering module for a virtual machine; select a GPU for the graphics rendering module to run on; and configure the virtual machine to render to the selected GPU. In addition to the foregoing, other aspects are described in the detailed description, claims, and figures.

Description

For carrying out the technology of load balance to the virtual machine enabling GPU

The cross reference of CROSS REFERENCE

The application on theme be entitled as the U.S. Patent application xth x/xxx of " Dynamic Virtual Device Failure Recovery (dynamic virtual machine equipment failure recovers) ", No. xxx (attorney docket is MVIR-0670/330194.01) is relevant, and the content of this application by reference entirety is incorporated into this.

Technical field

The present invention relates to Computer display technology, particularly relate to the technology for carrying out load balance to the virtual machine enabling GPU.

Background technology

Virtual machine platform performs while running this operating system allow multiple objective operating system in physical machine in the virtual machine of each operating system oneself.Can be virtual desktop session by the exemplary services provided in virtual machine.Virtual desktop session is the personal computer environment its user interface being sent to remote computer run in virtual machine in itself.This architecture is similar to remote desktop environment, but changes into multiple user is connected to server class operating system simultaneously, and in virtual desktop session, each user can access themselves the commercial operation system performed in virtual machine.

Modern operating system implements three-dimensional (3D) graphic user interface for 3D application/video-game and operating system user interface thereof.User enjoys the experience with 3D environmental interaction, and expects to send 3D graphical stream to client computer in virtual desktop session; But, be difficult to realize stream due to many reasons and transmit 3D figure.Such as, the action of stream transmission 3D figure needs bandwidth and/or compression.In addition, needs are comprised 3D Graphics Processing Unit (GPU) by virtual desktop server, and it can perform 3D function.3D graphics process is resource-intensive, and GPU only can play up several 3D graphic user interface usually at every turn.If these GPU have to play up multi-graph user interface, then they can become overload rapidly.Regrettably, in virtual desktop is disposed, GPU may have to play up a large amount of 3D graphic user interface simultaneously.This can cause one or more operation overtime, and graphics driver and then will reset Graphics Processing Unit---it can cause 3D application program to stop.Therefore, need for preventing GPU from becoming overload and the technology of collapse.

Summary of the invention

Illustrative embodiments comprises a kind of system.In this example, this system includes but not limited to processor and when computer system can operate and the storer of this processor communication.In this example, storer can comprise computer-readable instruction, and this computer-readable instruction makes processor when performing: assign one group of virtual machine to play up 3D figure in a 3D Graphics Processing Unit; At least be used for that response institute's time spent amount is carried out to order based on a 3D Graphics Processing Unit and determine that a 3D Graphics Processing Unit (504) is overused; At least in response to the judgement be overused a 3D Graphics Processing Unit, the first virtual machine is moved to second graph processing unit from this group virtual machine.In addition to the above, other technologies are described in claims, embodiment and accompanying drawing.

Another illustrative embodiments comprises computer-readable recording medium.In this example, computer-readable stores instruction and comprises computer-readable instruction, and this computer-readable instruction makes processor when performing: the amount estimating the graphic memory for playing up 3D figure for the first virtual machine; The latency value be associated in response to the direct memory access (DMA) affairs determined with sent by the one group of virtual machine being assigned render graphics in the first Graphics Processing Unit is less than first threshold and the estimated amount for the graphic memory playing up 3D figure is less than Second Threshold, selects a 3D Graphics Processing Unit from multiple 3D Graphics Processing Unit; And assign described first virtual machine to play up 3D figure in a 3D Graphics Processing Unit.In addition to the above, other technologies are described in claims, embodiment and accompanying drawing.

Another illustrative embodiments comprises a kind of method.In this example, method includes but not limited to: the amount estimating the graphic memory be used for for virtual machine render graphics; The amount of the amount of estimated graphic memory and the estimated available graphic memory controlled by multiple 3D Graphics Processing Unit is compared; Be chosen as from multiple 3D Graphics Processing Unit the 3D Graphics Processing Unit that virtual machine plays up 3D figure according to comparing; And assign selected 3D Graphics Processing Unit to play up 3D figure for virtual machine.In addition to the above, other technologies are described in claims, embodiment and accompanying drawing.

It will be understood by those skilled in the art that one or more various aspect of the present disclosure can include but not limited to circuit for realizing aspect cited herein and/or programming; This circuit and/or programming can be in fact any combinations being configured to realize the hardware of the aspect quoted herein, software and/or firmware, and this depends on the design alternative of system designer.

Be more than general introduction, and therefore must comprise the simplification of details, vague generalization and omission.It will be apparent to one skilled in the art that this general introduction is just illustrative and be never intended to be restrictive.

Accompanying drawing explanation

Fig. 1 depicts the high level block diagram of computer system.

Fig. 2 depicts the high level block diagram of virtual machine server.

Fig. 3 depicts the high level block diagram of virtual machine server.

Fig. 4 depicts the high level block diagram of virtual desktop server.

Fig. 5 depicts the high level block diagram of virtual desktop server.

Fig. 6 depicts the high level block diagram of virtual desktop server.

Fig. 7 depicts the high level block diagram of data center.

Fig. 8 depicts the operating process that the GPU for balancing between virtual machine uses.

Fig. 9 shows the operating process of the Fig. 8 comprising additional operations.

Figure 10 depicts the operating process that the GPU for balancing between virtual machine uses.

Figure 11 shows the operating process of the Figure 10 comprising additional operations.

Figure 12 depicts the operating process that the GPU for balancing between virtual machine uses.

Figure 13 shows the operating process of the Figure 12 comprising additional operations.

Embodiment

The present invention can use one or more computer system.Fig. 1 and following discussion aim to provide the brief, general description that can realize the suitable computing environment of disclosed theme wherein.

The term circuit running through the use of this instructions can comprise the nextport hardware component NextPorts such as such as hardware interrupt controller, hard disk driver, network adapter, graphic process unit, hardware based video/audio codec, and for operating the firmware of these hardware.Term " circuit " can also comprise the microprocessor carrying out instruction fetch and execution, special IC and the processor that are configured by firmware and/or software, such as the core of multinuclear General Porcess Unit.Processor can be configured by the instruction loaded from the storer such as such as RAM, ROM, firmware and/or massive store, thus specializes and can be used for the logic that configuration processor carrys out n-back test.Circuit comprises in the example embodiment of the combination of hardware and software wherein, and implementor can write the source code of specific logic, and this source code is compiled into the machine readable code that can be performed by hardware subsequently.Because those skilled in the art can understand almost do not have differentiated stage between the function that prior art has evolved to hard-wired function or software simulating, thus selecting hardware or software to realize function described herein is a design alternative.In other words, owing to it will be appreciated by those skilled in the art that software process can be transformed into the hardware configuration of equivalence, and hardware configuration itself can be transformed into the software process of equivalence, therefore selects hardware implementing or software simulating to leave implementor for.

With reference now to Fig. 1, depict exemplary computer system 100.Computer system 100 can comprise processor 102, such as, performs core.Although show a processor 102, in other embodiments, computer system 100 can have multiple processor, and such as each processor substrate has multiple execution core, and/or can have multiple processor substrates of multiple execution core separately.As shown in the figure, various computer-readable recording medium 110 can by one or more interconnects, and various system component is coupled to processor 102 by described system bus.System bus can be any one in the bus structure of several types, comprises memory bus or Memory Controller, peripheral bus and uses the local bus of any one in various bus architecture.In example embodiment, computer-readable recording medium 110 can comprise such as random access memory (RAM) 104, memory device 106 (such as motor hard disk driver, solid state hard disc driver etc.), firmware 108 (such as flash RAM or ROM) and movable memory equipment 118 (such as CD-ROM, floppy disk, DVD, flash driver program, External memory equipment etc.).It will be understood by those of skill in the art that the computer-readable recording medium that can use other types, as tape cassete, flash card and/or digital video disc.

Computer-readable recording medium 110 can provide the non-volatile and volatile storage of other data for computing machine 100 to processor executable 122, data structure, program module and such as executable instruction and so on.Basic input/output (BIOS) 120 can be stored in firmware 108, and it comprises the basic routine helping transmission of information between each element such as between the starting period in computer system 100.The multiple programs comprising operating system and/or application program can be stored on firmware 108, memory device 106, RAM 104 and/or movable memory equipment 118, and can be performed by processor 102.

Order and information can be received by input equipment 116 by computing machine 100, and input equipment can include but not limited to keyboard and pointing device.Other input equipments can comprise microphone, operating rod, game paddle, scanner etc.These and other input equipments are connected to processor 102 usually through the serial port interface being coupled to system bus, but also connect by other interfaces, as parallel port, game port or USB (universal serial bus) (USB).The part that the display device of display or other types can be able to be also graphics processor unit 112 via such as video adapter etc. maybe can be connected to the interface of graphics processor unit 112 to be connected to system bus.Except display, computing machine generally includes other peripheral output devices, as loudspeaker and printer (not shown).The example system of Fig. 1 also can comprise primary adapter, small computer system interface (SCSI) bus and be connected to the External memory equipment of SCSI bus.

Computer system 100 can use to one or more remote computer, and the logic as remote computer is connected in networked environment and operates.Remote computer can be another computing machine, server, router, network PC, peer device or other common network nodes, and about the many or whole element described in computer system 100 above generally including.

When using in LAN or WAN networked environment, computer system 100 is connected to LAN or WAN by network interface unit 114.NIC 114 can be internal or external, and it can be connected to system bus.In networked environment, can be stored in remote memory storage device relative to the program module described by computer system 100 or its part.It is exemplary for being appreciated that described network connects, and can use other means setting up communication link between the computers.Although be particularly useful for computerized system it is conceivable that many embodiments of the present invention, but do not mean that in the present note and limit the invention to those embodiments.

Turn to Fig. 2, show the example virtual platform that can be used to generating virtual machines.In this embodiment, hypervisor microkernel 202 can be configured to control and arbitrate the access of the hardware to computer system 200.Hypervisor microkernel 202 can generate the execution environment being called subregion, as child partition 1 to child partition N (wherein N be greater than 1 integer).At this, child partition is the basic isolated location that hypervisor microkernel 202 is supported.Hypervisor microkernel 202 can isolate the process in a subregion, makes it can not access the resource of another subregion.Each child partition can be mapped to one group of hardware resource under hypervisor microkernel 202 controls, such as, and storer, equipment, processor cycle etc.In each embodiment, hypervisor microkernel 202 can be independently software product, operating system a part, be embedded in the firmware of mainboard, special IC or its combination.

Hypervisor microkernel 202 can implement a point Division by the view of memory of the objective operating system in restriction physical computer system.When hypervisor microkernel 202 instantiation one virtual machine, the page of system physical memory (SPM) (such as, having the regular length memory block of start and end address) can be distributed to virtual machine as objective physical storage (GPM) by it.At this, the limited system storage view of visitor is controlled by hypervisor microkernel 202.Term " objective physical storage " is the shorthand way describing storage page from the viewpoint of virtual machine, and term system physical memory is the shorthand way describing storage page from the viewpoint of physical system.Therefore, the storage page being assigned to virtual machine has objective physical address (address that virtual machine uses) and system physical address (actual address of page).

Visitor's operating system can virtual objective physical storage.Virtual memory is a kind of administrative skill, and it allows operating system excessively to submit storer to, and gives the unique access of application program to continuous working storer.In virtualized environment, objective operating system can use one or more page table that the virtual address translation being known as location, virtual place far way from home is become objective physical address.In this example, storage address can have objective virtual address, objective physical address and system physical address.

In described example, father's division component also can be considered to the territory 0 of the open source system supervisory routine being similar to Xen, and father's division component can comprise main frame 204.Main frame 204 can be operating system (or a group configuration utility routine), and main frame 204 can be configured to by using Virtual Service supplier 228 (VSP) to provide resource to the objective operating system performed in child partition 1-N.VPS 228 (generally increase income in community be called as rear end driver) can be used to carry out multiplexed by virtualization services client computer (VSC) (being commonly referred to as front-end driven program increasing income in community or class virtual equipment) to the interface to hardware resource.As shown in the figure, Virtual Service client computer can perform in the context of objective operating system.But these drivers are different from all the other drivers in passenger plane, because to they providing system supervisor but not passenger plane.In an illustrative embodiments, Virtual Service supplier 228 and Virtual Service client computer 216 the used path that communicates with 218 can be regarded as virtual route.

As shown in the figure, can be configured to run in main frame 204 and be attached to can for the resource of objective operating system 220 and 222 for emulator 234 (such as virtual IDE equipment, virtual video adapter, virtual NIC etc.).Such as, when the memory location residing for the register that the contact of objective operating system is mapped to equipment or the mapped equipment of storer, micro-kernel system supervisor 202 can intercept this request and value transmit passenger plane being attempted write to the emulator be associated.At this, the resource in this example can be considered to the position residing for virtual unit.The use by this way of emulator can be regarded as simulation paths.Simulation paths is poor efficiency compared with virtual path, because compared with pass-along message between VSP and VSC, it needs more cpu resource to carry out emulator.Such as, can by be used for via simulation paths value write dish, a hundreds of action on the storer that is mapped to register is reduced to single message, this message is passed to VSP from VSC in virtual route.

Each child partition can comprise one or more virtual processor (230 and 232), and objective operating system (220 and 222) ALARA Principle scheduling thread to perform on these virtual processors.Generally speaking, the virtual processor executable instruction that is to provide the expression of the concurrent physical processor with certain architectures and the status information that is associated.Such as, virtual machine can have the virtual processor with Intel x86 processor characteristic, and another virtual processor can have the characteristic of PowerPC processor.Virtual processor in this example can be mapped to the processor of computer system, makes the instruction realizing virtual processor to obtain the support of processor.Thus, in the embodiment comprising multiple processor, virtual processor can be performed by processor simultaneously, such as other processor executive system supervisory routine instructions simultaneously.In subregion, the combination of virtual processor and storer can be considered to virtual machine.

Visitor's operating system (220 and 222) can be any operating system, as from micro- apple to increase income the operating system of community etc.Visitor's operating system can comprise user/kernel operations pattern, and can have the kernel that can comprise scheduler, memory manager etc.Generally speaking, kernel mode can comprise the execution pattern in processor, and this execution pattern is authorized at least to the access of franchise processor instruction.Each objective operating system can have the file system be associated, and this file system stores application program and the objective operating systems itself such as such as terminal server, e-commerce server, e-mail server.Visitor's operating system schedulable thread performs on virtual processor, and can realize the example of this type of application program.

With reference now to Fig. 3, show the replacement virtual platform in above-mentioned Fig. 2.Fig. 3 depicts the assembly similar with the assembly of Fig. 2; But, in the illustrated embodiment, system supervisor 302 can comprise micro-kernel assembly and the assembly similar with the assembly (as Virtual Service supplier 228 and device driver 224) in the main frame 204 of Fig. 2, and MOS 304 can comprise such as the configuration utility of configuration-system supervisory routine 302.In the architecture, system supervisor 302 can perform the same or analogous function of hypervisor microkernel 202 with Fig. 2; But in the architecture, the objective operating system that system supervisor 304 can be configured to performing in child partition provides resource.The system supervisor 302 of Fig. 3 can be independently software product, operating system a part, be embedded in the firmware of mainboard, or a part for system supervisor 302 can be realized by special IC.

With reference now to Fig. 4, it illustrates the high level block diagram of virtual desktop server 400.In one embodiment, virtual desktop server 400 can be configured to the virtual desktop session (VDS) being deployed to client computer (such as, the mobile device of such as smart phone), the computer system with the assembly similar with assembly shown in Fig. 1 etc.In brief, virtual desktop technology makes user can be remotely mutual with the objective operating system run in virtual machine.Be different from remote desktop session, in virtual desktop session, only user to sign in in objective operating system and this user can have overhead control to it, and such as, this user can run as keeper and on passenger plane, have authority completely.In the example shown, virtual desktop server 400 can have the assembly similar with the computer system 200 or 300 of Fig. 2 and Fig. 3.The logical abstraction of the virtualization infrastructure assembly that virtual platform 402 is described in figs. 2 and 3 above being.Be described as be in following chapters and sections virtual platform 402 " within " function can realize in the one or more elements described in Fig. 2 and Fig. 3.Such as, by paragraph below in greater detail 3D graphical services manager 404 can realize in the main frame 204 of Fig. 2.In one more specifically example, realize in the host operating system that 3D graphical services manager 404 can run in father's subregion.

Start virtual desktop Dialog request and instantiation is carried out to the objective operating system in virtual machine.In the exemplary embodiment, virtual desktop manager (module of such as processor executable) can start virtual machine 414 (it can guide objective operating system 428) in response to request.Virtual desktop manager can perform on logic processor, and virtual platform 402 (such as micro-kernel system supervisor 202) can be indicated to be subregion allocate memory.Virtual platform 402 can perform and virtual unit in the storer of configuring virtual machine 414, and boot loader program is loaded in the storer distributing to VM 414.Boot loader program can perform on virtual processor (itself and then can run on a processor), and objective operating system 428 can be carried in virtual machine 414.Session manager 408 can be loaded by objective operating system 428, and it can load the environment subsystem of subsystem 426 and so on when such as running, and described environment subsystem can comprise the kernel mode portion of such as operating system core 410 and so on.In one embodiment, environment subsystem can be configured to the subset of service is showed application program and provides accessing points to kernel 420.When objective operating system 428 is loaded, boot loader program can exit, and hands to objective operating system 428 by the control of virtual machine 414.Visitor's operating system 428 can perform the modules shown in Fig. 4, and himself is configured to the session of host virtual desktop.Such as, objective operating system 428 can comprise cause long-range present engine 406, session manager 408 etc. guide when start registry value.

After objective operating system 428 is run sometime, it can receive connection request from client computer.First this connection request imported into can be processed by the long-range engine 406 that presents, and this remote rendering engine 406 can be configured to monitor connection message, and when it receives a message, it can produce stack example.The long-range engine 406 that presents can the protocol stack example of Run Sessions, and the 3D graphic user interface (being described in more detail in the following paragraphs) played up by virtual platform 402 can be remotely displayed subsystem 418 receives and send to client computer by protocol stack example.Generally speaking, protocol stack example can be configured to user interface to export and is routed to the client computer that is associated and the user received from the client computer be associated input is routed to operating system kernel 410.In brief, operating system kernel 410 can be configured to management screen output: collect the input from keyboard, mouse and miscellaneous equipment.

User's voucher (such as user name/password combination) can be received by the long-range engine 406 that presents and is passed to session manager 408.Voucher can be passed to login process by session manager 408, and voucher can be routed to authentication subsystem 424 to verify by login process.Authentication subsystem 424 can determine that user's voucher is effective and can starts virtual desktop session, and that is user can log on objective operating system 428.

The all right generation system token of authentication subsystem 424, system token can use when attempting execution one process whenever user to determine whether user has the security credence running this process or thread.Such as, when process or thread are attempted obtaining access (such as, open, close, delete and/or the object of modification as file, setting or application program), thread or process can by secure subsystem 422 certifications.Secure subsystem 422 can contrast the Access Control List (ACL) be associated with object and carry out check system token, and determines whether thread has license based on the information in system token with comparing of Access Control List (ACL).If secure subsystem 422 determines that thread is authorized to, then allow this thread accesses object.

Continue the description to Fig. 4, in one embodiment, operating system kernel 410 can comprise graphic display interface 416 (GDI) and input subsystem 412.In an example embodiment, input subsystem 412 can be configured to receive user's input via the protocol stack example of virtual desktop session from client computer, and this input is sent to operating system core 410.In some embodiments, user's input can comprise the signal indicating absolute and/or relative mouse motion command, mouse coordinates, mouse click, keyboard signal, control lever movement signal etc.Such as can be received by operating system core 410 user's input of the double mouse click of icon and so on, and input subsystem 412 can be configured to determine that icon is positioned at the coordinate place be associated with this double-click.When input subsystem 412 can be configured to the operation to the process that can perform the application be associated with this icon subsequently, subsystem 426 sends and notifies.

Forward Fig. 5 to, it illustrates and can be used to configuration virtual desktop server 400 to play up the 3D figure of virtual machine (414,414B and/or 414C), and load balance is carried out to the virtual machine in the Graphics Processing Unit of such as 3D GPU 504,504B and/or 504C and so on.In this illustrative embodiment, virtual desktop server 400 and then the client computer stream to such as client computer 520,520B and/or 520C and so on transmit the image of instruction three-dimensional graphic user interface.In brief, each client computer can be associated with running the virtual machine (414,414B and/or 414C) being configured to the objective operating system of carrying out virtual desktop session.Client computer 520,520B and/or 520C can comprise computer system, sports equipment and/or the thin client with the assembly being similar to those assemblies shown in Fig. 1.Such as, thin client can have commercial hardware and one chip web browser, and this web browser is configured to manage this hardware, user's input and output and be connected to the Internet.In this example, thin client also can comprise the such as user interface 222 of display and so on and the user input device of such as mouse and so on.

Virtual desktop server 400 also can be configured to play up 3D figure when it is activated.Such as, when virtual desktop server 400 is run, the example of the 3D graphical services manager of such as 3D graphical services manager 404 and so on can be activated.3D graphical services manager 404 (it is described in more detail in the following paragraphs) can perform on a processor, and virtual desktop session connection is imported in wait into.In brief, 3D graphical services manager 404 can be configured to by determining each virtual machine to be assigned to where and by carrying out mobile virtual machine based on performance information, to carry out load balance to the virtual machine in Graphics Processing Unit.

Before receiving the request of serving the 3D starting virtual machine, 3D graphical services manager 404 is configured to inquire each Graphics Processing Unit in system, to identify its feature and will represent that the information of its feature is stored in table.Characteristic information can be used to balance the use of Graphics Processing Unit subsequently.In the exemplary embodiment, the feature of Graphics Processing Unit can comprise one group of ability, i.e. hardware capability, and such as support surface and texture are shared or supported the ability of minimal version of 3D graph API (API).In addition, the feature of Graphics Processing Unit can comprise the quantity of the graphics random access memory (figure RAM) that the hwid of Graphics Processing Unit, attribute list and 3D Graphics Processing Unit control, that is, the storer in the adapter holding 3D Graphics Processing Unit is embedded in.Such as, graphics random access memory can be used to the bitmap and Z-buffer (depth coordinate in its management 3D figure), texture, vertex buffer and the compiling coloration program that store screen picture.In the exemplary embodiment, graphic memory can be high speed or multiport memory, the such as random access memory of video-ram, dynamic RAM or double data rate (DDR) (DDR) technology based on such as DDR2 and D graphics DR (GDDR3, GDDR4 and/or GDDR5) and so on.

When receiving virtual desktop session connection; When starting graph rendering module (being described in more detail in the following paragraphs) and send to 3D-GPU ISP 512 signal be loaded into by 3D assembly in objective operating system 428,3D graphical services manager 404 can be notified.In brief, 3D graphical services manager 404 can be the example that each virtual machine loads graph rendering module; Determine graph rendering module will run on which GPU; And the device identifier of determined Graphics Processing Unit is transmitted to graph rendering module.Graph rendering module can be tied to 3D GPU subsequently, and represents its virtual machine be associated and play up to it.In illustrated example, graph rendering module 506 corresponds to virtual machine 414.Fixed-point data, texture etc. for creating the graphic user interface of virtual machine 414 can be fed to graph rendering module 506 from virtual machine 414, and this graph rendering module can be virtualized platform 402 and treat as the process generating graphic user interface.Because most of 3D Graphics Processing Unit is designed to generate several graphic user interface to several process at every turn, therefore those skilled in the art can understand, and in the situation that a large amount of graph rendering module is just running wherein, GPU can become and more easily transships and reset.

Continue initialization procedure, 3D-GPU ISP 512 can cause 3D graphical services client computer 514 to disperse (spawn) in objective operating system 428.Such as, the storage address of the virtual mainboard in virtual machine 414 can have the intercepting set to some address being mapped to I/O device.When objective operating system 428 starts, plug and play module can perform and can query mappings to the storage address of IO.The supervisory routine of the supervisory routine 202 of such as Fig. 2 and so on can intercept and to read and with causing the device identifier of plug and play module loading 3D graphical services client computer 514 to respond.3D graphical services client computer 514 is by being mapped as one or more pages of storer shared and passing to 3D-GPU ISP 512 set up communication channel to 3D-GPU ISP 512 by mapping.In the page of storer, message-transmission communication channel 516 can be set up.In essence, message-transmission communication channel 516 is that in storer, 3D GPU ISP/client computer can write message to it and read the shared region of message from it.

3D graphical services client computer 514 also can cause virtual device driver 518 to disperse.Virtual device driver 518 also can be set up and share graphic aperture 526 between virtual machine 514 and virtual platform 402.In the exemplary embodiment, virtual machine driver 518 can distribute one group of visitor's storage page, and sends the signal comprising the address of connection request and storage page to 3D graphical services client computer 514.Graph rendering module 506 can receive the address of connection request and storage page.Graph rendering module 506 can send to supervisory routine the signal that this supervisory routine of instruction creates graphic aperture 526 in this group storage page.These pages can be mapped to system physical address by supervisory routine, and the graphic aperture 526 in the scope of execute store.In this example, virtual device driver 518 can be configured to dma buffer to write objective memory address range.When data are written into objective memory address range, dma buffer is routed to graph rendering module 506 by graphic aperture 526.

In the exemplary embodiment, virtual device driver 518 can show as the device driver of the Graphics Processing Unit of 3D ability for objective operating system 428, objective operating system 428 is caused it oneself to be configured to support 3D figure thus, such as, by the example of loading application programs interface 524 (API).Virtual device driver 518 is configured to and API 524 interface, and API 524 makes 3D application program 528 can produce 3D figure.3D application program 528---user interface of such as operating system figure user interface, application program/video-game etc.---can send instruction to API 524, and this API 524 can be such as the API of DirectX and so on.In brief, 3D figure API 524 provides the level of abstraction between the graphics application program of such as video-game and so on and driver (virtual device driver 518 in this case).On the one hand, API 524 provides the LLI with the Graphics Processing Unit interface shown by virtual device driver 518, on the other hand its provide can by the 3D graph command storehouse of application call.3D graph command storehouse can be mapped to the interface shown by virtual device driver 518 by API 524, makes game developer need not understand the characteristic of each graphics driver thus.

In operation, API 524 can generate the pel of such as basic geometric configuration and so on, this pel is used as the structure block of other shapes being expressed as summit and constant in computer graphical, and these fixed points are stored in multiple direct memory access (DMA) (DMA) impact damper by API 524.When API 524 writes impact damper, virtual device driver 518 can reformat the data be stored in dma buffer; They are bundled to one or more GPU token; And via graphic aperture 526, GPU token is sent to graph rendering module 506.Similarly, when API 524 gives an order, it can be inserted in dma buffer and in token, to be transmitted to graph rendering module 506.

Graph rendering module 506 can receive token; Extract dma buffer; And they are stored in the page of the storer be associated with virtual platform 402.In the exemplary embodiment, order in dma buffer and pel can be translated into API structure by graph rendering module 506, and this API structure can be processed by 3D graphics driver 510 and give an order together with the address of the dma buffer in virtual platform 402 to graphic kernel 508.

The graphic kernel 508 that can be configured to the execution of dispatching in the Graphics Processing Unit in virtual desktop server 400 can receive the address of instruction and dma buffer, and determines when to give an order to 3D graphics driver 510.When the pel from each dma buffer will be played up, graphic kernel 508 can send rendering order to 3D graphics driver 510, this 3D graphics driver 510 can perform on a processor and can Instruction Graphics processing unit 504 fill order and process dma buffer in data.

Graphics Processing Unit 504 can perform and generate the bitmap of the picture frame represented in storer, such as array of pixel values.Graph rendering module 506 can captured bitmap this bitmap is passed to compression module, and passes to via graphic aperture 526 subsequently and long-rangely present engine 406.Bitmap can be sent to client computer 520 via one or more information block by the long-range engine 406 that presents.

In some instances, the data stream sent by one or more graph rendering module can cause Graphics Processing Unit to be reset.Such as, transship in the situation of---namely GPU must play up too much image---in Graphics Processing Unit, it understands time-out and GPU will be reset.This causes 3D graphics driver 510 to reset, itself so that cause graphic kernel 508 to reset the connection of so far GPU, this causes any graphic package module stopping being tied to GPU.

In the exemplary embodiment, 3D graphical services manager 404 can be configured to balance the load in the Graphics Processing Unit in virtual desktop server 400, to reduce the chance resetting and occur.Such as, and with reference to figure 6, it illustrates another high level block diagram of the virtual desktop server 400 comprising 5 virtual machines (414-414F, wherein virtual machine 414 can be considered to virtual machine 414A in this example).What each virtual machine can have being associated of running in the virtual platform 402 (506-506F) plays up process.As shown in drawings, each graph rendering module (506-506F) can be bound, such as, be attached to Graphics Processing Unit.Such as, virtual machine 414,414D and 414E are bound to 3D GPU 504, and virtual machine 414B is bound to 3D GPU 504B, and virtual machine 414C and 414F is bound to 3D GPU 504C.Operationally period, each virtual machine can be moved by 3D graphical services manager 404, and the dotted line in illustrated example shows virtual machine 414 is moved to 3D GPU 504B from 3D GPU 504.

3D graphical services manager 404 can be configured to use round-robin algorithm and/or dynamic Placement to balance GPU.Such as, round-robin algorithm can be used to find Graphics Processing Unit and assign this Graphics Processing Unit with the information from virtual machine render graphics for determining whether.If information indicating graphic processing unit transships, then round-robin algorithm can be located next GPU and be performed same operation.In addition, or alternatively, 3D graphical services manager 404 can be configured to the Graphics Processing Unit using this information to select the figure playing up self virtualizing machine, and checks each discontinuously.

In the exemplary embodiment, for assigning the information of virtual machine can comprise the estimated amount that can be 3D Graphics Processing Unit graphic memory used to the GPU of such as 3D GPU 504 and so on.Whether the amount of estimated available graphic memory can be used to prediction GPU by time-out, and whether connection will be reset.In this example, 3D graphical services manager 404 can be configured to the amount of the useful graphic memory estimated by use to arrange threshold value.When 3D graphical services manager 404 receives the request of assigning virtual machine, 3D graphical services manager 404 can estimate that virtual machine is by the amount of attempting the graphic memory used or the amount be used at least enough needed for virtual machine rendering image, and by estimated amount and threshold value (such as, estimated by can be the amount of Graphics Processing Unit graphic memory used) make comparisons, and when the amount of the estimated graphic memory used by virtual machine is less than threshold value, assign image rendering process to GPU.

In particular example, assuming that 3D Graphics Processing Unit 504 controls the impact damper of 4 GB, 3D graphical services manager 404 estimates that virtual machine 414E is by the storer of use 1 GB, threshold value is set to the amount of the estimated available memory controlled by GPU, the amount of the estimated graphic memory used by virtual machine 414 is 1 GB, and the amount of the estimated graphic memory used by virtual machine 414D is 512 megabyte.In this example, the estimated amount (1 GB) being used for virtual machine 414E and estimated available quantity (roughly 2.5 GB) can be made comparisons by 3D graphical services manager 404, and determine that GPU can operate virtual machine.Virtual machine can be assigned to GPU by 3D graphical services manager 404; Deduct the estimated amount for virtual machine 414E, to obtain the estimated amount (roughly 1.5 GB) that can be used for GPU, and value is stored in table.

In the exemplary embodiment, threshold value can be set to the amount of the estimated available graphic memory controlled by GPU.In another embodiment, threshold value is by monitoring Graphics Processing Unit in time and determining have when Graphics Processing Unit resets how many estimated available graphic memory to can be used to arrange threshold value.Such as, can calculate according to performance data the probability that Graphics Processing Unit will be reset, and it is associated with estimated available graphic memory.In example embodiment, the amount of the available graphic memory estimated by this can be set to threshold value.In other embodiments, the amount of the available graphic memory estimated by this can be multiplied by scalar value, and can threshold value be used as through the amount of the available graphic memory estimated by convergent-divergent.Such as, 80% of the amount of estimated available graphic memory can be set to threshold value.

In the embodiment of demonstration, the amount of the spendable graphic memory of virtual machine estimated by 3D graphical services manager 404 based on information, this information including, but not limited to: Pixel Dimensions, color depth, quantity for the display of virtual machine, handling capacity, ratio of compression, estimated by the amount on the surface of playing up for virtual machine, store the storer needed for the pel that generates needed for summit amount, store storer needed for the texture being applied to summit, to object application choice anti-aliasing effect needed for storer etc.In response to the instruction receiving virtual machine activation, 3D graphical services manager 404 can be configured to estimate that virtual machine is by the amount of the graphic memory of use.In this example, can be stored in the configuration file be associated with virtual machine for some or all estimating the information of the amount of used graphic memory.In another embodiment, the amount of the graphic memory that estimated virtual machine operationally uses can be monitored, and is used by 3D graphical services manager 404, and is stored in configuration file and uses for during follow-up virtual desktop session.

In addition to the foregoing, in the exemplary embodiment, other information can be used to assign virtual machine to available GPU.Such as, whether the quantity being assigned to the virtual machine of GPU can be used to prediction GPU and transship.The quantity of the virtual machine being assigned to GPU can be monitored in time, and when Graphics Processing Unit resets, the quantity of the virtual machine bound can be recorded.The value reflecting following content can be comprised in table: attached 3 virtual machines of 3D GPU 504, attached 1 virtual machine of 3D GPU 504B and attached two virtual machines of 3D GPU 504C referring to Fig. 6,3D graphical services manager 404.If 3D Graphics Processing Unit resets, then 3D graphical services manager 404 can determine have how many virtual machines to be attached when it collapses according to table.The probability that dissimilar Graphics Processing Unit will be reset can be calculated according to the performance data caught in time, and its amount with estimated attached virtual machine is associated.3D graphical services manager 404 also uses the amount of estimated graphic memory in the illustrative embodiments selecting GPU wherein, and the amount of estimated available graphic memory can have the weight larger than the quantity of attached virtual machine.

In identical or another illustrative embodiments, 3D graphical services manager 404 is used for determining whether that assigning the information of virtual machine can comprise description Graphics Processing Unit to GPU is used for ordering the spent average length of time that responds, the i.e. stand-by period, can be used to determine whether Graphics Processing Unit has pressure.In one embodiment, the average latency can be stored in table by 3D graphical services manager 404, and when collapse, the average latency of order and threshold value can make comparisons by 3D graphical services manager 404.If the average latency is less than threshold value, then 3D graphical services manager 404 can be configured to assign virtual machine to GPU.

Threshold value can be arranged by keeper etc. and can depend on the Graphics Processing Unit of use.In the exemplary embodiment, the experiment dissimilar GPU loading being increased progressively to the virtual machine of quantity can be performed.Along with when more multi-dummy machine is added, the stand-by period can be recorded and can calculate the probability of replacement by the amount of replacement over a time period being made comparisons with the amount of the replacement in the same time section when the stand-by period is lower.What monitored resets with height the stand-by period that probability correlation joins and can be used to arrange threshold value.In example embodiment, the stand-by period monitored can be set to threshold value.In other exemplary implementations, the stand-by period monitored can be multiplied by scalar value, and can use the stand-by period through convergent-divergent.Such as, 80% of the stand-by period monitored can be set to threshold value.

In the exemplary embodiment, the stand-by period can be measured until receive the confirmation (" ACK ") from graphic kernel 508 to the time that GPU gives an order.After receiving each ACK, graphic kernel 508 can send to 3D graphical services manager 404 by with asking the stand-by period be associated.3D graphical services manager 404 can reflect the value of the average latency of this GPU in updating form.

In another example embodiment, can rise self-virtualizing device driver 518 time of starting direct memory access (DMA) affairs and measure the stand-by period.Such as, API 524 can write pel to dma buffer, and it can be caught by virtual device driver 518 and be sent to graph rendering module 506.API 524 can finally give an order.When virtual device driver 518 receives order, it can start timer.This order can be sent to graph rendering module 506 by virtual device driver 518, graph rendering module 506 and then will send order to graphic kernel 508.Graphic kernel 508 can send order to 3D graphics driver 510, and it can cause 3D Graphics Processing Unit 504 to process pel in dma buffer.3D GPU 504 can generate ACK, and situation can be propagated back to virtual machine driver 518, and virtual device driver 518 can stop timer.Virtual device driver 518 subsequently can via message-send communication channel 516 to send to 3D graphical services manager 404 stand-by period.3D graphical services manager 404 can reflect the value of the average latency of virtual machine in updating form.

Such as, virtual device driver 518 can send to 3D graphical services client computer 514 message comprising the stand-by period of affairs.3D GPU ISP 512 can receive this message and this message is sent to 3D graphical services manager 404, the renewable table of the latter.In this example, the stand-by period of affairs can be used to average latency of the DMA affairs determining to be sent by virtual machine 414, and whether the resource of this reflection 3D GPU 504 is by excessive use.

In the exemplary embodiment, except carrying out except load balance when virtual machine disperses to them, in response to conditions present, 3D graphical services manager 404 can be configured to more multi-dummy machine to move to another (by again binding its graphics associated rendering module) from a 3DGPU.Such as, Fig. 6 shows by the graph rendering module 506 moving to 3D GPU 504B from 3D GPU 504.In this exemplary embodiment, when contextual processing occurs to run virtual platform 402, when detecting that 3D graphical services manager 404 can run to when starting the request of virtual machine and/or after predetermined time amount is expired and generated interruption.

In this exemplary embodiment, 3D graphical services manager 404 can be configured to determine whether any GPU transships, such as, whether be in replacement risk, and whether mobile virtual machine (risk of overload GPU can be reduced in migration operation and can not unacceptably increase in the situation of the risk causing other GPU to reset).Such as, 3D graphical services manager 404 can be configured to determine the virtual machine being bound to each 3D GPU quantity, for each 3D GPU can the amount of graphic memory, average latency of DMA affairs of being initiated by each virtual machine being tied to each 3D GPU, each graph rendering module use average latency of the number percent of 3D GPU, the handling capacity be associated with each graph rendering module and/or graphic kernel order.In the situation of this information instruction 3D GPU overload, under 3D graphical services manager 404 can be configured to represent which virtual machine is in maximum pressure.3D graphical services manager 404 can assign the virtual machine having pressure subsequently to another Graphics Processing Unit.Appointment part in this operating process is similar to appointment process described above.

In the exemplary embodiment, 3D graphical services manager 404, by sending the device identifier of movement directive together with new GPU to graph rendering module (such as, graph rendering module 506), initiates mobile operation.In this example, graph rendering module 506 can receive order and find the GPU be associated with new identifier, such as, and 3D GPU 504B.In this example, graph rendering module can be configured to copy any kernel; Be tied to 3D GPU 504B, and dismounting is to the mapping of 3D GPU 504.After bindings completes, graph rendering module can there occurs movement to 3D graphical services manager 404 notice, and 3D graphical services manager 404 its table renewable is to reflect that graph rendering module 506 is attached to 3DGPU 504B.

Referring to Fig. 7, in the exemplary embodiment, mobile operation can be used to virtual machine (vm) migration to different virtual desktop server (400B-400D).As shown in the drawings, each virtual desktop server (400-400D) can comprise the example of one group of one or more Graphics Processing Unit (704-704D) and 3D graphical services manager (404-404D).Also show the management system 702 that can be configured to into control data center 700.Management system 702 optionally can comprise main 3D graphical services manager (being considered to optional to indicate it with dotted line instruction), and this main 3D graphical services manager can control the every other 3D graphical services manager (404-404D) in data center 700.

The unique information identifying each virtual machine can be used to affect migration operation.Such as, when each virtual machine is activated, start the unique identifier that the 3D graphical services manager (such as, 3D graphical services manager 404) that the server of VM runs can create the virtual machine in data center wherein.Unique identifier can comprise the type of the Graphics Processing Unit that virtual machine is played up thereon, the amount etc. of its estimated graphic memory used.In example embodiment, if virtual machine 414 has pressure, and not having suitable GPU to use on the server, then 3D graphical services manager 404 to each virtual desktop server in data center or optionally can send unique identifier to management system 702.

Unique identifier is sent in the example of each virtual desktop server wherein, and whether graphical services manager (404-404D) can be assigned to Graphics Processing Unit based on the information determination virtual machine described in previous paragraph.Determine that it can operate in the situation of virtual machine 414 at graphical services manager (such as, graphical services manager 404C), virtual desktop server 400 can be configured to virtual machine 414 to move to virtual desktop server 400C.

Unique identifier is sent in the situation of main 3D graphical services manager 706 wherein, and main 3D graphical services manager 706 can be configured to comprise all properties information from each 3D graphical services manager (404-404D) in data center 700; And main 3D graphical services manager 706 can determine based on the information described in previous paragraph whether virtual machine can be assigned to any Graphics Processing Unit in data center 700.

Below a series of process flow diagrams describing operating process.For ease of understanding, process flow diagram is organized into and makes initial flowchart viewpoint demonstrates each realization by overall " scheming greatly ", and follow-up process flow diagram provides further condiment shown in broken lines and/or details.In addition, those skilled in the art will appreciate that the operating process that dotted line is described is considered to optional.

Referring to Fig. 8, it illustrates for using the operating process balanced to the graphic process unit between virtual machine, comprise operation 800,802,804 and 806.The beginning of operation 800 instruction operating processes, and operate 802 and illustrate that appointment one group of virtual machine plays up 3D figure in a 3D Graphics Processing Unit.Such as, and with reference to figure 6, in the exemplary embodiment, 3D graphical services manager 400 can assign one group of virtual machine to play up in Graphics Processing Unit, such as, virtual machine 414, virtual machine 414D and virtual machine 414E is assigned to play up 3D figure in 3D Graphics Processing Unit 504.In this particular example, bot 502 includes, each virtual machine (414-414F) can be associated with graph rendering module (506,506D and 506), these graph rendering modules perform in virtual platform 402, such as perform in the main frame 204 of Fig. 2, these graph rendering modules can call via application programming interfaces and be tied to 3D Graphics Processing Unit 504.In this example, graph rendering module (506,506D and 506) can be configured to collect the dma buffer comprising pel and the API structure representing order; Optionally, by order that command translation becomes to be disposed by 3D graphics driver 510; And these orders are issued graphic kernel 508 together with the page of the storer comprising dma buffer.

Continue the description of Fig. 8, operation 804 illustrates that being at least used for carrying out response institute's time spent amount to order based on a 3D Graphics Processing Unit determines that a 3D Graphics Processing Unit is overused.Such as, in the exemplary embodiment, 3D graphical services manager 404 can perform on a processor, and determine that 3D Graphics Processing Unit 504 is overused, such as, 3D Graphics Processing Unit 504 does not have the enough resources for serving its requests all, and can be reset, and causes graph rendering module (506,506D and 506) to stop thus.In this particular example, bot 502 includes, determine at least to be used for replying the average length of time spent of graphic kernel 508 with ACK by graphic based processing unit (such as 3D GPU504).In this example, can be used for hardware the stand-by period that the order of response pattern kernel spends and upgrade the table controlled by 3D graphical services manager 404.Mean value and threshold wait times value can be made comparisons, and if mean value is greater than threshold value, then 3D graphical services manager 404 can be configured to determine that 3D GPU 504 is overused.

Continue the description of Fig. 8, operation 806 illustrates and at least in response to the judgement be overused a 3D Graphics Processing Unit, the first virtual machine is moved to second graph processing unit from this group virtual machine.Such as, and again referring to Fig. 6, in the exemplary embodiment, 3D graphical services manager 404 can be configured to selection first virtual machine (such as, virtual machine 414) and moves to second graph processing unit (such as, 3D GPU504B).In this example, virtual machine 414 is moved to 3D GPU 504B to relate to graph rendering module 506 is moved to 3D GPU 504B from 3D GPU 504.Such as, movement directive can be sent to graph rendering module 506 together with the device identifier of 3D GPU 405B by 3D graphical services manager 404.Graph rendering module 506 can receive order; Copy internal buffer; And be tied to 3D GPU 504B.After bindings completes, settling signal detachably to the mapping of 3D GPU 504, and is sent to 3D graphical services manager 404 by graph rendering module 506, and the renewable table of this 3D graphical services manager is to reflect that virtual machine 414 is bound to 3D GPU 504B.Further, graph rendering module 506 can send rendering command to 3D GPU 504B, and can be monitored the average latency of order by 3D graphical services manager 404.

Get back to Fig. 9 now, it illustrates the operating process of Fig. 8, comprises additional operations/refinement 908-916.Operation 908 shows in response to determining that the amount of the estimated available graphic memory controlled by second graph processing unit is greater than threshold value, and the first virtual machine is moved to second graph processing unit.Such as, and referring to Fig. 6, in the exemplary embodiment, the virtual machine that 3D graphical services manager 404 can be configured to be bound to 3D GPU 504 (such as, virtual machine 414) moves.In this example, determining that virtual machine 414 is moved to 3D GPU 504B can at least based on the judgement estimated amount that can be 3D GPU 504B graphic memory used being greater than to threshold value, and this threshold value can based on the amount of the graphic memory of the estimated figure for playing up virtual machine 414.In this case, 3D graphical services manager 404 can be configured to carry out mobile virtual machine 414 in response to the judgement of transshipping to 3D GPU504 and to the judgement that 3D GPU 504B can hold virtual machine 414.

Continue the description of Fig. 9, operation 910 illustrates that the average latency based on the direct memory access (DMA) affairs initiated by the first virtual machine determines that a 3D Graphics Processing Unit is by excessively used.In this example, 3D graphical services manager 404 movable virtual machine 414 because the average latency of the DMA affairs be associated with virtual machine 414 be greater than the every other graph rendering module (506D and 506E) being such as tied to 3D GPU 504.In this example, 3D graphical services manager 404 can be configured to the average latency first determining the order sent by graphic kernel 508, and determines that virtual machine 414 is the heaviest virtual machines that are stressed be associated with GPU in the average latency of DMA affairs used subsequently.

Forward operation 912 to, the average latency that it illustrates in response to the direct memory access (DMA) affairs of the second virtual machine determining to be assigned to second graph processing unit is less than threshold value to move the first virtual machine.In the exemplary embodiment, 3D graphical services manager 404 can be configured to be less than threshold value in response to the average latency of the DMA affairs of the one or more virtual machines determining to be tied to 3D GPU504B virtual machine is moved to 3D GPU 504B.Such as, 3D graphical services manager 404 can check the table of the average latency of the DMA affairs comprising virtual machine 414B.Average latency and threshold value can make comparisons by 3D graphical services manager 404, and determine that the average latency is less than threshold value.Load in this instruction virtual machine 414B is lower, and indicates the load on 3D GPU 504B lower in theory.In this case, 3D graphical services manager 404 can be configured at least determine mobile graphics rendering module 506 based on this.

Forwarding operation 914 to, it illustrates in response to determining that the quantity of the virtual machine being assigned to second graph processing unit is less than threshold value to move the first virtual machine.In the exemplary embodiment, 3D graphical services manager 404 can be configured in response to determining that the quantity of the virtual machine being tied to 3D GPU 504B is less than number of thresholds and virtual machine 414 is moved to 3D GPU 504B.In this example, 3D graphical services manager 404 can be configured to the value of the quantity (it is 1 in the example shown) of the virtual machine reading the current 3D of the being bound to GPU 504B of instruction from its table and determines that the quantity of current bound virtual machine is less than number of thresholds (namely based on the quantity of performance and/or management preferences).In this case, 3D graphical services manager 404 can be configured at least determine mobile graphics rendering module 506 based on this.

Forward operation 916 to, it illustrates the feature determining the first Graphics Processing Unit and second graph processing unit.Such as, in one embodiment, 3D graphical services manager 404 can be configured to the feature of each 3D Graphics Processing Unit determined in computer system.Such as, and continue previous example, 3D graphical services manager 404 can be configured to the feature determining 3D Graphics Processing Unit 504,504B and 504C.In this example, these features can be used to determine whether Graphics Processing Unit can be virtual machine process 3D figure.Such as, 3D graphical services manager 404 can be configured to carry out checking to check whether graphics card is supported to share surface and texture, and whether it at least supports such as particular version, amount of the graphic memory be associated with Graphics Processing Unit etc.In this example, this information can be used to fill the table that can be used to the Graphics Processing Unit use balanced between virtual machine.

Referring now to Figure 10, it illustrates for using the operating process balanced to the Graphics Processing Unit between virtual machine, comprise operation 1000,1002,1004,1006 and 1008.Forward operation 1000 to, it starts operating process, and operate 1002 show receive to the request the first virtual machine being assigned to a 3D Graphics Processing Unit.Such as, and referring to Fig. 5, in the exemplary embodiment, 3D graphical services manager 404 can perform and receive the instruction to the first virtual machine (such as, virtual machine 414) being assigned to 3D Graphics Processing Unit (such as, 3D GPU 504).In this example, request can be the signal indicating the client computer of such as client computer 520 and so on to ask virtual desktop session.

Continue the description of Figure 10, operation 1004 shows the amount of the graphic memory estimated for playing up 3D figure for the first virtual machine.Such as, in one embodiment, 3D graphical services manager 404 can estimate that virtual machine 414 can be used to the amount of the graphic memory playing up 3D figure, such as, can at least allow to be enough the amount of the graphic memory of virtual machine render graphics.Such as, estimating can based on the information obtained from the configuration file be associated with virtual machine 414, the average magnitude of its graphic memory indicating such as virtual machine 414 to use at the last time durations that it runs.In another illustrative embodiments, the amount of the estimated graphic memory needed for virtual machine render graphics can be estimate according to the information of the minimum memory tolerance described needed for display image.In another example embodiment, the amount of estimated graphic memory can based on amount of the storer needed for estimated high-speed cache pel etc.

Operation 1006 shows the latency value be associated in response to the direct memory access (DMA) affairs determined with sent by the one group of virtual machine being assigned render graphics in the first Graphics Processing Unit and is less than first threshold and the estimated amount for the graphic memory playing up 3D figure is less than Second Threshold, comes to select a 3D Graphics Processing Unit from multiple 3D Graphics Processing Unit.Such as, 3D graphical services manager 404 can store the information that instruction is bound to the latency value of the DMA affairs of each virtual machine of 3D GPU 504.Such as, when 3D graphical services manager 404 receives when virtual machine 414 is assigned to the request of 3D GPU, virtual machine 414D and 414E can be bound to 3D GPU 504.In this example, 3D graphical services manager 404 can obtain latency value from table, and will be worth waiting time and make comparisons with threshold value.In this example, the estimated amount being used for the graphic memory of the 3D figure playing up virtual machine 414 and threshold value also can be made comparisons by 3D graphical services manager 404.When the average latency of virtual machine 414D and 414E be less than threshold value and the amount of the graphic memory of the estimated 3D figure for playing up virtual machine 414 is less than Second Threshold, 3D graphical services manager 404 can select 3D GPU 504.

In the exemplary embodiment, Second Threshold (that is, with the estimated threshold value be associated for the amount of the graphic memory playing up 3D figure) can based on the amount of the graphic memory of the total amount of the graphic memory controlled by 3D GPU 504 and the estimated 3D figure for playing up virtual machine 414D and 414E.Such as, when virtual machine 414D and 414E is activated, 3D graphical services manager 404 may calculate the amount of the graphic memory of the estimated 3D figure for playing up virtual machine 414D and 414E, and these values is stored in table.When 3D graphical services manager 404 have selected 3D GPU 504 for each in virtual machine 414D and 414E, 3D graphical services manager 404 can deduct from the amount of the estimated available graphic memory controlled by GPU 504 its be associated estimated by the amount of the graphic memory for playing up 3D figure, and this value to be stored in table.This value can be used as Second Threshold subsequently, or Second Threshold can at least in part based on this value.

Operation 1008 illustrates that appointment first virtual machine plays up 3D figure in a 3D Graphics Processing Unit.Such as, and again referring to Fig. 6, in the exemplary embodiment, 3D graphical services manager 404 can be configured to initialize graphic rendering module 506 and send to it order being tied to 3D Graphics Processing Unit 504.Graph rendering module 506 can receive this order and be tied to 3D Graphics Processing Unit 504.After graph rendering module 504 is tied to 3D Graphics Processing Unit 504 and sets up communication channel between virtual machine 414 and graph rendering module 506, graph rendering module 506 can receive the dma buffer on summit, and plays up them on 3D processing unit 504.

Forward Figure 11 to, it illustrates the operating process of Figure 11, comprises additional operations 1110-1118.Operation 1110 shows the amount at least estimating the graphic memory for playing up 3D figure for the first virtual machine according to the information of the Pixel Dimensions of each virtual machine in identified group.Such as, in one embodiment, the amount for playing up the graphic memory of 3D figure for each virtual machine can based on the Pixel Dimensions be associated with virtual machine.Such as, when virtual machine is initialised, configuration file can be received, the information of this configuration file required display resolution comprising such as virtual machine and so on.3D graphical services manager 404 can receive this information and use it to estimate plays up the amount of the graphic memory of 3D figure for virtual machine for this reason.

Such as, Pixel Dimensions describes the quantity of the distinct pixel in each size that can be described by monitor.Because monitor is two-dimensional surface, therefore these sizes only bidimensional used (height and width).In the exemplary embodiment, Pixel Dimensions can be multiplied by the degree of depth and divided by 8.This result describes the amount of the graphic memory being used in rendering image under required display resolution.

Continue the description of Figure 11, operation 1112 shows the information estimation of ratio of compression at least used according to mark the one 3D Graphics Processing Unit for playing up the amount of the graphic memory of 3D figure for the first virtual machine.Such as, in embodiments, ratio of compression (that is, the ratio between the size of compressed image and the image of uncompressed) can be used to the image of estimating that virtual machine will use how many graphic memories to play up the first virtual machine fully.Such as, 3D Graphics Processing Unit can be configured to use compressed texture to reduce its size.When hardware inquired by 3D graphical services manager 404, it can determine whether texture compression can be activated and will reach which type of ratio of compression.3D graphical services manager 404 can use this information to estimate that virtual machine is by the amount of the graphic memory of use.

Referring to operation 1114, which depict at least according to mark estimated by, the information of the quantity on the surface of playing up for the first virtual machine is estimated to be used for be the amount that the first virtual machine plays up the graphic memory of 3D figure.Such as, in one embodiment, the quantity on surface of attempting playing up can be used to estimate the amount of graphic memory that this virtual machine will make for enough playing up 3D figure for virtual machine by estimated virtual machine.

Forwarding operation 1116 to, which depict in response to determining that the stand-by period of at least one order sent by graphic kernel is less than the 3rd threshold value and selects a 3D Graphics Processing Unit from multiple 3D Graphics Processing Unit.Such as, and look back Fig. 6, in the exemplary embodiment, 3D graphical services manager 404 can perform on a processor, and in response to determining that Graphics Processing Unit is (such as, 3D GPU 504) be used for ACK to sent by graphic kernel 508 at least one order make reply institute's time spent be less than threshold value, determine assign virtual machine 414 in 3D Graphics Processing Unit 504, play up 3D figure.In this example, useful hardware is used for the stand-by period that the order of response pattern kernel spends and upgrades the table controlled by 3D graphical services manager 404, and in an illustrative embodiments, can the stand-by period be used, or can be 3D GPU504 calculating mean value.Use wherein in the example of mean value, mean value and threshold wait times value can be made comparisons, and if mean value is greater than threshold value, then 3D graphical services manager 404 can be configured to determine that 3D GPU 504 is overused.

Referring now to operation 1118, it illustrates in response to determining that the quantity of the virtual machine being currently assigned to a 3D Graphics Processing Unit is less than the 4th threshold value and selects a 3D Graphics Processing Unit from multiple 3D Graphics Processing Unit.3D graphical services manager 404 can be configured to select 3D Graphics Processing Unit 504 to come for virtual machine 414 render graphics in response to determining that the data of the virtual machine being tied to 3D Graphics Processing Unit 504 are less than number of thresholds to determine.In this example, 3D graphical services manager 404 can be configured to the value of the quantity (it is 2 in the example shown) of the virtual machine reading the current 3D of the being bound to Graphics Processing Unit 504 of instruction from its table and determines that the quantity of current bound virtual machine is less than number of thresholds (namely based on the quantity of performance and/or management preferences).

With reference now to Figure 12, it illustrates comprise operation 1200,1202,1204,1206 and 1208, operating process.Operation 1200 starts operating processes, and operates 1202 the amount at least estimated for the graphic memory for virtual machine render graphics is shown.Such as, in one embodiment, 3D graphical services manager 404 can sink virtual machine (such as, virtual machine 414) just by initial instruction, and 3D graphical services manager 404 can start operating process to select the Graphics Processing Unit (such as, 3D Graphics Processing Unit 504) that will run.In this example, operating process can comprise the amount of the graphic memory estimated being used to the figure playing up virtual machine 414 fully.Such as, the amount of estimated graphic memory can be used as virtual machine operationally on have multi-memory intensive designator.

The estimation of the aforementioned amount to graphic memory used can based on the minimum memory tolerance needed for display image.In another example embodiment, the amount of estimated graphic memory can based on estimated for storing the amount of the necessary storer of pel generated needed for summit, what color the texture being applied to these summits, these summits should be, for by antialiasing technique storer being applied to object etc.In one example, this estimation can be static, that is, made for arranging based on expection by keeper, or it can be dynamic.Such as, the amount of the graphic memory that virtual machine 414 uses or attempts use can be monitored in time, such as, across virtual desktop session, and estimates that can be used to refinement estimates.

Forward now operation 1204 to, which depict and the amount of the amount of estimated graphic memory and the estimated available graphic memory controlled by multiple 3D Graphics Processing Unit is compared.Such as, 3D graphical services manager 404 can be configured to by estimated by the amount of the graphic memory used by virtual machine 414 and estimated for be attached to virtual desktop server 400 each Graphics Processing Unit can the amount of graphic memory compare.In this example, 3D graphical services manager 404 previously can obtain the amount of graphic memory integrated in graphics card (this graphics card holds Graphics Processing Unit), and this information was stored in table.In addition, each virtual machine is bound to Graphics Processing Unit, and 3D graphical services manager 404 just can be configured to deduct estimated virtual machine by the amount on the graphic memory summit of use from total quantity, and this value is stored in table.At this, the amount of the available graphic memory in the amount of the graphic memory needed for virtual machine 414 and estimated 3D GPU 504,3D GPU 504B and 3D GPU 504C can compare by 3D graphical services manager 404.

Continue the description of Figure 12, operation 1206 shows according to comparing from multiple 3D Graphics Processing Unit selection 3D Graphics Processing Unit for virtual machine plays up 3D figure.Such as, 3D graphical services manager 404 result at least based on the comparison can select the GPU (such as, 3D GPU 506) that will be tied to virtual machine.Such as, and continue exemplified earlier, 3D graphical services manager 404 can be estimated as based on 3D GPU 504 at least in part to be had this fact of enough graphic memories and determines that 3D GPU 504 can main memory virtual machine 414.

Forward now operation 1208 to, it illustrates and assign selected 3D Graphics Processing Unit to play up 3D figure for virtual machine.Such as, and again referring to Fig. 6, in the exemplary embodiment, 3D graphical services manager 404 can be configured to initialize graphic rendering module 506 and send to it order being tied to 3D Graphics Processing Unit 504.Graph rendering module 506 can receive this order and be tied to 3D Graphics Processing Unit 504.

With reference to Figure 13, which depict the operating process of Figure 13, comprise additional operations 1310-1322.In response to the average latency at least based on the direct memory access (DMA) figure affairs initiated by virtual machine, operation 1310 shows to be determined that 3D Graphics Processing Unit is overused virtual machine is moved to second graph processing unit.Such as, and look back Fig. 6, in the exemplary embodiment, 3D graphical services manager 404 can perform on a processor, and determine that 3D Graphics Processing Unit 504 is overused, such as, 3D Graphics Processing Unit 504 does not have the enough resources for serving its requests all, and can be reset, cause graph rendering module (506,612 and 614) to stop thus.In this particular example, bot 502 includes, determining can based on the average latency of the DMA figure affairs initiated by virtual machine 414.In this example, the average latency of virtual machine and threshold value can be made comparisons, and 3D graphical services manager 404 can be configured at least determine that the resource of 3D GPU 504 is overused based on this information.In response to determining, 3D graphical services manager 404 can be configured to graph rendering module 506 to move to another GPU, such as 3D GPU 504B or 504C.

Continue the description of Figure 13, in response to the average latency at least based on the graph command sent to 3D Graphics Processing Unit by graphic kernel, operation 1312 illustrates determines that 3D Graphics Processing Unit is overused, virtual machine is moved to second graph processing unit.Such as, in the exemplary embodiment, describe and can be used to determine whether 3D GPU is overused by the give an order information of spent average length of time of graphic kernel 508 before an ack is received.In this example, the table controlled by 3D graphical services manager 404 can be upgraded with the stand-by period about graphic kernel order, and can be 3D GPU 504 calculating mean value.Mean value and threshold wait times value can be made comparisons, and if mean value is greater than threshold value, then 3D graphical services manager 404 can be configured to determine that 3D GPU 504 is overused.In response to determining, 3D graphical services manager 404 can be configured to graph rendering module 506 to move to another GPU, such as 3D GPU 504B or 504C.

Forward operation 1314 to, it illustrates and determine that the quantity of the virtual machine being currently assigned to selected 3D Graphics Processing Unit is less than threshold value.3D graphical services manager 404 can be configured in response to determining that the quantity of the virtual machine being tied to 3DGPU 504B is less than threshold value and virtual machine 414 is moved to 3D GPU 504B.In this example, 3D graphical services manager 404 can be configured to the value of the quantity (it is 1 in the example shown) of the virtual machine reading the current 3DGPU of the being bound to 504B of instruction from its table and determines that the quantity of current bound virtual machine is less than number of thresholds (namely based on the quantity of performance and/or management preferences).In this case, 3D graphical services manager 404 can be configured at least determine mobile graphics rendering module 506 based on this.

Forward operation 1316 to, it illustrates the hardware characteristics of each 3D Graphics Processing Unit determined in multiple 3D Graphics Processing Unit.Such as, in one embodiment, 3D graphical services manager 404 can be configured to the feature of each 3D Graphics Processing Unit determined in computer system.Such as, and continue previous example, 3D graphical services manager 404 can be configured to the feature determining 3D Graphics Processing Unit 504 and 3D Graphics Processing Unit B616.In this example, these features can be used to determine whether Graphics Processing Unit can be virtual machine process 3D figure.Such as, 3D graphical services manager 404 can be determined to be and carry out checking to check whether graphics card is supported to share surface and texture, and whether it at least supports such as particular version, amount of the graphic memory be associated with Graphics Processing Unit etc.In this example, this information can be used to fill table that controlled by 3D graphical services manager 404, that can be used to the Graphics Processing Unit use balanced between virtual machine.

With reference to operation 1318, it illustrates the amount at least estimating the graphic memory for playing up 3D figure for virtual machine according to the information of the Pixel Dimensions of mark virtual machine.Such as, in one embodiment, the amount for playing up the graphic memory of 3D figure for each virtual machine can based on the Pixel Dimensions be associated with virtual machine.Such as, when virtual machine is initialised, configuration file can be received, the information of this configuration file required display resolution comprising such as virtual machine and so on.3D graphical services manager 404 can receive this information and use it to estimate plays up the amount of the graphic memory of 3D figure for virtual machine for this reason.

Forward operation 1320 to, the information that it illustrates the ratio of compression at least used by selected 3D Graphics Processing Unit according to mark estimates the amount of the graphic memory for playing up 3D figure for virtual machine.Such as, in embodiments, ratio of compression (that is, the ratio between the size of compressed image and the image of uncompressed) can be used to the image of estimating that virtual machine will use how many graphic memories to play up the first virtual machine fully.Such as, a 3D Graphics Processing Unit can be configured to use compressed texture to reduce its size.When hardware inquired by 3D graphical services manager 404, it can determine whether texture compression can be activated and will reach which type of ratio of compression.3D graphical services manager 404 can use this information to estimate that virtual machine is by the amount of the graphic memory of use.

Operation 1322 shows at least estimated by mark, to estimate that the information of the quantity on the surface of playing up for the first virtual machine is used for the graphic memory playing up 3D figure for virtual machine amount.Such as, in one embodiment, the quantity on surface of attempting playing up can be used to estimate the amount of graphic memory that this virtual machine will make for enough playing up 3D figure for virtual machine by estimated virtual machine.Such as, surface is the mode of the object represented in 3D figure, and is stored in a buffer.

Aforesaid detailed description has set forth each embodiment of system and/or process by example and/or application drawing.In the scope that such block diagram and/or example comprise one or more function and/or operation, it will be appreciated by those skilled in the art that, such block diagram, or each function in example and/or operation can respectively and/or commonly by far-ranging hardware, software, firmware or almost its any combination realize.

Although illustrate and describe the particular aspects of described herein theme, but, what it will be apparent to those skilled in the art that is, based on principle herein, when do not depart from theme described herein and its widely in, can make change and amendment, therefore, appended claims will be included in their scope, and all such changes and amendment are all by the real spirit and scope of theme described herein.

Claims (22)

1. the graphic process unit between being configured to virtual machine uses the method balanced, and comprising:

One group of virtual machine is assigned to play up 3D figure in a 3D Graphics Processing Unit;

At least be used for that response institute's time spent amount is carried out to order based on a described 3D Graphics Processing Unit and determine that a described 3D Graphics Processing Unit is overused; And

At least in response to the judgement be overused a described 3D Graphics Processing Unit, the first virtual machine is moved to the 2nd 3D Graphics Processing Unit in same computer system from this group virtual machine.

2. the method for claim 1, is characterized in that, the first virtual machine is moved to the 2nd 3D Graphics Processing Unit and also comprises:

In response to determining that the amount of the estimated available graphic memory controlled by described 2nd 3D Graphics Processing Unit is greater than threshold value, described first virtual machine is moved to described 2nd 3D Graphics Processing Unit.

3. the method for claim 1, is characterized in that, determines that a described 3D Graphics Processing Unit is overused and also comprises:

Average latency based on the direct memory access (DMA) affairs initiated by described first virtual machine determines that a described 3D Graphics Processing Unit is overused.

4. the method for claim 1, is characterized in that, the first virtual machine is moved to the 2nd 3D Graphics Processing Unit and also comprises:

Average latency in response to the direct memory access (DMA) affairs determining the second virtual machine being assigned to described 2nd 3D Graphics Processing Unit is less than threshold value to move described first virtual machine.

5. the method for claim 1, is characterized in that, the first virtual machine is moved to the 2nd 3D Graphics Processing Unit and also comprises:

In response to determining that the quantity of the virtual machine being assigned to described 2nd 3D Graphics Processing Unit is less than threshold value to move described first virtual machine.

6. the method for claim 1, is characterized in that, also comprises:

Determine the feature of a described 3D Graphics Processing Unit and described 2nd 3D Graphics Processing Unit.

7. the graphic process unit between being configured to virtual machine uses the system balanced, and comprising:

For the device of assigning one group of virtual machine to play up 3D figure in a 3D Graphics Processing Unit;

The device that a described 3D Graphics Processing Unit is overused is determined for being at least used for carrying out response institute's time spent amount to order based on a described 3D Graphics Processing Unit; And

For at least in response to the judgement be overused a described 3D Graphics Processing Unit, the first virtual machine is moved to the device of the 2nd 3D Graphics Processing Unit in same computer system from this group virtual machine.

8. system as claimed in claim 7, is characterized in that, also comprise for the device the first virtual machine being moved to the 2nd 3D Graphics Processing Unit:

For in response to determining that the amount of the estimated available graphic memory controlled by described 2nd 3D Graphics Processing Unit is greater than threshold value, described first virtual machine is moved to the device of described 2nd 3D Graphics Processing Unit.

9. system as claimed in claim 7, is characterized in that, for determining that the device that a described 3D Graphics Processing Unit is overused also comprises:

For determining based on the average latency of the direct memory access (DMA) affairs initiated by described first virtual machine the device that a described 3D Graphics Processing Unit is overused.

10. system as claimed in claim 7, is characterized in that, also comprise for the device the first virtual machine being moved to the 2nd 3D Graphics Processing Unit:

For being less than threshold value to move the device of described first virtual machine in response to the average latency of the direct memory access (DMA) affairs determining the second virtual machine being assigned to described 2nd 3D Graphics Processing Unit.

11. systems as claimed in claim 7, is characterized in that, also comprise for the device the first virtual machine being moved to the 2nd 3D Graphics Processing Unit:

For in response to determining that the quantity of the virtual machine being assigned to described 2nd 3D Graphics Processing Unit is less than threshold value to move the device of described first virtual machine.

12. systems as claimed in claim 7, is characterized in that, also comprise:

For determining the device of the feature of a described 3D Graphics Processing Unit and described 2nd 3D Graphics Processing Unit.

13. 1 kinds for using the equipment balanced to the Graphics Processing Unit between virtual machine, described equipment comprises:

For receiving the device of the request the first virtual machine being assigned to a 3D Graphics Processing Unit;

For estimating the device for playing up the amount of the graphic memory of 3D figure for described first virtual machine;

Latency value for being associated in response to the direct memory access (DMA) affairs determined with sent by the one group of virtual machine being assigned render graphics in a described 3D Graphics Processing Unit is less than first threshold and the estimated amount for the graphic memory playing up 3D figure is less than Second Threshold, selects the device of a described 3D Graphics Processing Unit from the multiple 3D Graphics Processing Unit in same computer system;

For the device of assigning described first virtual machine to play up 3D figure in a described 3D Graphics Processing Unit.

14. equipment as claimed in claim 13, is characterized in that, also comprise:

Estimate to be used for be the device that described first virtual machine plays up the amount of the graphic memory of 3D figure for the information at least according to the Pixel Dimensions of each virtual machine in described group of mark.

15. equipment as claimed in claim 13, is characterized in that, also comprise:

Information for the ratio of compression at least used by a described 3D Graphics Processing Unit according to mark is estimated to be used for be the device that described first virtual machine plays up the amount of the graphic memory of 3D figure.

16. equipment as claimed in claim 13, is characterized in that, also comprise:

For at least estimated by mark, the information of the quantity on the surface of playing up for described first virtual machine estimated be used for for described first virtual machine plays up the device of the amount of the graphic memory of 3D figure.

17. equipment as claimed in claim 13, is characterized in that, also comprise:

For in response to determining that the stand-by period of at least one order sent by graphic kernel is less than the 3rd threshold value, select the device of a described 3D Graphics Processing Unit from described multiple 3D Graphics Processing Unit.

18. equipment as claimed in claim 13, is characterized in that, also comprise:

For in response to determining that the quantity of the virtual machine being currently assigned to a described 3D Graphics Processing Unit is less than the 4th threshold value, select the device of a described 3D Graphics Processing Unit from described multiple 3D Graphics Processing Unit.

19. equipment as claimed in claim 13, is characterized in that, also comprise:

For in response at least determining that a described 3D Graphics Processing Unit is overused based on the average latency of the direct memory access (DMA) figure affairs initiated by described first virtual machine, described first virtual machine is moved to the device of the 2nd 3D Graphics Processing Unit.

20. equipment as claimed in claim 13, is characterized in that, also comprise:

For determining that a described 3D Graphics Processing Unit is overused in response to the average latency at least based on the graph command sent to a described 3D Graphics Processing Unit by graphic kernel, described first virtual machine is moved to the device of the 2nd 3D Graphics Processing Unit.

21. equipment as claimed in claim 13, is characterized in that, also comprise:

For determining the device of the hardware characteristics of each 3D Graphics Processing Unit in multiple 3D Graphics Processing Unit.

22. 1 kinds, for the method balanced the load in Graphics Processing Unit, comprising:

Estimate the amount of the graphic memory be at least used for for virtual machine render graphics;

The amount of the amount of estimated graphic memory and the estimated available graphic memory controlled by the multiple 3D Graphics Processing Unit in same computer system is compared;

Be chosen as from the multiple 3D Graphics Processing Unit in same computer system the 3D Graphics Processing Unit that virtual machine plays up 3D figure according to comparing; And

Selected 3D Graphics Processing Unit is assigned to play up 3D figure for virtual machine.