CN106462446B - Host is selected using hardware multithreading parameter for virtual machine - Google Patents

Host is selected using hardware multithreading parameter for virtual machine Download PDF

Info

Publication number
CN106462446B
CN106462446B CN201580024412.1A CN201580024412A CN106462446B CN 106462446 B CN106462446 B CN 106462446B CN 201580024412 A CN201580024412 A CN 201580024412A CN 106462446 B CN106462446 B CN 106462446B
Authority
CN
China
Prior art keywords
computer system
hardware
cpu
host computer
host
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201580024412.1A
Other languages
Chinese (zh)
Other versions
CN106462446A (en
Inventor
J·W·特纳
J·W·克洛珀
K·L·亨德森
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Business Machines Corp
Original Assignee
International Business Machines Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US14/297,707 external-priority patent/US9400672B2/en
Priority claimed from US14/297,677 external-priority patent/US9384027B2/en
Priority claimed from US14/297,724 external-priority patent/US9304805B2/en
Application filed by International Business Machines Corp filed Critical International Business Machines Corp
Priority claimed from PCT/IB2015/054229 external-priority patent/WO2015186088A1/en
Publication of CN106462446A publication Critical patent/CN106462446A/en
Application granted granted Critical
Publication of CN106462446B publication Critical patent/CN106462446B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/5044Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering hardware capabilities
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061Partitioning or combining of resources
    • G06F9/5072Grid computing

Abstract

Cloud manager monitors the available resources on host computer system, the hardware thread quantity supported including the CPU on host computer system.Cloud manager receives the request of supply virtual machine (VM), which includes the hardware multithreading parameter of the quantity of hardware multithreading required in given host computer system.Then cloud manager selects host computer system by considering hardware multithreading parameter for VM.

Description

Host is selected using hardware multithreading parameter for virtual machine
Technical field
The disclosure broadly relate to host computer system dispose virtual machine, and more particularly, to by using Hardware multithreading parameter to select host for virtual machine.
Background technique
In cloud environment, cloud manager is to host computer system deployment virtual machine to create virtual server.Typically, Cloud manager receives request, requirement of the request designated virtual machine image for system resource (such as, memory, disk and CPU). Cloud manager then determines the available host computer system with required system resource, selects in available host computer system One, and by the host computer system of deploying virtual machine to the selection.
OpenStack is the open source software for establishing private and public cloud.In OpenStack, referred to as " flavor The virtual hardware template of (style) " is that virtual machine specifies required system resource.For example, the flavor in OpenStack can To specify the quantity of the memory size for virtual machine, root disk size and virtual cpu.When virtual machine needs are disposed, Calling is executed to cloud manager using the flavor for virtual machine resource needed for specified.Cloud manager, which is then found, to be had One or more host computer system for the resource specified in flavor, and deploying virtual machine is being met flavor's In host computer system on a host computer system.
Hardware multithreading in host can provide virtual machine to coordination host computer system.In the prior art In, if hyperthread (hyperthreading) is activated on host computer system, when for virtual machine selection master When machine computer system, it may be considered that the quantity of hardware thread, and virtual cpu is assigned independent of physical core To hardware thread.If hyperthread is not activated on host computer system, when for virtual machine selection Framework computing Then only consider physical processor core when machine system, and virtual cpu is only when VM is in selected or when being supplied from above, quilt Distribute to physical processor core.Utilize the modern processors frame of such as POWER (it includes multiple hardware threads in each core) Structure, the result is that the use of process resource is very insufficient.
When processing has separation core, the supply of virtual machine can be further complicated.Separation core processor refers to CPU Core can be separated into multiple daughter nucleus, can respectively include multiple hardware threads, and wherein daughter nucleus for guest operation system Core is shown as system.For example, the core of the separation core with 8 hardware threads is supported to can be separated into 4 daughter nucleus, every height Core supports 2 hardware threads.It is dynamic for separating core enabling, it means that separation can be in host operating system programmably Change.Known cloud manager will not separate core processor and distinguish in any way pair with without the processor for separating core To.As a result, known cloud manager may not recognize that the advantage using separation core processor when providing virtual machine.
Summary of the invention
According in a first aspect, providing a kind of device, comprising: at least one processor;Described in being coupled at least one Manage the memory of device;The resident cloud manager executed in the memory and by least one described processor, the cloud pipe Managing device includes: host MA monitoring agency, determines the available CPU's in multiple main frames computer system that virtual machine can be deployed to Quantity, and also determine the hardware thread quantity supported by each CPU on the multiple host computer system;And host Selection mechanism receives virtual machine (VM) request including virtual cpu quantity and hardware multithreading parameter, wherein the Selection of chiller Mechanism selects in the multiple host computer system to include the CPU number with the hardware thread quantity for meeting the VM request The host computer system of amount.
According to second aspect, a kind of selection executed by least one processor is provided for disposing the host of virtual machine The computer implemented method of computer system, which comprises determine that the multiple main frames that virtual machine can be deployed to calculate The quantity of available CPU in machine system;Determine the hardware thread supported by each CPU on the multiple host computer system Quantity;Receive virtual machine (VM) request including virtual cpu quantity and hardware multithreading parameter;And the multiple host of selection In computer system includes the host computer system of multiple CPU with the hardware thread quantity for meeting the VM request.
The present invention may be implemented as computer program.
According to preferred embodiment, cloud manager monitors the available resources on host computer system, including Framework computing The hardware thread quantity supported by CPU in machine system.Cloud manager receives the request of supply virtual machine (VM), which includes The hardware multithreading parameter of the quantity of hardware multithreading needed in given host computer system.Then cloud manager is by examining Hardware multithreading parameter is considered to select host computer system for VM.
According to one embodiment, a kind of solution that virtual cpu is placed by using hardware multithreading parameter is provided Scheme.
According to one embodiment, the present disclosure relates generally to by deploying virtual machine to host computer system, and more Body it is related to that virtual cpu is deployed to host computer system by using hardware multithreading parameter.
According to one embodiment, it provides a kind of for multi-thread using hardware in the host with separation core processor Journey parameter supplies the solution of virtual cpu.
According to one embodiment, the present disclosure relates generally to by deploying virtual machine to host computer system, and more Body it is related to that virtual cpu will be disposed to the host for including one or more separation core processors by using hardware multithreading parameter Computer system.
According to one embodiment, a kind of device is provided, comprising: at least one processor;It is coupled to described at least one The memory of a processor;The resident cloud manager executed in the memory and by least one described processor, institute Stating cloud manager includes: host MA monitoring agency, determine in multiple main frames computer system that virtual machine can be deployed to can With the quantity of CPU, and also determine the hardware thread quantity supported by each CPU on the multiple host computer system; Selection of chiller mechanism receives virtual machine (VM) request including virtual cpu quantity and hardware multithreading parameter, wherein the host Selection mechanism selects in the multiple host computer system to include with the hardware thread quantity for meeting the VM request The host computer system of multiple CPU;And virtual cpu (vCPU) placement mechanism, it is being selected using the hardware multithreading parameter Multiple virtual cpus (vCPU) is placed on the host computer system selected.
According to one embodiment, provide it is a kind of by least one processor execute in host computer system The computer implemented method of upper deployment virtual machine, which comprises determine the multiple main frames meter that virtual machine can be deployed to The quantity of available CPU in calculation machine system;Determine the hardware lines supported by each CPU on the multiple host computer system Number of passes amount;Receive virtual machine (VM) request including virtual cpu quantity and hardware multithreading parameter;Select the multiple host meter In calculation machine system includes the host computer system of multiple CPU with the hardware thread quantity for meeting the VM request;With And multiple virtual vCPU are placed on the host computer system of selection using the hardware multithreading parameter.
According to one embodiment, a kind of computer implemented method executed by least one processor, institute are provided Method is stated for placing multiple virtual cpus (vCPU) on host computer system, which comprises determine the quantity of CPU, The amount of memory, and the amount of available disk space in the multiple main frames computer system of virtual machine can be disposed;Described in determination The quantity for the hardware thread that each CPU in multiple main frames computer system is supported;Receive the virtual machine including the following contents (VM) it requests: specifying the storage requirement of the minimum of the memory of the VM;The disk demand of the minimum of the disk of specified VM;With And cpu demand, the quantity and hardware multithreading parameter for the virtual cpu of the VM are specified, wherein the hardware multithreading The first value instruction hardware multithreading of parameter is closed, the second value instruction hardware thread quantity of the hardware multithreading parameter Numerical value, and the third value of the hardware multithreading parameter indicates that the host computer system of selection is not consider hardware Thread is on or off that in the case where is selected on the host computer system of selection;Select the multiple master In machine computer system includes a host computer of multiple CPU with the hardware thread quantity for meeting the VM request System;And the multiple vCPU is placed on selected host computer using the hardware multithreading parameter in the following way In system: when hardware multithreading parameter instruction hardware multithreading is closed, by placing each vCPU in the VM On different physical cores in selected host computer system;When hardware multithreading parameter indicates hardware thread quantity, by VM In each vCPU be placed on the different hardware thread in selected host computer system.
According to one embodiment, the method as described in upper section is provided, further includes being joined by using hardware multithreading VM is sized by number.
According to one embodiment, a kind of device is provided, comprising: at least one processor;It is coupled to described at least one The memory of a processor;The resident cloud manager executed in the memory and by least one described processor, it is described Cloud manager includes: host MA monitoring agency, determines: can dispose the available CPU in the multiple main frames computer system of virtual machine Quantity;The hardware thread quantity that each CPU on the multiple host computer system is supported;And it is directed to the multiple master Whether each CPU in machine computer system enables separation core;Selection of chiller mechanism receives the virtual machine including the following terms (VM) it requests: virtual cpu quantity;And hardware multithreading parameter;Wherein the Selection of chiller mechanism selects the multiple host In computer system includes the number for having multiple CPU of hardware thread quantity and meeting the virtual cpu in the VM request One host computer system of the separation core setting of amount and hardware multithreading parameter.
According to one embodiment, provide it is a kind of by least one processor execute in host computer system The upper computer implemented method for placing multiple virtual cpus (vCPU), which comprises determination can dispose the multiple of virtual machine Available CPU quantity on host computer system;Determine the hardware that each CPU on the multiple host computer system is supported The quantity of thread;Determine whether each CPU on the multiple host computer system enables separation core, and if it is, Determine the subset quantity of each core;Receive virtual machine (VM) request including virtual cpu quantity and hardware multithreading parameter;And It selects in the multiple host computer system to include that there are multiple CPU of hardware thread quantity and meet the VM to ask One host computer system of the separation core setting of the quantity and hardware multithreading parameter of the virtual cpu in asking.
According to one embodiment, a kind of computer implemented method executed by least one processor, institute are provided Method is stated for placing multiple virtual cpus (vCPU) on host computer system, which comprises determine the quantity of CPU, The amount of memory, and the amount of available disk space in the multiple main frames computer system of virtual machine can be disposed;Described in determination The quantity for the hardware thread that each CPU in multiple main frames computer system is supported;Determine the multiple host computer system On each CPU whether enable separation core;It receives the virtual machine (VM) including the following contents to request: specifying the memory of the VM Minimum storage requirement;The disk demand of the minimum of the disk of specified VM;And cpu demand, it specifies for the VM Virtual cpu quantity and hardware multithreading parameter, wherein the hardware multithreading parameter the first value indicate hardware multithreading It is closed, the numerical value of the second value instruction hardware thread quantity of the hardware multithreading parameter, and the hardware multithreading is joined Several third values indicates that the host computer system of selection is not consider hardware thread in the Framework computing of selection It is selected in the case where being on or off in machine system;Including multiple CPU with hardware thread quantity and meet One host computer system of the separation core setting of the quantity and hardware multithreading parameter of the virtual cpu in the VM request; And be arranged in the following way using the separation core of the hardware multithreading parameter and the host computer system of selection, it will The multiple vCPU is placed on selected host computer system: when hardware multithreading parameter instruction hardware multithreading is closed When, by the way that each vCPU in the VM is placed on the different physical cores in selected host computer system;When hardware is more When the separation core setting of thread parameter instruction hardware thread quantity and selected computer system is activated, by each of VM VCPU is placed on the different hardware thread in selected host computer system.
According to one embodiment, the available resources on cloud manager monitoring host computer system, including by host meter The hardware thread quantity that CPU in calculation machine system is supported.Cloud manager receives the request of supply virtual machine (VM), which includes The hardware multithreading parameter for whether allowing hardware multithreading on host computer system specified.Then cloud manager passes through consideration Hardware multithreading parameter is that VM selects host computer system.Then VM is placed on by using hardware multithreading parameter selected On host computer system.
According to one embodiment, the available resources on cloud manager monitoring host computer system, including Framework computing Whether separation core is enabled by the CPU hardware thread quantity supported and CPU in machine system.Cloud manager receives supply virtual machine (VM) request, the request include that the hardware multithreading for whether allowing hardware multithreading on host computer system is specified to join Number.Then, cloud manager is by considering hardware multithreading parameter, the hardware thread of CPU support and separation core setting, for VM choosing Select host computer system.Then VM is placed on selected host computer system by using hardware multithreading parameter.Knot Fruit is more effectively to utilize the cpu resource in the host of virtual machine.
According to describing in greater detail below as shown in drawings, foregoing and other feature and advantage be will be apparent.
Detailed description of the invention
It will refer to attached drawing hereafter, describe embodiments of the present invention only by means of way of example.Similar attached drawing indicates Indicate similar element:
Fig. 1 is the block diagram of the cloud computing node of a preferred embodiment according to the present invention;
Fig. 2 is the block diagram of the cloud computing environment of a preferred embodiment according to the present invention;
Fig. 3 is the block diagram of the level of abstraction model layer of a preferred embodiment according to the present invention;
Fig. 4 is the block diagram for showing certain features of cloud manager of a preferred embodiment according to the present invention;
Fig. 5 is the block diagram for showing certain features of cloud VM request of a preferred embodiment according to the present invention;
Fig. 6 is a preferred embodiment according to the present invention for being registered in potential host computer system system Available resources cloud manager method flow chart;
Fig. 7 is the cloud manager of a preferred embodiment according to the present invention to supply on host computer system The flow chart of the method for VM;
Fig. 8 is the block diagram of the x86 double-core CPU of the prior art;
Fig. 9 is flow chart of the cloud manager to the art methods for VM request selecting host;
Figure 10 is to show the block diagram that cpu demand how is specified in prior art cloud VM request;
Figure 11 is the potential host meter for showing three x86 dual core processors of a preferred embodiment according to the present invention The block diagram of calculation machine system;
Figure 12 is that showing for a preferred embodiment according to the present invention is potential into Figure 11 including that can be assigned The block diagram of the virtual machine of the vCPU of host;
Figure 13 is the flow chart that the art methods of vCPU are placed on the physical core shown in host computer system;
Figure 14 is the six core CPU of Power8 for showing two in six cores of a preferred embodiment according to the present invention Framework block diagram;
Figure 15 be a preferred embodiment according to the present invention for selected using multithreading parameter host and Control the flow chart for the method that the vCPU on host is placed;
Figure 16 is the quantity and hardware multithreading parameter including vCPU of a preferred embodiment according to the present invention Example cpu demand;
Figure 17 is the first sample cpu demand including hardware multithreading parameter according to the preferred embodiment of the present invention;
Figure 18 shows the potential Power8 master of the cpu demand met in Figure 17 according to the preferred embodiment of the present invention Machine;
Figure 19 shows the second sample CPU including hardware multithreading parameter according to the preferred embodiment of the present invention and needs It asks;
Figure 20 shows the potential Power8 master of the cpu demand met in Figure 19 according to the preferred embodiment of the present invention Machine;
Figure 21 shows the third sample CPU including hardware multithreading parameter according to the preferred embodiment of the present invention and needs It asks;
Figure 22 shows the potential Power8 master of the cpu demand met in Figure 21 according to the preferred embodiment of the present invention Machine;
Figure 23 shows the 4th sample CPU including hardware multithreading parameter according to the preferred embodiment of the present invention and needs It asks;
Figure 24 shows the potential Power8 master of the cpu demand met in Figure 23 according to the preferred embodiment of the present invention Machine;
Figure 25 is according to the preferred embodiment of the present invention for selecting to meet the CPU need including hardware multithreading parameter The flow chart of the method for the host asked;
Figure 26 is the flow chart for placing the method for vCPU on selected host computer system using hardware multithreading parameter;
Figure 27 is the host-level statistics shown for each potential host collection according to the preferred embodiment of the present invention Block diagram;
Figure 28 is showing and can realize in OpenStack flavor to refer to according to the preferred embodiment of the present invention Determine the block diagram of hardware multithreading parameter additionally standardized;
Figure 29 is for executing the side for being sized (resize) operation to the existing VM that may include hardware multithreading parameter The flow chart of method;
Figure 30 is to show the block diagram of the framework of the six core CPU of Power8 using separation core, and it illustrates in six cores Two;
Figure 31 be include the number of vCPU and the sample cpu demand of hardware multithreading parameter;
Figure 32 is the first sample cpu demand for including hardware multithreading parameter;
Figure 33 shows the potential Power8 host of the cpu demand met in Figure 32;
Figure 34 shows the second sample cpu demand including hardware multithreading parameter;
Figure 35 shows the potential Power8 host of the cpu demand met in Figure 34;
Figure 36 shows the third sample cpu demand including hardware multithreading parameter;
Figure 37 shows the potential Power8 host of the cpu demand met in Figure 36;
Figure 38 shows the 4th sample cpu demand including hardware multithreading parameter;
Figure 39 shows the potential Power8 host of the cpu demand met in Figure 38;
Figure 40 is for selecting the host for meeting cpu demand when host has separation core and responsible method Flow chart, the cpu demand include hardware multithreading parameter;
Figure 41 is the block diagram for showing filter, which is used to be arranged to 4 or 8 Shi Keyong when hardware multithreading parameter In the host for filtering out enabled separation kernel normal form;And
Figure 42 is for using hardware multithreading parameter in selected host computer system when host has separation core VCPU is placed on system and the flow chart of responsible method.
Specific embodiment
Cloud manager monitor host computer system on available resources, including by host computer system by CPU The separation the core whether multiple hardware threads and CPU supported enable.Cloud manager receives the request of supply virtual machine (VM), should Virtual machine includes hardware multithreading parameter, which specifies whether allow hardware more on host computer system Thread (or in one embodiment, the quantity of hardware multithreading required on host computer system).Then, Yun Guan Device is managed by considering hardware multithreading parameter, the hardware thread by CPU support and separation core setting, selects Framework computing for VM Machine system.Then, VM is placed on selected host computer system using hardware multithreading parameter.The result is that more effective land productivity Virtual machine is used for the cpu resource in host.
It is understood in advance that although the disclosure includes the detailed description about cloud computing, the technical solution being described Realization be but not limited to cloud computing environment, but can be in conjunction with the calculating ring of any other type currently known or develop later Border and realize.
Cloud computing is a kind of service offering mode, for carrying out conveniently, on demand to shared configurable computing resource pool Network access.Configurable computing resource is can to carry out least interaction energy with the smallest management cost or with ISP The resource of rapid deployment and release, for example, can be network, network bandwidth, server, processing, memory, storage, using, it is virtual Machine and service.This cloud mode may include at least five features, at least three service models and at least four deployment models.
Feature includes:
On-demand self-help service: the consumer of cloud can be single in the case where being not necessarily to artificially be interacted with ISP Aspect automatically disposes the computing capability of server time and network storage etc. on demand.
Extensive network insertion: computing capability can be obtained on network by standards body, and this standards body promotes By different types of thin client platform or thick client computer platform, (such as mobile phone, laptop computer, individual digital are helped Manage PDA) use to cloud.
Resource pool: the computing resource of supplier is included into resource pool and by multi-tenant (multi-tenant) mode service In multiple consumer, wherein different actual resource and virtual resource are dynamically distributed and are reallocated on demand.Under normal circumstances, Consumer not can control or even and the accurate location of resource provided by being unaware of, but can specify on higher level of abstraction Position (such as country, state or data center), therefore there is location independence.
Rapidly elasticity: can rapidly, flexiblely (sometimes automatically) dispose computing capability, to realize Quick Extended, And it can discharge rapidly and carry out rapid drop.In consumer, the available computing capability for deployment often seem be it is unlimited, And any number of computing capability can be obtained when any.
Measurable service: cloud system passes through using being suitable for service type (such as storage, processing, bandwidth and any active ues Account number) certain level of abstraction metrology capability, automatically control and optimize resource effectiveness.It can monitor, control and report money Source service condition provides transparency for ISP and consumer both sides.
Service model is as follows:
Software is to service (SaaS): the ability provided the consumer with is answering of being run in cloud infrastructure using supplier With.It can be set by the thin client interface (such as network-based Email) of such as web browser from various client computer Standby access application.Other than the limited application configuration setting specific to user, consumer neither manages nor controls including net The bottom cloud architecture of network, server, operating system, storage or even single application power etc..
Platform services (PaaS): the ability provided the consumer with is to dispose consumer's creation in cloud infrastructure or obtain Application, these application using the program design language that supplier supports make peace tool creation.Consumer neither manages nor controls System includes network, server, operating system or the bottom cloud architecture of storage, but the application disposed to it is possessed of control power, It may also be possessed of control power to application hosting environment configuration.
Architecture services (IaaS): the ability provided the consumer with is that consumer can dispose wherein and run packet Include processing, storage, network and other basic calculation resources of any software of operating system and application.Consumer neither manages The cloud infrastructure of bottom is not controlled, but is possessed of control power to operating system, storage and the application of its deployment, to the net of selection Network component (such as host firewall) may have limited control.
Deployment model is as follows:
Private clound: cloud infrastructure operates independently for a certain mechanism.Cloud infrastructure can be by the tissue or third party tube It manages and can reside in the organization internal or outside.
Community Cloud: cloud infrastructure is shared by several tissues and has supported common interests (such as task mission, peace Full demand, policy and close rule and consider) specific community.Community Cloud can be by common intracorporal multiple tissues or third party tube It manages and can reside in inside or outside the community.
Public cloud: cloud infrastructure provides to the public or large-scale industrial colony and is possessed by the tissue of sale cloud service.
Mixed cloud: cloud infrastructure by two or more deployment models cloud (private clound, community Cloud or public cloud) group At these clouds are still unique entity, but standardized technique or proprietary technology by making data and using that can transplant (such as the cloud burst flow for the load balance between cloud shares technology) binds together.
Cloud computing environment be it is service-oriented, feature concentrates on the mutual behaviour of statelessness, lower coupling, modularity and the meaning of one's words The property made.The core of cloud computing is the architecture comprising interconnecting nodes network.
Referring now to Figure 1, which show the block diagrams of an example of cloud computing node.The cloud computing node that Fig. 1 is shown 100 be only an example of suitable cloud computing nodes, should not function to embodiment of the present invention and use scope bring Any restrictions.In short, cloud computing node 100 can be utilized to implement and/or execute above-described any function.
Cloud computing node 100 has computer system/server 110, can be with other numerous general or specialized calculating System environment or configuration operate together.It is well known that the computing system for being suitable for operating together with computer system/server 110, ring The example of border and/or configuration includes but is not limited to: personal computer system, server computer system, table top computer system, Thin client, thick client computer, hand-held or laptop devices, microprocessor-based system, set-top box, programmable consumer electronics produce Product, NetPC Network PC, little type Ji calculate machine Xi Tong ﹑ large computer system and the distributed cloud computing including above-mentioned arbitrary system Technological accumulation and inheritance, etc..
Computer system/server 110 can the computer system executable instruction executed by computer system (such as Program module) general context under describe.In general, program module may include executing specific task or realizing specifically to take out Routine, program, target program, component, logic, data structure of image data type etc..Computer system/server 110 can be with Implement in the distributed cloud computing environment that the remote processing devices being linked through a communication network execute task.In distributed cloud meter It calculates in environment, it includes on the Local or Remote computing system storage medium of storage equipment that program module, which can be located at,.
As shown in Figure 1, the table in the form of universal computing device of the computer system/server 110 in cloud computing node 100 It is existing.The component of computer system/server 110 can include but is not limited to: one or more processor or processing unit 120, system storage 130 connects the bus 122 of different system components (including system storage 130 and processing unit 120).
Bus 122 indicates one of a few class bus structures or a variety of, including memory bus or Memory Controller, Peripheral bus, graphics acceleration port, processor or the local bus using any bus structures in a variety of bus structures.It lifts For example, these architectures include but is not limited to industry standard architecture (ISA) bus, microchannel architecture (MAC) Bus, enhanced isa bus, Video Electronics Standards Association (VESA) local bus and peripheral component interconnection (PCI) bus.
Computer system/server 110 typically comprises a variety of computer system readable media.These media can be energy Enough any obtainable media accessed by computer system/server 110, including volatile and non-volatile media, it is removable Dynamic and immovable medium.Fig. 1 shows an example of removable media comprising digital video disc (DVD) 192.
System storage 130 may include the computer of the volatibility or nonvolatile memory form of such as firmware 132 System-readable media.Firmware 132 provides the interface to the hardware of computer system/server 110.System storage 130 may be used also To include the computer system readable media of form of volatile memory, such as random access memory (RAM) 134 and/or height Fast buffer memory 136.It is removable/nonremovable, volatile that computer system/server 110 may further include other Property/nonvolatile computer system storage medium.Only as an example, storage system 140 can be used for reading and writing immovable, non- Volatibility magnetic medium (Fig. 1 do not show, commonly referred to as " hard disk drive ").Although being not shown in Fig. 1, can provide for can The disc driver of mobile non-volatile magnetic disk (such as " floppy disk ") read-write, and to removable anonvolatile optical disk (such as CD- ROM, DVD-ROM or other optical mediums) read-write CD drive.In these cases, each driver can pass through one A or multiple data media interfaces are connected with bus 122.Memory 130 may include at least one program product, the program Product has one group of (for example, at least one) program module, these program modules are configured to perform each embodiment of the present invention Function is as follows described in detail.
Program/utility 150 with one group of (at least one) program module 42, can store in memory 130, Such program module 152 includes but is not limited to operating system, one or more application program, other program modules and journey It may include the realization of network environment in ordinal number evidence, each of these examples or certain combination.Program module 152 is usually held Function and/or method in row embodiment described in the invention.
Computer system/server 110 can also with one or more external equipments 190 (such as keyboard, sensing equipment, Display 180, disc driver etc.) communication, it can also be enabled a user to one or more and the computer system/server The equipment communication of 110 interactions, and/or with enable the computer system/server 110 and one or more other to calculate equipment Any equipment (such as network interface card, modem etc.) communication communicated.This communication can pass through input/output (I/ O) interface 170 carries out.Also, computer system/server 110 can also pass through network adapter 160 and one or more Network (such as local area network (LAN), wide area network (WAN) and/or public network, such as internet) communication.As shown, network is suitable Orchestration 160 is communicated by bus 122 with other modules of computer system/server 110.It should be understood that although not showing in figure Out, other hardware and/or software module can operate together with computer system/server 110, including but not limited to: micro- generation Code, device driver, redundant processing unit, external disk drive array, redundant array of independent disks (RAID) system, tape drive Dynamic device and data backup storage system etc..
Referring now to Figure 2, which show illustrative cloud computing environments 200.As shown, cloud computing environment 200 wraps One or more cloud computing node 100 of letter can be communicated therewith by including the local computing device that cloud computing consumer uses, this Ground, which calculates equipment, for example can be personal digital assistant (PDA) or mobile phone 210A, desktop computer 210B, laptop 210C and/or Automotive Computer System 210N.It can be in communication with each other between cloud computing node 100.Can include but is not limited to It as described above will in private clound, community Cloud, public cloud or mixed cloud or one or more network of their combination Cloud computing node 100 carries out physics or virtual group (not shown).In this way, the consumer of cloud is not necessarily in local computing device It is upper maintenance resource just can request that cloud computing environment 200 provide architecture service (IaaS), platform service (PaaS) and/ Or software services (SaaS).It should be appreciated that all kinds of calculating equipment 210A-N that Fig. 2 is shown are only schematical, cloud computing Node 100 and cloud computing environment 200 can in any type of network and/or any type of meter that network addressable is connect Calculate equipment (such as using web browser) communication.
Referring now to Figure 3, which show one group of functional abstraction layers that cloud computing environment 200 (Fig. 2) is provided.First should Understand, component, layer and function shown in Fig. 3 are all only schematically that disclosure of the invention and claim are without being limited thereto. As shown in figure 3, providing following layers and corresponding function:
Hardware and software layer 310 includes hardware and software component.The example of hardware component includes: host, such as SystemSystem;Based on the server of RISC (Reduced Instruction Set Computer) architecture, such as IBM SystemSystem System;IBM System x system;IBM BladeCenter system;Store equipment;Network and networking component.The example of component software Son includes: network application server software, such as IBMApplication server software;Database software, example Such as IBMDatabase software.(IBM, System z, System p, WebSphere and DB2 are International Business Machine Company is in registered trademark from all parts of the world).
Virtual level 320 provides a level of abstraction, this layer can provide the example of following pseudo-entity: virtual server, void Quasi- storage, virtual network (including virtual private networks), virtual application and operating system and virtual client.
In one example, management level 330 can provide following function: resource provisioning function: provide in cloud computing The computing resource of task and the dynamic acquisition of other resources are executed in environment;Metering and pricing function: right in cloud computing environment The use of resource carries out cost tracing, and provides bill and invoice thus.In one example, which may include using soft Part license.Security function: providing authentication for the consumer and task of cloud, provides protection for data and other resources.User Portal function: the access to cloud computing environment is provided for consumer and system manager.Service level management function: cloud meter is provided The distribution and management of resource are calculated, to meet required service level.Service Level Agreement (SLA) plans and fulfils function: for root The offer of cloud computing resources tomorrow requirement is presetted and is supplied according to SLA prediction.Cloud manager 350 is to retouch as detailed below The example for the cloud manager stated.Although cloud manager 350 shown in Fig. 3 resides in management level 330, cloud manager 350 is also All ranks shown in Fig. 3 can be crossed over.
The example of the offer cloud computing environment function in the cards of workload layer 340.In this layer, it is possible to provide work The example of load or function includes: mapping and navigation;Software development and life cycle management;The teaching of Virtual Class mentions For;Data Analysis Services;Trading processing;And mobile desktop.
The present invention can be system, method and/or computer program product.Computer program product may include having thereon There is the computer readable storage medium (or medium) of computer-readable program instructions, for making processor execute each side of the invention Face.
Computer readable storage medium can be tangible device, can retain with store instruction for instruction execution equipment It uses.Computer readable storage medium can be such as but not limited to electronic storage device, magnetic storage apparatus, light storage device, Electric magnetic storage apparatus, semiconductor memory apparatus or any suitable combination above-mentioned.Computer readable storage medium it is more specific Exemplary non-exhaustive list include the following: portable computer diskette, hard disk, random access memory (RAM), read-only memory (ROM), Erasable Programmable Read Only Memory EPROM read-only memory (EPROM or flash memory), static random access memory (SRAM), Portable optic disk read-only storage (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, encoding device, such as wherein Record has punched card in the groove of instruction or bulge-structure and any suitable combination above-mentioned.As it is used herein, Computer readable storage medium is not interpreted as transient signal itself, the electromagnetism of such as radio wave or other Free propagations Wave, the electromagnetic wave (for example, light pulse passes through fiber optic cables) propagated by waveguide or other transmission mediums or passes through wire transmission Electric signal.
Computer-readable program instructions described herein can be counted via network from computer readable storage medium or outside Calculation machine or External memory equipment download to corresponding calculating/processing equipment, which is, for example, internet, local area network, wide area network Local Area Network and/or wireless network.Network may include copper transmission cable, optical delivery fiber, wireless transmission, router, fire prevention Wall, interchanger, gateway computer and/or Edge Server.Network adapter cards or network in each calculating/processing equipment connect Mouth receives computer-readable program instructions from network, and forwards computer-readable program instructions to be stored in corresponding calculating/processing In computer readable storage medium in equipment.
Computer-readable program instructions for executing operation of the invention can be assembly instruction, instruction set architecture (ISA) instruction, machine instruction, machine-dependent instructions, microcode, firmware instructions, condition setup data or source code or target generation Code is with one or more programming languages (including the programming language of the object-oriented of such as Smalltalk, C++ etc., and such as " C " The conventional process programming language of programming language or similar programming language etc) any combination write.Computer-readable program Instruction can execute on the user's computer completely, and part executes on the user's computer, execute as independent software package, Part executes on the user's computer, and part executes on the remote computer, or completely in remote computer or server Upper execution.In the latter case, remote computer can pass through any kind of network (including local area network (LAN) or wide area Net (WAN)) it is connected to the computer of user, or may be coupled to outer computer for example, mentioning by using Internet service For the internet of quotient).In some embodiments, including such as programmable logic circuit, field programmable gate array (FPGA) Or the electronic circuit of programmable logic array (PLA) can pass through the status information using computer-readable program instructions, Lai Zhihang Computer-readable program instructions are with personalized by electronic circuit, to execute embodiments of the present invention.
Herein with reference to the method for embodiment according to the present invention, the flow chart of device (system) and computer program product And/or block diagram describes each aspect of the present invention.It should be appreciated that each of flowchart and or block diagram piece and flow chart and/or The combination of block in block diagram can be realized by computer-readable program instructions.
These computer-readable program instructions can be provided to general purpose computer, special purpose computer or other programmable numbers According to the processor of processing unit to generate machine, so that being created via the processor of computer or other programmable data processing units Build the device for the function action specified in one or more boxes for realizing flowchart and or block diagram.These computers can Reader instruction can also store in a computer-readable storage medium, can indicate computer, programmable data processing dress It sets and/or other equipment works in a specific way, so that the computer readable storage medium for being wherein stored with instruction includes realizing The manufacture of instruction in terms of the function action specified in one or more frames of flowchart and or block diagram.
Computer-readable program instructions can also be loaded into computer, other programmable data processing units or other equipment On, so that series of operation steps are executed in other programmable devices or other equipment in computer, to generate computer reality Existing makes in computer, and the one of the instruction implementation flow chart and/or block diagram that are executed on other programmable devices or other equipment The function action specified in a or multiple boxes.
Flowcharts and block diagrams in the drawings show system according to various embodiments of the present invention, method and computer The framework in the cards of program product, function and operation.In this regard, each of flowchart or block diagram piece can indicate to refer to The module of order, section or part comprising for realizing one or more executable instructions of specified logic function.It is replaced some It can not occur according to sequence shown in the drawings for the function of in embodiment, being marked in frame.For example, depending on involved Function, two blocks continuously shown can actually be performed simultaneously substantially or these blocks sometimes can be in reverse order It executes.It will additionally note that, the frame in each frame and block diagram and or flow chart diagram in block diagram and or flow chart diagram Combination can execute specific use hardware and calculating by executing realizing based on the system of specialized hardware for specified function or movement The combination of machine instruction.
Fig. 4 shows a suitable example of cloud manager 350 shown in Fig. 3.Cloud manager 350 includes that cloud supplies machine Structure 410 comprising resource request interface 420.Resource request interface 420 allows software entity to request virtually from cloud manager 350 Machine is without manual intervention.Cloud manager 350 further includes user interface 430, and user is allowed to be interacted with cloud manager to execute Any suitable function, the offer including VM, VM's is destructed, the performance evaluation etc. of cloud.Resource request interface 420 and user connect Difference between mouth 430 is that user must execute the function of being specified by user using user interface 430 manually, and resource is asked Ask interface 420 that can be used to request to be provided by cloud mechanism 350 cloud resource by software entity without the input from human user.
Cloud manager 350 further includes the host MA monitoring agency for monitoring the available resource on potential host computer system 440, thus cloud manager 350 can will be directed to the available resources having in the specified resource and potential host computer system of VM It is compared.Selection of chiller mechanism 450 allows based on will be on the specified resource of requested VM and potential host computer system Available resources be compared, to select potential host computer system.VCPU placement mechanism 460 is multi-thread by using hardware VCPU is placed on selected host computer system by journey parameter.VM, which is sized mechanism 470, to be allowed to use hardware multi-thread The mode of journey parameter adjusts the size of existing VM, even if being also such when original VM does not use hardware multithreading parameter.
Fig. 5 is shown can connect via resource request interface 420 or via the user on the cloud organization of supply 410 in Fig. 4 Mouthfuls 430 and the suitable example of cloud VM request 510 submitted, cloud VM request 510 are mentioned to request by cloud manager 350 For VM.Cloud VM request 510 may include any specification of the demand to resource required on VM.The example of suitable demand includes depositing Memory requirements 520, disk demand 530 and cpu demand 540.Amount of memory needed for storage requirement 520 specifies VM.Disk demand 530 The amount of disk needed for specified VM.The quantity of virtual cpu needed for cpu demand 540 specifies VM, and hardware can also be specified multi-thread Journey parameter, discusses in greater detail below.Of course, it is possible to specify other unshowned demands in Fig. 5, such as network to need for VM It sums any other suitable demand.In the prior art, the things that OpenStack is referred to as " flavor ", which is used as cloud VM, asks It asks, specifies the minimum essential requirement to the resource on VM.
As shown in fig. 6, method 600 is preferably executed by the host MA monitoring agency 440 in Fig. 4, to monitor potential master Available resources in machine computer system.Determine the available resources on the potential host computer system of (step 610) and record (step It is rapid 620).A kind of known way of implementation method 600 is, using driver inquiry host to determine its available resources.
With reference to Fig. 7, method 700 starts (step 710) by receiving cloud VM request.Then the resource in cloud VM request is needed It asks and is compared (step 720) with the available resources on potential host computer system.Note that can be from the step 620 in Fig. 6 Available resources in the log of middle creation in read step 720.Then selection, which has, meets the resource requirement in cloud VM request Host (the step 730) of resource.Then, VM is deployed in (step 740) on selected host computer system.
Many modern processors (or CPU) include multiple processing cores.Multi-core processor another example is shown in Fig. 8 The double-core CPU 810 of x86.X86 double-core CPU 810 includes tool there are two the first core 820 of hardware thread 822 and 824 and has Second core 840 of two hardware threads 842 and 844.The presence of multiple hardware threads on each core can be to host computer System, which is supplied, generates challenge in VM.In the prior art, the quantity for usually ignoring the hardware thread in multi-core CPU, only considers object Manage core.For example, art methods 900 in Fig. 9, which read cloud VM, requests (step 910), then selection, which has, meets cloud VM and asks Host (the step 920) of multiple physical cores of vCPU quantity in asking.The cpu demand 1010 of the prior art as shown in Figure 10 refers to Determine the quantity of the virtual cpu (vCPU) in VM, and is an appropriate example of cpu demand 540 shown in fig. 5.Because for VM selects method 900 when host that physical core is used only, so multiple hardware threads on core will be under-utilized, from without utilizing Available cpu resource.Other in the prior art, the quantity of hardware thread is considered when enabling the hyperthread of host, and is held Row vCPU to hardware thread placement without consider physical core.This method may cause invalid VM, because a few thing is negative Load is benefited from using the thread in different IPs, and other workloads are then directed to same core and are preferably serviced by thread.
Step 920 in Fig. 9 may include for selecting any suitable of the host computer system for meeting VM request Criterion or exploration.For example, the quantity of physical core can increase according to some excessive ratios (overcommit ratio) such as 2.0 Add.Therefore, if host tool is there are three physical core is available, three physical cores will lead to multiplied by 2.0 excessive ratio available Processor number is 6, it means that can dispose the VM for needing six vCPU on host, and cannot dispose and need seven vCPU's VM.It will be discussed below a simply example.
Figure 11 indicates the potential host computer system of sample including three double-core x86CPU.We assume that this host is Through so that VM1 shown in Figure 12 operation thereon, two in two physical cores and CPU2 which is deployed in CPU1 First in physical core, as shown in the shade in Figure 12 in those cores.We are it is further assumed that VM2 shown in Figure 12 needs It is disposed.The quantity for determining available core is nine for the example in Figure 11, i.e., six cores are subtracted multiplied by excessive ratio 2 Three cores that VM1 in the potential host 1 as shown in Figure 11 is used.Because VM2 needs two vCPU, and potential host 1 With 9 available cores, so potential host 1 is effective potential host of VM2 shown in Figure 12.
Note that by vCPU it is practical be placed into be independently of on potential host selection be used for VM potential host independent step Suddenly.With reference to Figure 13, method 1300 starts (step 1310) by reading cloud VM request.Then the vCPU in cloud VM request is placed (the step 1320) on physical core.In Figure 11, the first core of the thread T1 and CPU2 of the first core and the second core of CPU1 are negative Shadow is shown, and has been placed on these three corresponding cores with three vCPU indicated in VM1 shown in Figure 12.It is noted that Placement is not static.In different times, vCPU can be run on any hardware thread on any active nucleus.By making With hyperthread, 12 hardware threads are shared in three double-cores of distribution.VM1 needs 3 such threads, and leaves 9 for VM It is a.However, vCPU is assigned to any thread, this may mean that the work run in the operating system thread of shared memory Work will preferably be run in the hardware thread of identical vCPU, and the work run in independent operating system thread preferably exists It is run in hardware thread in different core.Therefore, the prior art is identifying the mode of potential host and selected for VM Being run in the mode of VM on host leads to poor efficiency, this is because the distribution of VM to potential host does not consider the core on CPU currently Difference between hardware thread.
The prior art identifies physical processor core or hardware thread, but the two cannot be identified as to virtual cpu can dispose The entity arrived.In the case where no hyperthread, there are two available CPU, forms for single double-core CPU tool as shown in Figure 8 Two physical cores 820 and 840 that can be deployed to for vCPU.Using hyperthread, single double-core CPU can be supported more vCPU;However, whether vCPU is not deterministic by the hardware thread operation in same physical core or different IPs.By recognizing VCPU needs are assigned to multiple hardware threads rather than just single physical core or single hardware thread, this disclosure Finer granularity level is created with claim.
When the quantity of hardware thread increases, the problem of vCPU is assigned to single hardware thread or core aggravation.For Fig. 8 With 11 shown in x86 double-core CPU, hyperthread leads to two threads of each core.For the processor with more multi-hardware thread, ask Topic becomes readily apparent from.For example, 6 core CPU 1410 of Power8 shown in Figure 14 is to include that (including eight hard for the first core 1420 1448) and the second core 1450 (further includes eight hardware part thread 1432,1434,1436,1438,1442,1444,1446 and Thread 1462,1464,1466,1468,1472,1474,1476 and 1478.It is not shown four other cores in Figure 14, but class Two core shown in being similar to.When the number based on the physical core as shown in the art methods 900 in Fig. 9 selects host, Many hardware threads can not used.Therefore, if two vCPU are deployed to two core 1420 and 1450 shown in Figure 14, knot Fruit is by seven in eight hardware threads wasted on each core.Final result is to put will use only at any given time to be used for Two in 16 threads of two core, this wastes 14 in 16 hardware threads.This means that Fig. 9 can be used Shown in art methods 900 utilize a available processes resource of 16 (12.5%) on two cores in Power8 core In two.Alternatively, if vCPU is distributed to hardware thread when each core has many hardware threads, have and need The VM of high degree of parallelism and the workload of shared memory will be divided by being located on physical core identical with other vCPU Cloth obtains better performance on many physical cores.
In order to avoid as above with reference to many not making the problem of crossing over the vCPU being distributed on physical core described in Figure 14 or The problem of with thread, identifies that potential host and the new paragon of deployment VM are based on following principle, by using hard in cloud VM request Part multithreading parameter considers hardware thread, the principle be the multiple hardware threads that can be assigned in physical core of vCPU without Only physical core core or single hardware thread.This has been come by using the new parameter of hardware multithreading involved in cloud VM request At.For the specific example of this paper, which is referred to as " SMT " parameter, indicates symmetrical multithreading, this is For describing the known term of the hardware multithreading on CPU in this field.With reference to shown in Figure 15, method 1500 is by using cloud VM SMT parameter in request and in one embodiment with the quantity of the hardware thread about each core and separate core setting The related information of host start, to select host computer system (step 1510).Once having selected host computer system System can be used the identical SMT parameter in cloud VM request and determine to make arrangement to vCPU when disposing VM on selected host (step 1520).In one embodiment, when disposing VM on selected host, it is related to the hardware of each core about host The layout that the quantity of thread and the information of separation core setting can be used for making vCPU determines (step 1520).
Figure 10 shows the prior art cpu demand 1010 for cloud VM request comprising required vCPU on specified VM 1020 quantity.The request of the cloud VM according to shown in Figure 16 of disclosure herein and claim 1610 includes vCPU's 1620 Quantity, but also comprise hardware multithreading (SMT) parameter 1630.The value of SMT parameter can be zero, indicate hardware multithreading quilt It closes, or can be the power of number 2 for indicating hardware lines number of passes.Some examples will be shown.With reference to Figure 17, cpu demand 1610A is a suitable examples of cpu demand 1610 shown in Figure 16, and is 2 including 2 vCPU 1720 and with value SMT parameter 1730.For the potential host with 6 core CPU 1410 of Power8 as shown in figure 14, which will have Using a core of two threads, lead to six not used threads, as shown in figure 18.
The second cpu demand 1610B in Figure 19 is another suitable example of cpu demand 1610 shown in Figure 16, and And the SMT parameter including 2 vCPU 1920 and with the instruction pent value of hardware multithreading at 1930.It is more in hardware In the pent situation of thread, for the potential host with 6 core CPU 1410 of Power8 as shown in figure 14, the potential master There are two cores by tool for machine, and a thread is used in each core, leads to 14 not used threads in two cores, such as Figure 20 It is shown.Note that when disabling hyperthread, because the cpu demand 1010 of the prior art does not consider to appoint when disabling host hyperthread What hardware multithreading, thus the net result of the potential host of the cpu demand 1610B in Figure 19 with pent SMT parameter It is identical as prior art cpu demand 1010 shown in Fig. 10.In the prior art, if enable hyperthread, it is unpredictable which The hardware thread of a core will handle vCPU.Consider that hardware multithreading passes through the value specified for hardware multithreading parameter, can subtract The quantity of not used hardware thread on few potential host, and can control and which vCPU run on the same core.
Example in Figure 16-19 shows how hardware multithreading parameter is directed to and how to dispose vCPU and provide more controls. For example, if the operation that the VM with 2 vCPU operates operation to identical data, it may be desirable to 2 vCPU are by portion Two different hardware threads in same core are affixed one's name to, as shown in the example in Figure 17 and 18, this is because transporting on the same core Two capable threads will have very high performance access to the available cache memory of core.If having 2 vCPU's VM will run operation independent of each other, then it may be desirable to 2 vCPU are deployed to two different cores, such as showing in Figure 19 and 20 Shown in example.
Third cpu demand 1610C in Figure 21 is another suitable example of cpu demand 1610 shown in Figure 16, and And the SMT parameter that the value including 8 vCPU 2120 and at 2130 is 8.For with 6 core of Power8 as shown in figure 14 The potential host of CPU 1410, potential host will have a core for using all eight threads, lead to no not used line Journey, as shown in figure 22.
The 4th cpu demand 1610D in Figure 23 is another suitable example of cpu demand 1610 shown in Figure 16, and And the SMT parameter with value 4 including 12 vCPU 2320 and at 2330.For with Power8 6 as shown in figure 14 The potential host of core CPU 1410, there are three cores by tool for potential host, wherein using four threads in each core, lead to 12 Not used thread, as shown in figure 24.These simple examples show how to specify firmly in selection host computer system Part multithreading parameter is provided to how vCPU being deployed to the control of host computer system.
Notice that example shown in Figure 17-24 assumes that vCPU can be deployed to each available thread.Above for Fig. 9- 12 discuss " excessive ratio ", and the quantity of CPU is increased some amount.According to disclosure herein and claim, selection master Similar excessive ratio can be applied in machine computer system.
With reference to Figure 25, method 2500 selects host in specified hardware multithreading parameter, such as the institute of Figure 16,17,19,21 and 23 Show.It reads cloud VM and requests (step 2510).When SMT parameter is OFF (closing) (step 2520=is), selection, which has, meets cloud Host (the step 2530) of multiple physical cores of vCPU quantity in VM request.Note that step 2530 is similar to existing in Fig. 9 Step 920 in technical method 900.When SMT parameter is ON (unlatching) (step 2520=is no), selection, which has, to be met as follows The number of threads satisfaction of host, each core of the host meets in cloud VM in the quantity of quantity and core in SMT parameter VCPU quantity is divided by the Thread Count (step 2540) specified in SMT parameter.This is to select to have to meet in cloud VM request A kind of concrete mode of the host of multiple hardware threads of vCPU quantity.Then method 2500 is completed.
Once having selected host computer system, then the vCPU of VM can be placed on selected host.With reference to figure Figure 26, when vCPU needs to be placed on selected host, method 2600 starts (step 2610).When SMT parameter is to close When closing (step 2620=is), each vCPU is placed on (step 2630) on different physical cores.When SMT parameter is to open (step 2620=is no), vCPU can be placed on (step 2640) on the different hardware thread on same physical core.Then method 2600 complete.
Using the hardware multithreading parameter in cloud VM request, cloud manager not only be must be known by the CPU in potential host Physical core quantity, and must also be noted that potential host in CPU support thread quantity.This is in order to ensure host Single core can support to request specified Thread Count for cloud VM.Referring again to FIGS. 6, this means that determination can in step 610 It not only include that determination can use CPU core, but also determine whether multiple threads on these CPU cores can be used with resource.Realize this A kind of mode of point is: is collected as a part of the available resources in the step 610 in Fig. 6 by host computer system branch The maximum SMT setting held.One example of such case is shown in Figure 27, and wherein max_guest_smt is indicated by the master in Fig. 4 The value that machine MA monitoring agency 440 is collected, to indicate whether potential host computer system supports hardware multithreading.max_guest_ The probable value of smt is 0, indicates that the SMT on host computer system is closed, the power of number 2 indicates available hardware on each CPU Thread Count.When all CPU in host computer system are same types, max_guest_smt will be applied to host The single value of all CPU in computer system.When host computer system includes the difference with the hardware thread of different number When the CPU of type, for each CPU in host computer system, there may be different max_guest_smt values.
OpenStack helps the expectation topology of given client CPU currently without any existing support.Realize this paper institute A kind of mode of the addition of the hardware multithreading parameter of discussion is to add additional specifications to OpenStack flavor, such as Figure 28 institute Show.Parameter powerkvm:smt is the new parameter in OpenStack flavor, the expectation SMT value of given client VM.Note that The probable value of this parameter is -1, is indicated " being indifferent to ";Zero, indicate that SMT is closed;Or some power of number 2.By in max_ It include collecting the ability of the information about the hardware multithreading on potential host and in OpenStack in guest_smt setting Flavor middle finger periodically hopes the ability of the hardware multithreading of rank, this disclosure and claim provide for selecting The potential host computer system for disposing VM thereon and the hardware multithreading ability preferably to utilize host computer system Mode dispose these VM, to improve the utilization rate of the cpu resource on host computer system.
Hardware multithreading parameter discussed in this article can be used for VM and be sized operation.Sometimes virtual machine needs more More or less resource, therefore can be sized to add or reduce the resource for being deployed to virtual machine.VM adjustment is big A kind of small proper method is to specify an OpenStack flavor, to change the resource allocation to VM.It is big when adjusting VM Hour, OpenStack flavor can have powerkvm:smt parameter discussed above and as shown in figure 28.Figure 29 shows Go out for by the adjusted size of method 2900 of VM, VM preferably as shown in Figure 4 to be sized the execution of mechanism 470.It reads Flavor (the step 2910) for being sized operation for existing VM.The flavor quoted in step 2910 is that have to VM New (that is, adjusted size of) resource allocation flavor.When flavor does not include SMT parameter (step 2920=is no), The SMT parameter of VM is arranged to OFF (step 2930).When flavor includes SMT parameter (step 2920=is), when SMT joins When several values is OFF or " being indifferent to " (step 2940=is), the SMT parameter of VM is arranged to OFF (step 2930).When When SMT in flavor has the value of 2 power (step 2940=is no), the SMT parameter of VM is arranged to specify in flavor SMT value (step 2950).Then method 2900 terminates.Method 2900 actually allows " improvement " existing VM to use SMT The mode of parameter.
6 core CPU of Power8 supports daughter nucleus, it means that when enable separate core when, eight threads in each core can be by It is separated into the daughter nucleus of two threads.The 6 core CPU 3010 of Power8 enabled with separation core is shown in FIG. 30, to include quilt Be divided into first core 3020 of four daughter nucleus 3022,3024,3026 and 3028, and be divided into four daughter nucleus 3052,3054, 3056 and 3058 the second core 3050.Four other cores are not shown in Figure 30, but two cores shown in being analogous to.Each daughter nucleus There are two hardware threads for tool, as shown in figure 30.By the way that each core is divided into daughter nucleus, daughter nucleus rather than core can be assigned to void Quasi- machine, this causes to save a large amount of not used hardware threads.
When existing, when separating core host, cloud manager can determine whether the host for VM supports separation core, this will be It is advantageous.With reference to Figure 31, cpu demand 3110 is preferably a part of cloud VM request 510 shown in fig. 5.Cpu demand 3110 is advised Determine the quantity and hardware multithreading parameter 3130 of vCPU 3120.Following examples shows how separation core influences the choosing of host It selects and places vCPU on the host.
In Figure 32, cpu demand 3110A is a suitable examples of cpu demand 3110 shown in Figure 31, and including 2 A vCPU 3220 and the SMT parameter 3230 for being 2 with value.For enabling separation core with as shown in figure 30 having The potential host of 6 core CPU 3010 of Power8, the potential host will have a daughter nucleus using two threads, and causing can be with Three daughter nucleus used by other VM, as shown in figure 33.
The second cpu demand 3110B in Figure 34 is another suitable example of cpu demand 3110 shown in Figure 31, and And the SMT parameter of the value of OFF is set as including 2 vCPU 3420 and at 3430.In the case where closing hardware multithreading, For having the potential host of the 6 core CPU 3010 of Power8 for enabling separation core as shown in figure 30, which will have There are two daughter nucleus, a thread is used in each daughter nucleus, leads to a not used thread in each daughter nucleus, and can be by other Two daughter nucleus that VM is used, as shown in figure 35.
Third cpu demand 3110C in Figure 36 is another suitable example of cpu demand 3110 shown in Figure 31, and And having value including 8 vCPU 3620 and at 3630 is 2 SMT parameter.It is 2 hardware multithreading parameter for having value, For having the potential host of the 6 core CPU 3010 of Power8 of enabling separation core as shown in figure 30, potential host will have four A daughter nucleus uses two threads in each daughter nucleus, causes no thread to be not used, as shown in figure 37.At one in the specific implementation, All eight vCPU can be placed on all eight threads (four daughter nucleus) of same physical core.Certainly, alternatively, Figure 37 In four daughter nucleus can be placed on two or more physical core.
The 4th cpu demand 3110D in Figure 38 is another suitable example of cpu demand 3110 shown in Figure 31, and And the value that has including 12 vCPU 3820 and at 3830 is 2 SMT parameter.For having value to join for 2 hardware multithreading Number, for having the potential host of the 6 core CPU 3010 of Power8 of enabling separation core as shown in figure 30, which will With six daughter nucleus, wherein using two threads in each daughter nucleus, and it is not used without the thread of any daughter nucleus, such as Figure 39 It is shown.At one in the specific implementation, eight in 12 vCPU can be placed on all eight threads (four of same physical core A daughter nucleus) on, and remaining four vCPU can be placed on four threads in two daughter nucleus in different physical cores.When So, in alternative solution, six daughter nucleus in Figure 39 can be placed on three or more physical cores.
All examples of cpu demand in Figure 32,34,36 and 38 are used for the case where host enables separation core.Work as host When separation core with disabling, other than host there will be the separation forbidden setting of core, it is as a result similar to figure 17 above -24 Shown in example.
It is 2 SMT parameter that example in Figure 32,36 and 38, which all has value,.Since Power8 daughter nucleus includes two threads, because This specifies two thread offers most effectively to place vCPU on Power8CPU in SMT parameter.It is noted that SMT parameter can The Thread Count that can be greater than on daughter nucleus.For example, if in Figure 36 3630 at SMT parameter be arranged to value 4, there are two lines for tool The host of the daughter nucleus of journey will be unsatisfactory for requesting.Therefore, when SMT setting is more than the Thread Count for daughter nucleus, do not have separation core Host be not potential host.
With reference to Figure 40, method 4000 shows the selection using the cloud VM host computer system requested, cloud VM request The hardware multithreading parameter of separation core whether is enabled on host computer system including considering.Method 4000 is preferably by scheming Selection of chiller mechanism 450 shown in 4 executes.It reads cloud VM and requests (step 4010).When SMT close (step 4020=is) and Potential host enabled separation core (step 4030=is) when, if potential host have meet cloud VM request in vCPU number Multiple daughter nucleus of amount then can choose potential host (step 4040).When SMT closes (step 4020=is) and potential host When having disabled separation core (step 4030=is no), if potential host has the multiple physics for meeting the vCPU quantity in cloud VM Core then can choose the potential host (step 4050).When SMT is ON (step 4020=is no) and potential host has enabled separation When core (step 4060=is), if the hardware thread number of daughter nucleus can satisfy in SMT parameter specify hardware lines number of passes, And the number of daughter nucleus can satisfy the quantity of the vCPU in cloud VM request divided by the quantity for the hardware thread specified in SMT parameter (step 4070) then can choose potential host.When SMT is ON (step 4020=is no) and point of the potential host with disabling When freestone (step 4060=is no), as the quantity of the hardware thread of fruit stone can satisfy the hardware thread specified in SMT parameter Quantity and if the quantity of physical core can satisfy cloud VM request in vCPU quantity divided by SMT parameter specify it is hard The quantity of part thread then can choose potential host (step 4080).Note that preferably for each potential host repetition methods 4000, this can form potential host pond, and any suitable standard can be used or sound out one come from wherein selection host It is a.For example, can choose the host with the separation core enabled rather than there is the host of disabling separation core, requested at this time in VM Lead to the more efficient use of CPU core and thread.Method 4000 show hardware multithreading parameter and from host about each How the separation core setting of core and the information of hardware thread provide better control to the selection of host computer system, this will lead VCPU is more effectively placed in cause.
As shown in figure 41, filter PowerKVMSMTFilter can be used, so that when the feelings in SMT=4 or SMT=8 When requesting VM under condition, cloud manager filters out the host of enabled separation kernel normal form.This filter can help cloud manager Quickly reduce the potential host number of VM.
With reference to Figure 42, when vCPU needs to be placed on selected host, method 4200 starts (step 4210).Side The vCPU placement mechanism 460 of method 4200 preferably as shown in Figure 4 executes.When SMT closes (step 4220=is) and in host When upper enabling separation core (step 4230=is), vCPU is placed on (step 4240) on different daughter nucleus.When SMT closes (step Rapid 4220=is) and host on disabling separation core when (step 4230=is no), vCPU is placed on different physical cores (step It is rapid 4250).When SMT opens (step 4220=is no) and enables separation core on host (step 4260=is), vCPU quilt It is placed on (step 4270) on the thread of daughter nucleus.When SMT is ON (step 4220=is no) and the disabling separation core on host (step 4260=is no), vCPU is placed on (step 4280) on the thread of same physical core.Note that if the quantity of vCPU is super The number of threads in physical core is crossed, then vCPU will be placed on and provide multiple cores of required amount of hardware thread for all vCPU On.Method 4200 is illustrated how by specifying hardware multithreading parameter and using the information about separation core setting to Framework computing The placement of vCPU in machine system provides more preferable control.
In order to support by considering that separation core in cloud VM request and hardware multithreading parameter place VM, cloud manager It must not only know the quantity of the physical core in the CPU in potential host, and must also be noted that by the CPU branch in potential host Separation core whether is enabled on the quantity for the thread held and potential host.Referring again to FIGS. 6, this means that in step 610 really Whether CPU core can be used by determining available resources not only and include determination, but also can be used including multiple threads on these CPU cores, and Separation core whether is enabled on these cores.Realizing a kind of mode of this point is, as the available money in the step 610 in Fig. 6 The a part in source collects separation core whether is enabled on potential host computer system.
Cloud manager monitors the available resources on host computer system, including is supported on host computer system by CPU Hardware thread quantity, and in one embodiment, including CPU whether enabled separation core.Cloud manager, which receives, to be supplied Answer the request of virtual machine (VM), which includes hardware multithreading parameter, the parameter specify on host computer system whether Allow hardware multithreading (or in one embodiment, the quantity of hardware multithreading required on host computer system). In one embodiment, then cloud manager, which passes through, considers hardware multithreading parameter, selects host computer system for VM.? In another embodiment, then cloud manager passes through the hardware thread and separation core for considering that hardware multithreading parameter, CPU are supported Setting, to select the host computer system for VM.Then VM is placed on selected host meter using hardware multithreading parameter In calculation machine system.The result is that more effectively utilizing the cpu resource in the host of virtual machine.
It will be understood by those skilled in the art that many variations within the scope of the claims are possible.Therefore, although on The disclosure has shown and described in mask body, it will be appreciated, however, by one skilled in the art that in the spirit and model that do not depart from claim In the case where enclosing, these and other changes can be executed in form and details.

Claims (15)

1. a kind of for selecting the device of host computer system, comprising:
At least one processor;
It is coupled to the memory of at least one processor;
The resident cloud manager executed in the memory and by least one described processor, the cloud manager include:
Host MA monitoring agency determines the quantity of the available CPU in multiple main frames computer system that virtual machine can be deployed to, And also determine the hardware thread quantity supported by each CPU on the multiple host computer system;And
Selection of chiller mechanism, receives the virtual machine VM request including virtual cpu quantity and hardware multithreading parameter, and the hardware is more Thread parameter designates whether to need hardware multithreading, and the minimal hardware line of each core is specified when needing hardware multithreading Number of passes, wherein the Selection of chiller mechanism selects to meet the hardware multithreading parameter in the multiple host computer system And including the host computer system with the multiple CPU for meeting the VM request.
2. the apparatus according to claim 1, wherein the first value of the hardware multithreading parameter indicates hardware multithreading quilt It closes and thus does not need hardware multithreading on selected host computer system.
3. the apparatus according to claim 1, wherein the second value of the hardware multithreading parameter indicates selected Framework computing Hardware multithreading is needed in machine system, the numerical value with hardware thread quantity required on selected host computer system.
4. the apparatus according to claim 1, wherein the third value of the hardware multithreading parameter indicates: the master of selection Machine computer system is not consider that hardware thread is on or off on the host computer system of selection In the case of, it is selected.
5. the apparatus according to claim 1, wherein VM request further includes storage requirement and disk demand.
6. the apparatus according to claim 1, wherein the host MA monitoring agency further determines that the multiple Framework computing The amount of the memory on each host computer system in machine system and the amount of disk space.
7. the apparatus according to claim 1, wherein when the virtual cpu quantity in VM request is greater than the hardware When the value of multithreading parameter, the Selection of chiller mechanism selects in the multiple host computer system to include multiple CPU's Host computer system, wherein each of the multiple CPU has multiple hardware threads.
8. a kind of calculating for being used to select the host computer system for disposing virtual machine executed by least one processor The method that machine is realized, which comprises
Determine the quantity of the available CPU in multiple main frames computer system that virtual machine can be deployed to;
Determine the hardware thread quantity supported by each CPU on the multiple host computer system;
The virtual machine VM request including virtual cpu quantity and hardware multithreading parameter is received, the hardware multithreading parameter is specified Hardware multithreading whether is needed, and specifies the minimal hardware Thread Count of each core when needing hardware multithreading;And
It selects to meet the hardware multithreading parameter and including having described in satisfaction in the multiple host computer system The host computer system of multiple CPU of the hardware thread quantity of VM request.
9. according to the method described in claim 8, wherein the first value of the hardware multithreading parameter indicates hardware multithreading quilt It closes and thus does not need hardware multithreading on selected host computer system.
10. according to the method described in claim 8, wherein the second value of the hardware multithreading parameter indicates selected Framework computing Need hardware multithreading in machine system, on selected host computer system required to hardware thread quantity numerical value.
11. according to the method described in claim 8, wherein the third value of the hardware multithreading parameter indicates the described of selection Host computer system is not consider that hardware thread opens or closes on the host computer system of selection In the case of, it is selected.
12. according to the method described in claim 8, wherein the VM request further includes storage requirement and disk demand.
13. according to the method described in claim 8, wherein the host MA monitoring agency further determines that the multiple Framework computing The amount of the memory on each host computer system in machine system and the amount of disk space.
14. according to the method described in claim 8, wherein hard greater than described when the virtual cpu quantity in VM request When the value of part multithreading parameter, the host computer system including multiple CPU in the multiple host computer system is selected, Wherein each of the multiple CPU has multiple hardware threads.
15. a kind of computer-readable medium, including program code are suitable for holding when said program code is run on computers Row method according to any one of claim 8 to 14.
CN201580024412.1A 2014-06-06 2015-06-04 Host is selected using hardware multithreading parameter for virtual machine Active CN106462446B (en)

Applications Claiming Priority (13)

Application Number Priority Date Filing Date Title
US14/297,707 US9400672B2 (en) 2014-06-06 2014-06-06 Placement of virtual CPUS using a hardware multithreading parameter
US14/297,707 2014-06-06
US14/297,677 2014-06-06
US14/297,677 US9384027B2 (en) 2014-06-06 2014-06-06 Selecting a host for a virtual machine using a hardware multithreading parameter
US14/297,724 2014-06-06
US14/297,724 US9304805B2 (en) 2014-06-06 2014-06-06 Provisioning virtual CPUs using a hardware multithreading parameter in hosts with split core processors
US14/305,853 2014-06-16
US14/305,955 US9304806B2 (en) 2014-06-06 2014-06-16 Provisioning virtual CPUs using a hardware multithreading parameter in hosts with split core processors
US14/305,703 US9372705B2 (en) 2014-06-06 2014-06-16 Selecting a host for a virtual machine using a hardware multithreading parameter
US14/305,703 2014-06-16
US14/305,853 US9400673B2 (en) 2014-06-06 2014-06-16 Placement of virtual CPUS using a hardware multithreading parameter
US14/305,955 2014-06-16
PCT/IB2015/054229 WO2015186088A1 (en) 2014-06-06 2015-06-04 Selecting a host for a virtual machine using a hardware multithreading parameter

Publications (2)

Publication Number Publication Date
CN106462446A CN106462446A (en) 2017-02-22
CN106462446B true CN106462446B (en) 2019-10-08

Family

ID=56682546

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201580024412.1A Active CN106462446B (en) 2014-06-06 2015-06-04 Host is selected using hardware multithreading parameter for virtual machine

Country Status (5)

Country Link
JP (1) JP6628742B2 (en)
KR (1) KR101884549B1 (en)
CN (1) CN106462446B (en)
DE (1) DE112015000216T5 (en)
GB (1) GB2541340B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110673928B (en) * 2019-09-29 2021-12-14 天津卓朗科技发展有限公司 Thread binding method, thread binding device, storage medium and server

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8352952B2 (en) * 2008-12-01 2013-01-08 Citrix Systems, Inc. Systems and methods for facilitating virtualization of a heterogeneous processor pool
US9317336B2 (en) * 2011-07-27 2016-04-19 Alcatel Lucent Method and apparatus for assignment of virtual resources within a cloud environment
US8850442B2 (en) * 2011-10-27 2014-09-30 Verizon Patent And Licensing Inc. Virtual machine allocation in a computing on-demand system
JP5503678B2 (en) 2012-03-07 2014-05-28 株式会社Nttドコモ Host providing system and host providing method
JP5813554B2 (en) 2012-03-30 2015-11-17 ルネサスエレクトロニクス株式会社 Semiconductor device
JP5894496B2 (en) 2012-05-01 2016-03-30 ルネサスエレクトロニクス株式会社 Semiconductor device
US20140007097A1 (en) * 2012-06-29 2014-01-02 Brocade Communications Systems, Inc. Dynamic resource allocation for virtual machines

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
Best Practices-Core allocation,Jeff Savit;Jeff Savit;《https://blogs.oracle.com/virtualization/best-practices-core-allocation-v2》;20120609;正文introduction、What to do about it *
Five SQL Server Settings to Change;Jeremiah Peschka;《https://www.brentozar.com/archive/2013/09/five-sql-server-settings-to-change/》;20130904;1-3 *
Hyper-Threading Gotcha with Virtual Machine vCPU Sizing;CHRIS WAHL;《http://wahlnetwork.com/2013/09/30/hyper-threading-gotcha-virtual-machine-vcpu-sizing/》;20130930;正文Over Sizing the vCPUs in a Virtual Machin *
Hyper-V Virtual CPUs Explained;Eric Siron;《https://www.altaro.com/hyper-v/hyper-v-virtual-cpus-explained/》;20140207;1-12 *

Also Published As

Publication number Publication date
JP2017520830A (en) 2017-07-27
CN106462446A (en) 2017-02-22
DE112015000216T5 (en) 2016-09-01
GB201620786D0 (en) 2017-01-18
KR101884549B1 (en) 2018-08-29
GB2541340B (en) 2017-07-19
JP6628742B2 (en) 2020-01-15
KR20160147958A (en) 2016-12-23
GB2541340A (en) 2017-02-15

Similar Documents

Publication Publication Date Title
CN104205056B (en) Across the method and system of the dynamically distributes of the live load of multiple clouds
CN103365725B (en) Method and system for dynamic allocation of workload deployment units across a plurality of clouds
CN104424013B (en) The method and apparatus for disposing virtual machine in a computing environment
US9619274B2 (en) Provisioning virtual CPUs using a hardware multithreading parameter in hosts with split core processors
US9639390B2 (en) Selecting a host for a virtual machine using a hardware multithreading parameter
CN105335207B (en) Method and apparatus for managing virtual machine instance
CN106385329B (en) Processing method, device and the equipment of resource pool
CN110580197A (en) Distributed computing architecture for large model deep learning
US10241834B2 (en) Bandwidth aware resource optimization
US10282208B2 (en) Cognitive thread management in a multi-threading application server environment
US10671438B2 (en) Providing additional memory and cache for the execution of critical tasks by folding processing units of a processor complex
US10176017B2 (en) Configuration management for a shared pool of configurable computing resources
CN108701132A (en) Resource management system and method
JP2023101462A (en) Computer implementation method, system, and computer program (data locality for big data on kubernetes)
JP2023052534A (en) Method for implementing dynamically and automatically altering user profile for enhanced performance, computer system, and program
CN106462446B (en) Host is selected using hardware multithreading parameter for virtual machine
US10409762B2 (en) Remote direct memory access-based on static analysis of asynchronous blocks
US9619294B2 (en) Placement of virtual CPUs using a hardware multithreading parameter
US10643228B2 (en) Standardizing customer and test data and information collection for run time and historical profiling environments and workload comparisons
Quintero et al. IBM Technical Computing Clouds
WO2015186088A1 (en) Selecting a host for a virtual machine using a hardware multithreading parameter
CN112925630A (en) Submission management method, device, equipment and medium for artificial intelligence task
Rana et al. Study and Implementation of Resource Allocation Algorithms in Cloud Computing

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant