Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L47/00—Traffic control in data switching networks
  • H04L47/70—Admission control; Resource allocation
  • H04L47/80—Actions related to the user profile or the type of traffic
  • H04L47/805—QOS or priority aware
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units
  • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
  • G06F9/46—Multiprogramming arrangements
  • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
  • G06F9/5061—Partitioning or combining of resources
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units
  • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
  • G06F9/44—Arrangements for executing specific programs
  • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
  • G06F9/45533—Hypervisors; Virtual machine monitors
  • G06F9/45558—Hypervisor-specific management and integration aspects
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units
  • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
  • G06F9/46—Multiprogramming arrangements
  • G06F9/48—Program initiating; Program switching, e.g. by interrupt
  • G06F9/4806—Task transfer initiation or dispatching
  • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
  • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units
  • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
  • G06F9/44—Arrangements for executing specific programs
  • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
  • G06F9/45533—Hypervisors; Virtual machine monitors
  • G06F9/45558—Hypervisor-specific management and integration aspects
  • G06F2009/45575—Starting, stopping, suspending or resuming virtual machine instances
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units
  • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
  • G06F9/44—Arrangements for executing specific programs
  • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
  • G06F9/45533—Hypervisors; Virtual machine monitors
  • G06F9/45558—Hypervisor-specific management and integration aspects
  • G06F2009/45579—I/O management, e.g. providing access to device drivers or storage
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units
  • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
  • G06F9/44—Arrangements for executing specific programs
  • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
  • G06F9/45533—Hypervisors; Virtual machine monitors

Definitions

• Sharing resources for networked computers can increase efficiency by reducing maintenance and operating costs, allowing flexibility with respect to individual resource usage, and simplifying resource management.
• the benefits include data consolidation, universal access to data, ease of storage management, and support for live migration of virtual machines (VMs) for virtualized environments.
• QoS Quality of Service
• the policy may guarantee a minimum and/or maximum level of service (e.g., as a percentage of shared resource).
• QoS suggests an ability to evenly distribute services or arbitrarily assign priority to selected applications, users, or data flows to maintain control over workload performance in shared storage environments.
• a system and method for providing Quality of Service (QoS) for clients running on host computers to access a common resource uses a resource pool module and a local scheduler in at least one of the host computers.
• the resource pool module operates to compute an entitlement of each client for the common resource based on a current capacity for the common resource and demands of the clients for the common resource.
• the resource pool module operates to assign a portion of the computed current capacity for the common resource to a particular host computer using the computed entitlement of each client running on the particular host computer.
• the local scheduler operates to allocate the portion of the computed current capacity among the clients running on the particular host computer.
• a method for providing QoS for clients running on host computers to access a common resource comprises computing a current capacity for the common resource based on a global average latency for accessing the common resource by the clients, computing an entitlement of each client for the common resource based on the computed current capacity and demands of the clients for the common resource, assigning a portion of the computed current capacity for the common resource to a particular host computer using the computed entitlement of each client running on the particular host computer, and allocating the portion of the computed current capacity among the clients running on the particular host computer.
• the steps of this method are performed when program instructions contained in a computer-readable storage medium is executed by one or more processors of the host computers.
• a system in accordance with an embodiment of the invention comprises at least one processor, a plurality of clients operably connected to the at least one processor, a resource interface with a host queue to store requests from the clients to access a common resource, a resource pool module operably connected to the at least one processor, and a scheduler operably connected to the resource pool module.
• the resource pool module comprises a first component configured to compute a current capacity for the common resource based a global average latency for accessing the common resource by the clients, a second component configured to compute an entitlement of each client for the common resource based on the computed current capacity and demands of the clients for the common resource, and a third component configured to assign a portion of the computed current capacity for the common resource to a host computer using the computed entitlement of each client.
• the scheduler is configured to allocate the portion of the computed current capacity among the at least one client running on the host computer.
• FIG. 1 is a block diagram of a network computer system in accordance with an embodiment of the invention.
• FIG. 2 is a block diagram of a host computer of the network computer system of Fig. 1 in accordance with an embodiment of the invention.
• Fig. 3 is a diagram of virtual machines (VMs), host computers and a storage of the network computer to illustrate different groups of VMs in accordance with an embodiment of the invention.
• VMs virtual machines
• host computers host computers
• storage of the network computer to illustrate different groups of VMs in accordance with an embodiment of the invention.
• Fig. 4 is a diagram of a resource pool hierarchical structure with VMs in accordance with an embodiment of the invention.
• Fig. 5 is a block diagram of a storage resource pool (SRP) module included in a host computer in accordance with an embodiment of the invention.
• SRP storage resource pool
• Fig. 6 is another diagram of the resource pool hierarchical structure shown in Fig. 4.
• Fig. 7 is a block diagram that shows a resource pool hierarchical structure being split based on different datastores in accordance with an embodiment of the invention.
• Fig. 8 is a flow diagram of a method for providing quality of service (QoS) for clients running on host computers to access a common resource in accordance with an embodiment of the invention.
• QoS quality of service
• QoS Quality of Service
• a network computer system 100 in accordance with an embodiment of the invention is shown.
• the network computer system includes a network 102, a number of host computers 104A,
• each of the host computers 104 is able to access the shared storage via the network and share the resource provided by the storage with the other host computers. Consequently, any process running on any of the host computers can also access the storage via the network.
• the host computers in a distributed manner implement a demand-based QoS mechanism to maintain control over workload performance with respect to storage resource being shared by the host computers.
• the network 102 can be any type of computer network or a combination of networks that allows communications between devices connected to the network.
• the network 102 may include the Internet, a wide area network (WAN), a local area network (LAN), a storage area network (SAN), a fibre channel network and/or other networks.
• the network 102 may be configured to support protocols suited for communications with storage arrays, such as Fibre Channel, iSCSI, FCoE and HyperSCSI.
• the host computers 104A, 104B...104N are physical computer systems that hosts or supports one or more clients so that the clients are executing on the physical computer systems.
• the host computers may be servers that are commonly found in data centers.
• client is any software entity that can run on a computer system, such as a software application, a software process or a virtual machine (VM).
• VM virtual machine
• the storage 106 is used to store data for the host computers 104A, 104B...104N, which can be accessed like any other storage device connected to computer systems.
• the storage can be accessed by entities, such as clients running on the host computers, using any file system, e.g., virtual machine file system (VMFS) or network file system (NFS).
• VMFS virtual machine file system
• NFS network file system
• the storage includes one or more computer data storage devices 108, which can be any type of storage devices, such as solid-state devices (SSDs), hard disks or a combination of the two.
• SSDs solid-state devices
• the storage devices may operate as components of a network- attached storage (NAS) and/or a storage area network (SAN).
• the storage includes a storage managing module 110, which manages the operation of the storage.
• the storage managing module maintains a request queue 112, which is a list of pending input/output (IO) request for the storage.
• the storage managing module 110 is a computer program executing on one or more computer systems (not shown) of the storage.
• the storage may support multiple data stores or logical unit numbers (LUNs).
• LUNs logical unit numbers
• the storage 106 can be any type of computer data storage, the storage 106 will be described herein as being a storage array.
• the host computer 104A is configured to support a number of clients 220A, 220B...220N, which are VMs.
• the number of VMs supported by the host computer can be anywhere from one to more than one hundred.
• the exact number of VMs supported by the host computer is only limited by the physical resources of the host computer.
• the VMs share at least some of the hardware resources of the host computer, which include system memory 222, one or more processors 224, a storage interface 226, and a network interface 228.
• the system memory 224 which may be random access memory (RAM), is the primary memory of the host computer.
• the processor 224 can be any type of a processor, such as a central processing unit (CPU) commonly found in a server.
• the storage interface 226 is an interface that allows that host computer to communicate with the storage array 106.
• the storage interface may be a host bus adapter or a network file system interface.
• the network interface 228 is an interface that allows the host computer to communicate with other devices connected to the network 102.
• the network interface may be a network adapter.
• the VMs 220 A, 220B ...220N run on top of a virtual machine monitor 230, which is a software interface layer that enables sharing of the hardware resources of the host computer 104A by the VMs.
• a virtual machine monitor 230 which is a software interface layer that enables sharing of the hardware resources of the host computer 104A by the VMs.
• one or more of the VMs can be nested, i.e., a VM running in another VM.
• one of the VMs may be running in a VM, which is also running in another VM.
• the virtual machine monitor may run on top of the host computer's operating system or directly on hardware of the host computer.
• the virtual machine monitor runs on top of a hypervisor that is installed on top of the hardware components of the host computer.
• Each VM includes a guest operating system 232 and one or more guest applications 234.
• the guest operating system is a master control program of the respective VM and, among other things, the guest operating system forms a software platform on top of which the guest applications run.
• the VMs 220A, 220B...220N are able to communicate with other computer systems connected to the network using the network interface 228 of the host computer 104A.
• the VMs are able to access the storage array 106 using the storage interface 226 of the host computer.
• the VMs of the host computer compete for the shared storage resource provided by the storage array for the host computer.
• the host computer competes with other host computers 104B...104N for the shared storage resource.
• Each of the host computers 104A, 104B...104N of the network computer system 100 is allowed to keep a certain maximum number of IO requests outstanding at the storage array 106 in an issue queue 236 of the storage interface 226 of that host computer, as illustrated in Fig. 2.
• the size of the issue queue (also referred to herein as "host queue depth") for a particular host computer reflects the capacity of the storage array to process IO request that is currently allocated to that particular host computer.
• the issue queues in the host computers are used to implement QoS control with respect to the storage resource provided by the storage array.
• the shared common resource i.e., the shared storage resource provided by the storage array 106
• a QoS management mechanism in the network computer system 100 to control distributions of the shared storage resource among the different entities, such as the VMs hosted by the host computers 104A, 104B...104N.
• the process of distributing the shared storage resource may be straightforward.
• some of the VMs may need greater amount of the shared storage resource than other VMs.
• an amount of the shared storage resource may be measured in IO operations per second (IOPS), wherein a higher IOPS value means greater access to the shared storage resource.
• IOPS IO operations per second
• the needs of the different VMs may vary based on changes in the demands of the VMs for the shared storage resource.
• the VMs running on different host computers may belong to different groups, which have needs and requirements with respect to access to the shared storage resource. An example of such groups of VMs is described below with reference to Fig. 3.
• Fig. 3 shows the host computers 104A and 104B connected to the storage array 106 to share the storage resource provided by the storage array.
• the host computer 104A includes the VMs 220A and 220B.
• the host computer 104B includes the VM 220C and 220D.
• the VM 220A running on the host computer 104A and the VM 220C running on the host computer 104B belong to the sales division of an enterprise.
• the VM 220B running on the host computer 104A and the VM 220D running on the host computer 104B belong to the finance division of the enterprise.
• the VMs 220A and 220C of the sales division may be handling sales in different continents, and thus, need an overall reservation of 1,000 IOPS based on the peaks and troughs of demand in the different time zones.
• the VMs 220B and 220D of the finance division may be running background data analytics, and thus, are restricted to a combined throughput of 500 IOPS to reduce their impact on the critical sales VMs.
• someone may want to allocate the 500 IOPS in ratio 1:2 between the VMs based on their importance. This is known as shares control.
• the QoS management mechanism of the network computers system 100 in accordance with embodiments of the invention is designed to provide a robust QoS control of the shared storage resource to address the requirements of different groups of VMs without the need to have a centralized resource scheduler, which can add complexity to the QoS mechanism and can increase susceptibility to system- wide failures.
• the QoS management mechanism of the network computer system uses a concept of storage resource pools (SRP) to manage QoS for clients distributed throughout the network computer system.
• SRP storage resource pools
• the SRP-based QoS management mechanism allows a user, such as a system administrator, to specify the desired QoS using throughput reservation values (lower bounds), limit values (upper bounds) and shares (proportional sharing). These values may be set for any node of a resource pool hierarchical structure, such as individual VMs in the resource pool hierarchical structure and/or groups of related VMs, as conceptually designated by nodes in the resource pool hierarchical structure that are situated at a higher level than the VMs.
• the reservation values are absolute guarantees that specify the minimum amount of the shared resource that the nodes, e.g., VMs and groups of VMs, in the resource pool hierarchical structure must receive.
• the limit values specify the maximum allocation that should be made to the nodes in the resource pool hierarchical structure. These values are useful for enforcing strict isolation and restricting tenants for contractually- set IOPS based on their service level objectives (SLOs).
• SLOs service level objectives
• the shares provide a measure of relative importance between the nodes in the resource pool hierarchical structure, and are used to prioritize allocation when capacity is constrained.
• the SRP-based QoS management mechanism also allows the user to group the clients running on the host computers 104A, 104B...104N in the network computer system 100 into storage resource pools (i.e. SRPs) so that the clients in a particular group or SRP can be treated as a single unit for resource allocation. These units can then be aggregated into larger resource pools or groups to create a resource pool hierarchical structure. The grouping of the clients can be made regardless of the underlying host computers on which the clients are running. Thus, clients running on a particular host computer may belong to different resource pools or groups. Such distributed architectures are very common in virtualized datacenters.
• the information defining the resource pool hierarchical structure may be stored in a shared file stored in the storage array 106 so that every host computer in the network computer system is able to access this information.
• the resource pool hierarchical structure information may be broadcasted to other host computers in the network computer system so that every host computer has these values from all the other host computers.
• FIG. 4 An example of a resource pool hierarchical structure with the VMs 220A, 220B, 220C and 220D is illustrated in Fig. 4.
• the resource pool hierarchical structure shown in Fig. 4 includes the four VMs 220A, 220B, 220C and 220D, which can be viewed as being nodes in the lowest level of the resource pool hierarchical structure.
• the VMs 220A and 220C are grouped together, as illustrated by a node 402A, which can be viewed as the parent node of the two VMs 220A and 220C.
• the two VMs 220A and 220C can be viewed as the children or child nodes of the node 402A.
• the VMs 220B and 220D are grouped together, as illustrated by another node 402B, which can be viewed as the parent node of the two VMs 220B and 220D.
• the two VMs 220B and 220D can be viewed as the children or child nodes of the node 402B.
• the two nodes 402A and 402B are further grouped together, as illustrated by a node 404, which is the root node of the resource pool hierarchical structure.
• the node 404 can also be viewed as the parent node of the two nodes 402A and 402B, and conversely, the two nodes 402A and 402B can be viewed as the children or child nodes of the node 404.
• This resource pool hierarchical structure may conceptually represent an organizational structure, such as a business enterprise with divisions or departments that use one or more VMs for operation. If representing a business enterprise, the root node 404 of the resource pool hierarchical structure may represent the entire business enterprise, and the two nodes 402A and 402B may represent divisions or departments of the enterprise, such as sales and financial divisions, respectively, where the VMs 220 A and 220C operate for the sales division and the VMs 220B and 220D operate for the financial division.
• the SRP-based QoS management mechanism uses a storage resource pool (SRP) module 238 and a local scheduling module 240, which are included in each host computer in the network computer system 100, as illustrated in Fig. 2.
• the SRP module in each host computer cooperatively operates with the SRP modules in the other host computers of the network computer system to determine how much of the capacity of the storage array 106 should be provided to that host computer, which is at least based on aggregate demand on the storage array by clients in the host computer and average latency of the storage array.
• the SRP module determines how much of the storage capacity allocated to the host computer should be provided to each client, e.g., each VM, in the host computer.
• the SRP module also distributes a global reservation value, a global limit value and shares at the root node of a resource pool hierarchical structure down to the clients based on their current individual demands of the shared storage resource, their static reservation, limit and share values.
• a share value is equivalent to the number of assigned shares.
• static values are those that are set by a user, such as a system administrator, or a managing program running on any computer in the network computer system 100. These static values may be stored in a shared file stored in the storage array 106 so that every host computer in the network computer system is able to access this information. Alternatively, these static values may be broadcasted to other host computers in the network computer system so that every host computer has these values from all the other host computers.
• each client is assigned a dynamic reservation value, a dynamic limit value and a dynamic share value for the current monitoring time interval.
• These dynamic values, as well as the allocations of the storage capacity to the clients, are then recalculated for each subsequent monitoring time interval.
• the local scheduler 240 in each host computer operates to schedule the 10 requests by the clients, e.g., the VMs, in that host computer in accordance with the dynamic reservation values, the dynamic limit values and the dynamic share values, which were computed by the SRP module 238 in the host computer.
• the local scheduler and the SRP module are illustrated in Fig. 2 as being separate from the virtual machine monitor 230, one or both of these components may be implemented as part of the virtual machine monitor.
• the SRP module and the local scheduler are implemented as software programs running on the host computer. However, in other embodiments, the SRP module and the local scheduler may be implemented using any combination of software and hardware.
• the SRP module includes a demand updating component 502, a storage queue depth updating component 504, a storage IOPS capacity computing component 506, a divvying component 508, and a host queue depth adjusting component 510.
• these components of the SRP module are shown as being distinct elements. However, in other embodiments, one or more of these components may be combined with other components and/or one or more of these components may be further divided into sub-components.
• the SRP module is implemented as a software module
• the components of the SRP module can be viewed as processing blocks of the software module.
• the clients in the host computer 104A are described as being VMs. However, as noted above, these clients can be any entities that can access the storage array 106 for the shared storage resource.
• the resource demand updating component 502 of the SRP module 238 operates to update the demand of each VM in the host computer 104A for the shared storage resource and the aggregated VM demand for the host computer, i.e., the sum of the demands of all the VMs in the host computer.
• the resource demand updating component determines the average latency ("avgLatency") for the host computer and the average measured IOPS ("avglops") using statistics maintained by the host computer, e.g., by the virtual machine monitor 230 or a hypervisor running on the host computer. These statistics maintained by the host computer include statistics on the aggregated latency and the total number of IOs performed by each VM of the host computer during a monitoring interval.
• the resource demand updating component then computes the demand for each VM in the host computer in terms of average number of outstanding IOs ("demandOIO”) using the following equation derived from Little's law:
• VM demand values are then made available so that every host computer in the network computer system 100 can get these VM demand values in terms of outstanding IOs (OIOs).
• these values are updated in a shared file stored in the storage array 106.
• every host computer in the network computer system is able to access the shared file to retrieve the demandOIO values for other host computers in the network computer system.
• these values may be broadcasted to other host computers in the network computer system so that every host computer has these values from all the other host computers.
• the resource demand updating component 502 then converts the demandOIO value to a normalized demand IOPS value ("demandlops") based on the storage device congestion threshold latency (“L c ”) using the following equation:
• the congestion threshold is the maximum latency at which the storage device is operated.
• the resource demand updating components controls the storage queue depth, i.e., the depth of the request queue 112 (shown in Fig. 1), to keep the latency close to L c , so that the storage array 106 is utilized in an efficient manner. This helps to avoid overestimating the demand of a VM based on local latency variations.
• the congestion threshold can be typically set to 30 milliseconds.
• L c can be set to a lower value, e.g., 5 to 10 milliseconds.
• the resource demand updating component 502 then adjusts the demandlops value to make sure that the value lies within the lower and upper bounds represented by reservation and limit settings for each VM using the following equation:
• the demand is then aggregated for the host computer by summing the demandlops values of the VMs and then applying the bound check at the host computer 104A to make sure that the aggregated value lies within the lower and upper bounds represented by reservation and limit settings for the host computer.
• the storage queue depth updating component 504 of the SRP module 238 operates to update the capacity of the storage array in terms of the storage queue depth of the storage array 106, which is then allocated to each host computer in the network computer system 100, including the host computer 104A in which the SRP module is operating.
• the storage queue depth updating component adjust the storage queue depth to keep the measured latency within the congestion threshold using the following equation:
• Q(t) denotes the storage queue depth at time t
• L(t) is the current average latency for all the host computers
• Lc is the device congestion threshold
• the storage IOPS capacity computing component 506 of the SRP module 238 operates to compute the IOPS capacity of the storage array 106.
• the storage IOPS capacity computing component converts the updated array queue depth value, which was computed by the storage queue depth computing component 504, to an equivalent storage IOPS capacity using the following equation derived using Little's Law:
• the conversion from queue depth to IOPS is done because the resource pool settings used in the divvying operation performed by the divvying component 508, as described below, are in terms of user-friendly IOPS, rather than the less transparent OIO values.
• the divvying component 508 of the SRP module 238 operates to compute dynamic reservation, limit and share values for the VMs that reflect the current demand distribution, as well as the entitlements of the VMs with respect to the computed arraylOPS value.
• the divvying component takes as input the structure of a resource pool hierarchical structure, the static reservation, limit and shares settings on nodes of the resource pool hierarchical structure (e.g., the nodes 402A, 402B and 404 shown in Fig. 4), as well as the demands of the VMs and the nodes.
• the divvying component then performs operations to distribute the reservation, limit, array IOPS and share values at the root node the resource pool hierarchical structure down to the VMs.
• the root node of a resource pool hierarchical structure holds four resource types that need to be divided or distributed among the nodes of the resource pool (RP) hierarchical structure:
• the divvying component 508 does a level-by-level pass of the resource pool hierarchical structure to divide the resources at each level of the resource pool hierarchical structure beginning with the root node. For each node of the resource pool hierarchical structure, the divvying component divides up the resources of the node among its children or child nodes.
• R-divvy, L-divvy, /-divvy and S-divvy operations are operations performed by the divvying component to distribute the R, L, I and S values, respectively.
• R, L, I and S values at the root node of the resource pool hierarchical structure will sometimes be referred to herein as global R, L, I and S values.
• the resulting R, L, S values for the VMs after the R-divvy, L-divvy and S-divvy operations are used as the dynamic R, L, S settings for the VMs during the next monitoring time interval.
• the value of / obtained per VM as part of /-divvy is known as the entitlement of the VM.
• the limits of the nodes to receive shares of the R, L and / values are temporarily capped at their aggregated demands, which allows the resources to be directed to VMs that currently have higher demands.
• the divvying component 508 will first divvy the reserved RP capacity R at the root node among its children or child nodes. At each child node, its allocated reservation is used as the capacity to divvy among its children. This process is repeated until all the VMs of the network computer system 100 have received their updated share of R.
• the divvying component follows a similar procedure to divvy the RP limit L and the array IOPS / so that each VM receives a new dynamic limit setting and entitlement
• the divvying component will divvy the total RP shares S at the root node among its children or child nodes based on the static share values of the child nodes. At each child node, its allocated shares are then divided among its children in the ratio of the children's share settings.
• the divvying component 508 performs the R-divvy, L-divvy, /-divvy and S-divvy operations to try to give each child node a portion of the parent capacity in proportion to its shares, subject to their reservation and limit constraints.
• One algorithm to accomplish this goal is to serially give a small fixed amount of the parent capacity to a selected child node until the entire parent capacity has been distributed to the children. To illustrate this algorithm, let ⁇ 3 ⁇ 4 denote the allocation made to a child i at some stage of the divvying process, and 3 ⁇ 4 be its share value.
• the divvying process first gives each child node its reservation, i.e., the initial value of ⁇ 3 ⁇ 4 ⁇ is the static reservation value of the child i. For the next quanta of the resource, the divvying process chooses the child node with the smallest normalized allocation ( ⁇ 3 ⁇ 4 3 ⁇ 4) among the children that are below their static limit value, and increases its allocation by a small amount ⁇ . The process continues until the entire parent capacity has been divvied out.
• a concern with this algorithm is that it has a runtime of 0(log n * capacity/ ⁇ ) for n VMs, which can be quite high for large capacity values. Another problem is to come up with a good value of ⁇ .
• other distribution algorithm can be employed by the divvying component to divide the resources of a parent node to its child nodes in a more efficient manner.
• one distribution algorithm that can be employed by the divvying component 508 for R-divvy, L-divvy and /-divvy operations involves using the demand of a node as its temporary limit (/) value during the distribution process, while its r and s values are the static reservation and share values, respectively. If the sum of the demands of the child nodes is smaller than the capacity being divvied at the parent, the static limits of the child nodes are used instead of their demands.
• the reservation set (R) at the root node is used as the capacity to divvy, while for the L-divvy and /-divvy operations, the capacities are the root limit setting (L) and the array IOPS (/), respectively.
• the parent's share value is simply divided in the ratio of the children's shares.
• the above algorithm has a runtime of 0(n*log n) for n VMs, bounded by the time to create the sorted sequence V. At the end of the process, some children would have been capped at their limit (LB set), some would not have received any allocation beyond their reservation (RB set), and the rest would have received allocation in proportion to their shares (PS set).
• FIG. 6 shows the same resource pool hierarchical structure depicted in Fig. 4.
• the static reservation, limit and share values for each node of the resource pool hierarchical structure are shown.
• the computed demands of the VMs 220A, 220B, 220C and 220D are shown.
• the results of the divvying process i.e., the dynamic reservation, limit and share values, are shown for the nodes 402 A and 402B and the VMs.
• the tuple U denotes static settings or values
• the tuple D denotes the dynamic divvy results for the reservation, limit and share values.
• the efficient distribution algorithm described above is used for the divvying process.
• the divvying component 508 uses the VM demands updated by the resource demand updating component 502 as temporary caps on the limit settings at the nodes of the resource pool hierarchical structure. Since the demands on the VMs are 600, 400, 400 and 100, respectively, the temporary limit caps on the VMs are set to 600, 400, 400 and 100, respectively.
• the divvying component also aggregates the VM demands to get the demand values for the nodes 402A and 402B. In this example, the aggregate demands for the nodes 402A and
• the temporary limits caps on the nodes 402A and 402B are set to 1,000 and 500 respectively.
• the divvying component 508 then proceeds level-by-level down from the root node 404 to the VMs 220 A, 220B, 220C and 220D to divvy the parent reservation among its children.
• the reservation value R which has been set to 1,200 by a user, is divvied between the nodes 402A and 402B in the ratio of their shares (3: 1), resulting in allocations of 900 and 300, respectively. Since these values lie between the reservation and limit values for the nodes 402A and 402B, these are the final results of the R-divvy operation at the root node.
• the divvying component 508 similarly divides the limit values of the parents among their children, level-by-level.
• the allocation to the node 402B is capped at its limit setting of 500, which results in allocations of 1,800 and 500 to the nodes 402A and 402B, respectively.
• the VM 220D is given 100, while the VM 220B gets the remaining amount of 400.
• the divvying component 508 simply divides the shares at a parent node among its child nodes in the ratio of their shares.
• the VMs 220B and 220D have identical static settings. However, due to the difference in their demands, the resulting dynamic settings are different for the VMs 220B and 220D. With respect to the VMs 220A and 220C, excess reservation was given to the VM 220C over the VM 220A since the VM 220A has a higher share value. However, to meet the user-set reservation for the VM 220A, the VM 220C received less than twice the reservation of the VM 220A.
• the host queue depth adjusting component 510 of the SRP module 238 operates to compute a new host queue depth value, i.e., the depth of the issue queue 236, based on the entitlements of the VMs 220A, 220B...220N in the host computer 104A with respect to the array IOPS, which were computed by the divvying component 508.
• the host queue depth adjusting component computes the new host queue depth value using the following equation to adjust the host queue depth:
• Q(t + 1) is the array queue depth value
• arraylOPS is the array IOPS capacity
• Ei is the entitlement of a VM in the host computer.
• the local scheduler 240 operates to allocate the share of the array capacity for the host computer 104A, i.e., the new host queue depth value computed by the host queue depth adjusting component 510 of the SRP module 238, among its VMs 220A, 220B...220N.
• the local scheduler uses the dynamic VM reservations, limits and shares settings computed by the SRP module to schedule the IO requests from the VMs.
• the local scheduler enforces the limit defined by the new host queue depth value on the total number of outstanding IOs at the host computer.
• the local scheduler is the mClock scheduler described in "mClock: Handling Throughput Variability for Hypervisor IO Scheduling" by Ajay Gulati, Arif Merchant and Peter Varman.
• any IO scheduler that can schedule IO requests of VMs in a host computer using the dynamic VM reservation, limit and shares settings computed by the SRP module, while abiding by the limit defined by the host queue depth value, can be used as the local scheduler.
• each host computer in the network computer system 100 is able to independently allocate a portion of the total capacity of the storage 106 to itself based on the average latency of the storage and manage the allocated storage resource among the clients running on that host computer using the computed dynamic reservation, limit and share values, which are computed based on the demands of the clients for the shared storage resource.
• a centralized QoS manager/scheduler is not required for the network computer system to efficiently allocate the shared storage resource.
• the clients running on the host computers 104A may be any suitable software and/or firmware.
• 104B...104N may include sub-components that also require the shared storage resource. Thus, in these embodiments, these sub-components may be considered as "clients" that consume the shared storage resource.
• a VM running on one of the host computers may be associated with one or more virtual machine files, such as virtual machine disk (VMDKs), which are stored in the storage 106.
• VMDKs of VMs consume the shared storage resource, and thus, may be assigned reservation, limit and share values to efficiently share the resource.
• the VMDKs of VMs are also included in a resource pool hierarchical structure, and considered by the SRP module 238 and the local scheduler 240 of each host computer for QoS control.
• VMDKs is shown in Fig. 7.
• the hierarchical structure 700 includes a root node 702, nodes 704A and 704B, VMs 706A, 706B, 706C, 706D and 706E, and VMDKs 708A, 708B, 708C, 708D, 708E, 708F, 708G and 708H.
• the SRP module 238 and the local scheduler 240 of each host computer would simply distribute the capacity of the storage 106 and the global reservation, limit and share values assigned to the root node 702 down to the VMDKs in the manner described above.
• the VMDKs may be stored in different datastores.
• the VMDKs 708A, 708B, 708D, 708E, and 708H may be stored in a datastore 1 and the VMDKs 708C, 708E and 708F may be stored in a datastore 2.
• the SRP module in each of the host computers 104A, 104B...104N may be configured to split the resource pool hierarchical structure into per datastore resource pool hierarchical structure using datastore information, which may be provided by a user.
• the resource pool hierarchical structure 700 may be split into resource pool hierarchical structure 750A and 750B, which correspond to the datastores 1 and 2, respectively.
• the SRP module in each of the host computers will then operate on each of the per datastore resource pool hierarchical structures 750A and 750B in the manner described above to provide QoS control.
• a method for providing quality of service (QoS) for clients running on host computers to access a common resource in accordance with an embodiment of the invention is described with reference to a flow diagram of Fig. 8.
• computing a current capacity for the common resource is computed based a global average latency for accessing the common resource by the clients.
• an entitlement of each client for the common resource is computed based on the computed current capacity and demands of the clients for the common resource.
• a portion of the computed current capacity for the common resource is assigned to a particular host computer using the computed entitlement of each client running on the particular host computer.
• the portion of the computed current capacity is allocated among the clients running on the particular host computer.
• operations may be implemented in an intermittent and/or alternating manner.
• an embodiment of a computer program product includes a computer useable storage medium to store a computer readable program that, when executed on a computer, causes the computer to perform operations, as described herein.
• embodiments of at least portions of the invention can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system.
• a computer-usable or computer readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
• the computer-useable or computer-readable medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device), or a propagation medium.
• Examples of a computer-readable medium include a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disc, and an optical disc.
• Current examples of optical discs include a compact disc with read only memory (CD-ROM), a compact disc with read/write (CD-R/W), a digital video disc (DVD), and a Blue-ray disc.

Landscapes

• Engineering & Computer Science (AREA)
• Software Systems (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Computer Networks & Wireless Communication (AREA)
• Signal Processing (AREA)
• Computer And Data Communications (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Publications (1)

Family

ID=49671702

Family Applications (1)

Country Status (5)

Cited By (3)

Families Citing this family (40)

Citations (5)

Family Cites Families (21)

• 2012
  • 2012-05-31 US US13/485,615 patent/US9244742B2/en active Active
• 2013
  • 2013-05-30 JP JP2015515211A patent/JP6017684B2/ja active Active
  • 2013-05-30 EP EP13796817.8A patent/EP2856721A4/en not_active Withdrawn
  • 2013-05-30 WO PCT/US2013/043473 patent/WO2013181464A1/en not_active Ceased
  • 2013-05-30 AU AU2013267279A patent/AU2013267279B2/en not_active Ceased
• 2016
  • 2016-01-25 US US15/005,842 patent/US10686724B2/en active Active

Patent Citations (5)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events