USRE48714E1 - Managing application performance in virtualization systems - Google Patents
Managing application performance in virtualization systems Download PDFInfo
- Publication number
- USRE48714E1 USRE48714E1 US16/940,583 US202016940583A USRE48714E US RE48714 E1 USRE48714 E1 US RE48714E1 US 202016940583 A US202016940583 A US 202016940583A US RE48714 E USRE48714 E US RE48714E
- Authority
- US
- United States
- Prior art keywords
- computer
- application
- container
- resources
- budget
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5077—Logical partitioning of resources; Management or configuration of virtualized resources
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
- G06Q10/06315—Needs-based resource requirements planning or analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/06—Buying, selling or leasing transactions
- G06Q30/0601—Electronic shopping [e-shopping]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/40—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks using virtualisation of network functions or resources, e.g. SDN or NFV entities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/50—Network service management, e.g. ensuring proper service fulfilment according to agreements
- H04L41/5003—Managing SLA; Interaction between SLA and QoS
- H04L41/5006—Creating or negotiating SLA contracts, guarantees or penalties
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/50—Network service management, e.g. ensuring proper service fulfilment according to agreements
- H04L41/5003—Managing SLA; Interaction between SLA and QoS
- H04L41/5019—Ensuring fulfilment of SLA
- H04L41/5025—Ensuring fulfilment of SLA by proactively reacting to service quality change, e.g. by reconfiguration after service quality degradation or upgrade
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/50—Network service management, e.g. ensuring proper service fulfilment according to agreements
- H04L41/5029—Service quality level-based billing, e.g. dependent on measured service level customer is charged more or less
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/50—Network service management, e.g. ensuring proper service fulfilment according to agreements
- H04L41/5041—Network service management, e.g. ensuring proper service fulfilment according to agreements characterised by the time relationship between creation and deployment of a service
- H04L41/5054—Automatic deployment of services triggered by the service manager, e.g. service implementation by automatic configuration of network components
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1023—Server selection for load balancing based on a hash applied to IP addresses or costs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45579—I/O management, e.g. providing access to device drivers or storage
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45591—Monitoring or debugging support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/50—Network service management, e.g. ensuring proper service fulfilment according to agreements
- H04L41/5003—Managing SLA; Interaction between SLA and QoS
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/50—Network service management, e.g. ensuring proper service fulfilment according to agreements
- H04L41/5003—Managing SLA; Interaction between SLA and QoS
- H04L41/5009—Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF]
Definitions
- the '011 patent also claims the benefit of U.S. Provisional Patent Application Nos. 62/077,898, 62/077,852, and 62/077,860, (each filed Nov. 10, 2014),
- the disclosures of each of which the '192 patent and '011 patent and U.S. Provisional Application Nos. 62/077,898, 62/077,852, 62/077,860, and 62/159,883 are hereby incorporated by reference in their entirety and for all purposes.
- This specification relates generally to managing resources in virtualization systems, and more specifically, but not exclusively, to the control of such resources based on application demands, to the control of access to such resources, and the management of such resources in a computer-implemented container system.
- Traditional computer system architectures typically include one or more dedicated computer servers for each application being run, and are often designed to include an excessive allocation of resources in order to be able to handle peak demands. Such partitioning of computer systems through dedicated servers and excessive allocation of resources can be costly, inefficient and difficult to scale and manage.
- Virtualization which refers to the abstraction of computer resources from their hardware or software-based physical constructs, is one manner of dealing with the aforementioned problems.
- One approach to virtualization is based on one or more virtual machines (VMs), each of which is a software implementation of a computer that executes programs or applications as if it was a physical computer.
- VMs virtual machines
- a virtual machine operates like a physical computer and contains, for example, its own virtual (e.g., software-based) central processing unit (CPU), random access memory (RAM), hard disk storage, and network interface card (NIC).
- Each virtual machine in a virtualization system generally runs its own guest operating system (OS), and the virtual machines generally share the underlying physical machine resources of the system.
- OS guest operating system
- Another approach to virtualization is based on one or more containers, each of which is allocated exclusive access to compute resources, using a separate name space, that it may use to execute applications or programs, as if it was a separate operating system.
- virtualization can be used for improving resource utilization and reducing the need for excess resources to absorb peak traffic. Virtualization can also be used to improve the availability and robustness of applications, by shifting workloads among servers to handle fail-over situations. Similarly, virtualization provides flexible partitioning of applications, deployment, and operations. Notwithstanding the potential benefits, operating in a virtualization system presents several challenges and potential pitfalls, including significant operations management challenges.
- I/O input/output
- VMs or containers request to execute heavy-storage tasks at the same time (e.g., VM reboots, anti-virus database updates, OS updates, virus scans, and so on)
- storage controllers can face unrecoverable I/O congestion.
- Managing resources in conventional virtualization systems includes evaluating alternative providers for a service or resource by comparing the attributes of the new service or resource and the current one. For example, when considering moving a VM, or a container, to a new storage device or array, conventional virtualization systems often consider the available storage amount and the data access latency of the new storage location, but ignore the time and resources it takes to move the VM, or container, to the new storage location. The impact of moving a VM can become significant when the amount of associated data to move is relatively large.
- conventional virtualization systems either do not allocate sufficient, or allocate too many, resources to an application that is initially deployed.
- a default configuration is used.
- the default configuration may not be application-specific, may not consider the particular demand profile of the application, and/or cannot account for varying actual demand of the application.
- modified configurations are based on best practices for an application type and/or artificial load tests in a simulated production environment. A single configuration generally cannot consider all possible application demands, and artificial load tests do not generally reflect with complete accuracy application demands in the production environment.
- Container systems provide an operating-system level virtualization in which the kernel of an operating system can allow for multiple isolated user space instances.
- a container is based on server virtualization that uses a shared operating system. Rather than virtualizing hardware and creating whole virtual machines, each with their own operating systems, containers run atop the shared operating system kernel and file system that looks and feels like a complete, isolated instance of the operating system. Like shipping containers for cargo, these software containers can ship applications across different network-based systems (e.g., cloud computing based systems) and limit the impact of one container's activities on another container.
- network-based systems e.g., cloud computing based systems
- a container system may include software abstractions to virtualize computer resources (or compute resources) which are used by applications running in the container (“containerized” applications).
- the container system provides means to provision containers, allocate and control the resources available to a container, deploy and execute applications in the container, and facilitate full use of the container resources by such containerized applications, while isolating them from other applications, sharing the underlying resources.
- a containerized application accesses a virtualized container resource (e.g., CPU, memory, storage I/O, Network I/O)
- the container system maps this access to a direct access of the underlying real resource.
- Container systems like virtual machine systems, provide means for abstracting computer resources (or compute resources), controlling and isolating the allocations of these resources to applications, distributing and migrating applications flexibly, among multiple servers, to support scalable, highly-available, robust and efficient datacenter (DC) and cloud systems. Additional information on containers can be found, for example, at Linux Containers (available at https://linuxcontainers.org), http://en.wikipedia.org/wiki/Docker_(software), and https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/YARN.html, each of which is hereby incorporated by reference in its entirety and for all purposes.
- containers are based on shared operating systems, unlike virtual machines, they do not require guest operating systems and thus avoid some of the overhead of virtual machines. For example, deploying and configuring a container may involve setting data structures to map container resources to server and OS resources. Therefore, deploying and configuring a container can often be accomplished in seconds; deploying a virtual machine and a guest OS and configuring both to run an application may require substantially more time.
- container-virtualization can offer significant efficiencies and performance advantages over virtual-machines—e.g., see “An Updated Performance Comparison of Virtual Machines and Linux Containers,” by W. Felter et al., IBM Research, Jul.
- a virtualization system may mix and match virtual machines and containers. For example, containers may run over virtual-machines. Additionally, a group of virtual machines may be containerized, much like any application, and executed by a container.
- the number of containers sharing a host is often an order of magnitude (e.g., 50-200) larger than the number of VMs (e.g., 5-20).
- containers perform a faster dynamic of change events (e.g., deploy, delete, increase/reduce resources, and so on). Accordingly, container system management presents challenging scalability and response time problems compared to VMs.
- This specification describes new technologies relating to the management of resources and performance in virtualization systems, including container systems and multi-cloud systems. For example, these technologies introduce the use of supply chain economics and other techniques to offer a unified platform to integrate, optimize or improve, and automate resource and performance management in a container system. The economics-based methods can also extend to other virtualization systems for managing application performance.
- one aspect of the subject matter described in this specification can be embodied in methods and systems that include the actions of determining, for example, by a Container Manager running on a data processor in a container system, a computer resource bundle to be purchased for a container of the container system using virtual currency units; identifying multiple computer servers available to the container system; offering the computer resource bundle; determining a purchase price for the computer resource bundle, in virtual currency units, for each of the identified computer servers; automatically selecting, by the Container Manager, one of the identified computer servers based at least in part on the purchase price for the computer resource bundle set for each of the identified computer servers; and allocating the computer resource bundle from the selected one of the multiple computer servers to the container.
- Other embodiments of this aspect include corresponding systems, apparatus, and computer program products.
- Another aspect of the subject matter described in this specification can be embodied in methods and systems that include determining, by a Container Manager running on a data processor in a first container system, a computer resource bundle to be purchased for a container in the first container system using virtual currency units; receiving, from a Proxy Manager of a second container system offering the computer resource bundle, a purchase price for the computer resource bundle in virtual currency units; automatically purchasing the computer resource bundle from the second container system based at least in part on the purchase price received from the Proxy Manager of the second container system; allocating the computer resource bundle from the second container system to the container in the first container system; and dispatching the container from the first container system to execute at the second container system.
- dispatching the container includes sending an image of a stateful container to another server or, in the case of stateless containers, initiating a new container on another server and terminating the server on the first server.
- Other embodiments of this aspect include corresponding systems, apparatus, and computer program products.
- a further aspect of the subject matter described in this specification can be embodied in methods and systems that include the actions of determining, by a Container Manager running on a data processor in a first container system, a computer resource bundle to be purchased for a container in the first container system using virtual currency units; identifying at least a second container system and a third container system; offering the computer resource bundle at respective purchase prices, in virtual currency units; receiving, from respective Proxy Managers of the second container system and the third container system, a first purchase price for the computer resource bundle for the second container system and a second purchase price for the computer resource bundle for the third container system; automatically selecting one of the second container system and the third container system based at least in part on the respective purchase prices for the computer resource bundle for the second container system and the third container system; and allocating the computer resource bundle from the selected one of the second container system and the third container system to the container in the first container system.
- Other embodiments of this aspect include corresponding systems, apparatus, and computer program products.
- Still another aspect of the subject matter described in this specification can be embodied in methods and systems that include the actions of allocating a computer resource bundle to an application executed by a container system; determining, by an Application Element Manager running on a data processor in the container system, the value of a service level agreement parameter for the application based on the allocated computer resource bundle; comparing the determined service level agreement parameter level for the application to a threshold service level agreement parameter level; automatically modifying the allocation of computer resources to the application depending on whether the identified service level agreement parameter level for the application is below or above the threshold service level agreement parameter level; and repeating the determining, comparing and automatically modifying steps until the operation of the application in the container system is suspended or terminated.
- Other embodiments of this aspect include corresponding systems, apparatus, and computer program products.
- Yet another aspect of the subject matter described in this specification can be embodied in methods and systems that include the actions of purchasing a computer resource bundle, using virtual currency units, for an application running at a container in a container system; allocating the purchased computer resource bundle to the application; determining, by a Container Manager running on a data processor in the container system, the value of a service level agreement parameter for the application; automatically modifying the purchase of the computer resource bundle based at least in part on a comparison of the determined value of a service level agreement parameter to a predetermined threshold service level agreement parameter level; and repeating the determining, comparing and automatically modifying steps until the operation of the application in the container system is suspended or terminated.
- Other embodiments of this aspect include corresponding systems, apparatus, and computer program products.
- Another aspect of the subject matter described in this specification can be embodied in methods and systems that include network-aware placement of containers to minimize input/output (I/O) latency.
- Some methods include the actions of determining a target I/O latency metric, for storage or network communications of an application running in a container system; determining, by a container manager, a first I/O pathway that meets the target latency and an amount of I/O bandwidth, over the first I/O pathway, to be acquired for the container; allocating the determined amount of I/O bandwidth, over the determined first I/O pathway to the application; automatically allocating the determined amount of I/O bandwidth to the application, from the second I/O pathway, based at least in part on the determined I/O bandwidth utilization of the first I/O pathway.
- Other embodiments of this aspect include corresponding systems, apparatus, and computer program products.
- Another aspect of the subject matter described in this specification can be embodied in methods and systems that include the actions of determining, by a Container Manager running on a data processor in a container system, an amount of input/output (I/O) bandwidth—for example, for storage or network communications—to be acquired for a container of the container system; identifying at least a first computer server and a second computer server in the container system, each offering at least the determined amount of I/O bandwidth; allocating, from the first computer server, the amount of I/O bandwidth to the container; determining the I/O bandwidth utilization of the first computer server; automatically allocating the amount of I/O bandwidth to the container, from the second computer server, based at least in part on the determined I/O bandwidth utilization of the first computer server.
- Other embodiments of this aspect include corresponding systems, apparatus, and computer program products.
- Yet another aspect of the subject matter described in this specification can be embodied in methods and systems that include the actions of determining, by a Container Manager running on a data processor in a container system, an amount of I/O bandwidth—for example, for storage or network communications—to be acquired from a first computer server for a container running on the first computer server; allocating, from the first computer server, the amount of I/O bandwidth to the container for a first period of time; after the first period of time has elapsed, determining that the I/O bandwidth utilization of the first computer server is greater than a threshold limit; automatically identifying at least a second computer server in the virtualization system offering at least the determined amount of I/O bandwidth of the first computer server; moving the container from the first computer server to the identified second computer server; and allocating from the second computer server, the determined amount of I/O bandwidth, to the container, for a second period of time.
- Other embodiments of this aspect include corresponding systems, apparatus, and computer program products.
- Yet another aspect of the subject matter described in this specification can be embodied in methods and systems for fractional resource allocation that include the actions of determining, by a Container Manager running on a data processor in a container system, a resource bundle to be acquired from a first computer server for a container running on the first computer server; allocating, from the first computer server, the determined resource bundle for a first period of time; after the first period of time has elapsed, determining that the utilization of the first computer server exceeds a threshold limit; determining the excess resources corresponding to the excess utilization; automatically identifying a second computer server to replicate the container; replicating the container at the identified second computer server; allocating from the identified second computer server, a resource bundle equal or lower than the determined excess resources, to the replicated container, for a second period of time.
- Other embodiments of this aspect include corresponding systems, apparatus, and computer program products.
- Yet another aspect of the subject matter described in this specification can be embodied in methods and systems that include the actions of determining, by a Container Manager running on a data processor in a container system, an amount of I/O bandwidth—for example, for storage or network communications—to be acquired from a first computer server for a container running on the first computer server; allocating, from the first computer server, the amount of I/O bandwidth to the container for a first period of time; after the first period of time has elapsed, determining that the I/O bandwidth utilization of the first computer server exceeds an I/O bandwidth threshold limit; determining the excess I/O bandwidth corresponding to the excess I/O bandwidth utilization; automatically identifying a second computer server in the container system to replicate the container; allocating from a second computer server, an amount of I/O bandwidth equal or lower than the determined excess I/O bandwidth, to the replicated container, for a second period of time; and using a Load Balancer to divide the processing of I/O bandwidth between the container, at the first computer server, and the replicate
- Still another aspect of the subject matter described in this specification can be embodied in methods and systems that include the actions of determining, by an Application Element Manager running on a data processor in a container system, a software license to be purchased using virtual currency units by an application operating in the container system; determining, by a Licensing Element Manager running on the data processor, the number of copies of the software licenses available for purchase in the container system; determining, by the Licensing Element Manager, the purchase price for a copy of the software license; automatically purchasing, by the Application Element Manager, a copy of the software license for the application based at least in part of the determined purchase price for a copy of the software license; and allocating the purchased copy of the software license to the application.
- Other embodiments of this aspect include corresponding systems, apparatus, and computer program products.
- a further aspect of the subject matter described in this specification can be embodied in methods and systems that include the actions of allocating, from a computer server in a container system, a computer resource bundle to a container operating in the container system; determining, by a Container Manager running on a data processor in the container system, a productivity score of the container; determining that the productivity level of the container is below a threshold productivity level; and automatically initiating, by the Container Manager, at least one of a suspension sequence and a termination sequence for the container based at least in part on the determination that the productivity level of the container is below a threshold level.
- Other embodiments of this aspect include corresponding systems, apparatus, and computer program products.
- Another aspect of the subject matter described in this specification can be embodied in methods and systems that include the actions of allocating a computer resource bundle to a computer server operating in a container system; determining, by a Computer Server Element Manager running on a data processor in the container system, a productivity score of the computer server; determining that the productivity level of the computer server is below a threshold productivity level; and automatically initiating, by the Computer Server Element Manager, at least one of a suspension sequence and a termination sequence for the computer server based at least in part on the determination that the productivity level of the computer server is below a threshold level.
- Other embodiments of this aspect include corresponding systems, apparatus, and computer program products.
- Yet another aspect of the subject matter described in this specification can be embodied in methods and systems that include the actions of determining a computer resource bundle to be purchased for a containerized container running at a parent container in a container system; purchasing the determined computer resource bundle; and allocating the purchased computer resource bundle to the containerized container.
- Other embodiments of this aspect include corresponding systems, apparatus, and computer program products.
- the subject matter includes methods to place containers at hosts and allocate computer resources (or compute resources) to the containers based on a supply-chain marketplace for buying/selling resources; migrate containers to hosts based on purchasing resources in a supply chain marketplace; automatically replicate an overloaded container, allocate resources to the replicated containers, and partition the container workload among its replicated containers; automatically replicate an overloaded container, allocate resources to the replicated containers, and partition the I/O stream among the replicated containers using a load balancer; allocate resources to containers to handle the workload of an application and assure a target Service Level Agreement (SLA) performance metric; allocate software licenses to applications running in containers; and select a production system and to dispatch a container from a development system to a production system; assign a productivity score to a host running a container system, based on its virtual Profit and Loss (P&L) performance, remove containers from a host with low productivity and set it into standby mode, or shut it down to save electricity.
- SLA Service Level Agreement
- the subject matter further includes methods to place highly communicating containers “close” to one another to minimize their I/O latency and maximize its bandwidth.
- “close” can be measured by the number of switches along the I/O pathway.
- Additional subject matter disclosed includes methods to cluster containers with related applications into container points of delivery (cPODs) and migrate an entire cPOD between clouds through an atomic action. Further disclosure includes methods and systems to containerize a container to create a supply chain hierarchy of nested containers and use it to provide scalable resource allocation whereby a parent container “sells” its resources, using virtual currency, to its daughter containers.
- Virtual currency units can be used to control the purchase of computer resources.
- Computer resources can be allocated in the event a budget of the acquiring computer element is sufficient.
- the allocation of computer resources can be in exchange for the payment of virtual currency units.
- Virtual currency units can be converted into real currency, such as United Stated dollars, or euros.
- a performance metric of an application executing in a container can be determined, and the allocation of computer resources is based on such performance metric.
- a performance metric of an application executing in a container can be determined, and the allocation of virtual currency units to the application or the container can be based on such performance metric.
- the computer resources to be allocated can include at least one of an allotment of computer memory, an allotment of program execution scheduling for one or more central processing units, an allotment of storage interface bandwidth, an allotment of network or I/O interface bandwidth, and an allotment of a storage array.
- a computer resource provider can be selected based on the price offered for the computer resources.
- the price offered for the computer resources can be determined by an auction process, or by the utilization of the computer resources (e.g., congestion pricing), or by a service level metric provided by the resources, or by the cost of the resource, or any combination of these and other pricing factors.
- the amount of computer resources purchased can be dependent on a computed computer resource utilization.
- a proxy manager of a second container system optionally controls the allocation of computer resources to a first container system.
- Computing resources may be monitored and managed in container systems based on economic and non-economic principles, both in single-cloud and multi-cloud systems. Such management of resources may be used to control the delivery of service level agreements, and the distribution of software licenses, in container systems. They also may be used to suspend or terminate computing resources in container systems.
- Another aspect of the subject matter described in this specification can be embodied in systems and methods that assure application performance by matching the supply of resources (e.g., application resources, VM resources, or physical resources) with the fluctuating demand placed on the application.
- resources e.g., application resources, VM resources, or physical resources
- the systems and methods disclosed herein can be used to ensure that the application is allocated sufficient resources when it is initially deployed to handle anticipated demand; dynamically alter the resources allocated to the application during operation by matching the resource requirements to the actual measured application demand; and predict future resource requirements based on planning assumptions related to future application demand.
- the systems and methods described herein apply to applications that scale vertically—e.g., to scale the additional resources that are allocated to the existing application components. Additionally and/or alternatively, the systems and methods described herein apply to applications that scale horizontally. Horizontally scaling applications are scaled by provisioning additional application components and dividing the work to be done among the old and new components (compared to adding resources to existing components).
- the application component that horizontally scales buys additional resources from a new provider (e.g., virtual server)—rather than the current provider—and is provisioned along with a new application component.
- the new application component can purchase the required resource from the new provider.
- the provider and the application component can be decommissioned.
- the resources can be removed.
- the system can be used in real time in order to efficiently allocate the required resources to applications based on a balance of supply and demand.
- the system is able to scale an application either vertically (reconfigure) or horizontally (auto-scale) such that more detailed control can be achieved by considering target Quality of Service (QoS) metrics such as transactions throughput rate or transaction response time targets defined for these measures. These targets represent the desired service level of the application.
- QoS Quality of Service
- the application may perform below its desired service level and not achieve the defined targets for these QoS metrics. If the application achieves the target, it is considered to perform at its desired service level.
- An application that is below its desired service level can be brought into compliance by either adding more resources (vertical scale) or more application components (horizontal scale).
- the ability of an application to buy resources is a function of its revenue and the expenses based on the price of the resources.
- the system can allocate additional budget to the application, providing a greater power to buy resources. Therefore, QoS metrics can be used to gain additional granularity in the allocation of resources.
- the system can also be used when deploying new applications in order to determine the required or preferred resource allocation.
- application demand e.g., transactions
- the system can determine the required or preferred number of application components and the appropriate allocation of resources.
- the system can be used when planning for future application needs. By simulating future application demand (e.g., transactions), the system can determine the required number of application components and the appropriate allocation of resources.
- Computing resources may be monitored and managed in virtualization systems based on economic and non-economic principles, both in single-cloud and multi-cloud systems. Such management of resources may be used to control the delivery of service level agreements, and the distribution of software licenses, in virtualization systems. They may also be used to suspend or terminate computing resources in virtualization systems.
- the subject matter further includes methods relating to the management of resources in virtualization systems, including multi-cloud systems.
- These technologies introduce the use of economic-based cost analysis, and other techniques to offer a comprehensive framework to manage the number and rate of actions and prioritize resource management in a virtualization system.
- ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇
- the AM can sell multiple types of permits, regulating access to a number of resources. Each permit can be associated with a predetermined price. Additionally and alternatively, this price can be dynamically adjusted, for example, taking into consideration the availability of permits the AM possesses.
- Other embodiments of this aspect include corresponding systems, apparatus, and computer program products.
- the consumer acquires the permits from the resource or service itself.
- “formulating” can include the attributes taken into account when considering the option to move to the new provider.
- the cost of moving can be part of the comparison between two or more alternatives (e.g., keeping a VM in an existing infrastructure or moving the VM to an external cloud provider). Cost can be expressed in actual currency or any unit suitable for the comparison. For example, moving time can be expressed in a real value that quantifies the cost of the VM downtime. In contrast, if there is a strict limit on acceptable downtime, the cost of moving the VM can be expressed in terms of a cost metric of downtime.
- “evaluating” includes making the decision (e.g., initiating an action based on the decision) and determining the right time to take the action.
- the decision e.g., initiating an action based on the decision
- determining the right time to take the action e.g., determining the right time to take the action.
- one embodiment described herein postpones the decision for the future, advantageously waiting for a sufficient amount of time until the decision-maker is convinced that the decision is the right one.
- Other embodiments of this aspect include corresponding systems, apparatus, and computer program products.
- Computing resources may be monitored and managed in virtualization systems based on economic and non-economic principles, both in single-cloud and multi-cloud systems. Such management of resources may be used to control the delivery of service level agreements based on access control, and the distribution of software licenses, in virtualization systems. They may also be used to suspend or terminate computing resources in virtualization systems.
- Another aspect of the subject matter described in this specification can be embodied in methods and systems that include determining, by an application manager running on a data processor in a virtualization system that runs at least one guest operating system, a computer resource bundle to be purchased, using virtual currency units, for an application running on a virtual machine in the virtualization system; allocating the computer resource bundle from a computer server in the virtualization system to the application; and initiating, following an increase in the purchase price for the allocated computer resource bundle, a credit check process by the application manager, wherein the credit check process results in an increase in budget of virtual currency units available for usage by the application, wherein the virtual machine is a software implementation of a computer that runs at least one guest operating system and executes one or more applications on the guest operating system.
- the methods and systems further include receiving, from a proxy manager of a second virtualization system offering the computer resource bundle, a purchase price for the determined computer resource bundle in virtual currency units, wherein the second virtualization system also runs at least one guest operating system; determining, following initiation of the credit check process, whether the determined computer resource bundle is to be purchased from a second computer server in the second virtualization system based at least in part on the purchase price received from the proxy manager of the second virtualization system; and allocating the computer resource bundle from the second computer server in the second virtualization system to the application.
- the methods and systems further includes maintaining a budget, in virtual currency units, for the application; determining whether the budget of virtual currency units for the application is sufficient to pay the purchase price for the computer resource bundle; allocating the computer resource bundle to the application only if the budget of the application has been determined to be sufficient to pay the purchase price for the computer resource bundle; and transferring virtual currency units from the budget for the application to a budget for the computer server based on the purchase price for the computer resource bundle determined for the computer server.
- the computer resource bundle includes one or more computer resources
- the method further includes determining a performance metric level of the application; and adjusting the allocation of at least one of the one or more computer resources to the application based on a comparison of the determined performance metric level to a predetermined threshold performance metric level.
- the performance metric level of the application is correlated to at least one of the transaction rate for the application, the number of transactions performed by the application, and the completion time or delay for the application.
- the computer resource bundle includes one or more computer resources
- the method further includes determining a performance metric level of the application; and adjusting a budget of virtual currency units for the application based on a comparison of the determined performance metric level to a predetermined threshold performance metric level.
- the computer resource bundle includes at least one of an allotment of computer memory, an allotment of program execution scheduling for one or more central processing units, an allotment of storage interface bandwidth, an allotment of network interface bandwidth, and an allotment of a storage array.
- allocating the computer resource bundle from the computer server to the application comprises moving the application to a second computer server.
- the allocated computer resource bundle includes at least a first computer resource and a second computer resource, the method further comprising de-allocating the first computer resource based at least in part on a comparison of a determined usage of the first computer resource to a determined usage of the second computer resource.
- the computer resource bundle includes multiple computer resources, and wherein the determining a computer resource bundle to be purchased is based at least in part on a computed utilization of one of the multiple computer resources.
- the method further includes increasing the amount of the computer resource bundle to be purchased if it is determined that the computed utilization of the one of the multiple computer resources is below a predetermined threshold.
- the method further includes decreasing the amount of the computer resource bundle to be purchased if is determined that the computed utilization of the one of the multiple computer resources is above a predetermined threshold.
- the determining a purchase price for the computer resource bundle includes receiving one or more auction bids for purchase of the computer resource bundle.
- the methods and systems further include converting virtual currency units to government issued monetary currency based on a predetermined exchange rate.
- FIG. 1 Another aspect of the subject matter described in this specification can be embodied in methods and systems for regulating access to computer resources, comprising: a computer-based virtualization system that runs at least one guest operating system; and instructions stored on a non-transitory computer readable medium in the virtualization system and executable by a data processing apparatus to cause the data processing apparatus to perform operations comprising: determining, by an application manager running on a data processor in a virtualization system that runs at least one guest operating system, a computer resource bundle to be purchased, using virtual currency units, for an application running on a virtual machine in the virtualization system, allocating the computer resource bundle from a computer server in the virtualization system to the application; and initiating, following an increase in the purchase price for the allocated computer resource bundle, a credit check process by the application manager, wherein the credit check process results in an increase in budget of virtual currency units available for usage by the application, and wherein the virtual machine is a software implementation of a computer that runs at least one guest operating system and executes one or more programs or applications on the guest
- the operations further include receiving, from a proxy manager of a second virtualization system offering the computer resource bundle, a purchase price for the determined computer resource bundle in virtual currency units, wherein the second virtualization system also runs at least one guest operating system; determining, following initiation of the credit check process, whether the determined computer resource bundle is to be purchased from a second computer server in the second virtualization system based at least in part on the purchase price received from the proxy manager of the second virtualization system; and allocating the computer resource bundle from the second computer server in the second virtualization system to the application.
- the operations further include maintaining a budget, in virtual currency units, for the application; determining whether the budget of virtual currency units for the application is sufficient to pay the purchase price for the computer resource bundle; allocating the computer resource bundle to the application only if the budget of the application has been determined to be sufficient to pay the purchase price for the computer resource bundle; and transferring virtual currency units from the budget for the application to a budget for the computer server based on the purchase price for the computer resource bundle determined for the computer server.
- the operations further include determining a performance metric level of the application; and adjusting a budget of virtual currency units for the application based on a comparison of the determined performance metric level to a predetermined threshold performance metric level.
- the computer resource bundle includes at least one of an allotment of computer memory, an allotment of program execution scheduling for one or more central processing units, an allotment of storage interface bandwidth, an allotment of network interface bandwidth, and an allotment of a storage array.
- allocating the computer resource bundle from the computer server to the application comprises moving the application to a second computer server.
- the allocated computer resource bundle includes at least a first computer resource and a second computer resource, the instructions further causing the data processing apparatus to de-allocate the first computer resource based at least in part on a comparison of a determined usage of the first computer resource to a determined usage of the second computer resource.
- FIG. 1 is a block diagram of an example container environment in which resources are managed.
- FIG. 2 is a block diagram of an example software system for managing resources in a container system.
- FIG. 3 is a flow diagram of an example process for using a platform manager in a container system.
- FIG. 4 is an example model for service provision and consumption in a supply chain container system.
- FIG. 5 is a flow diagram of an example process for deploying a new consumer element with a provider element in a container system.
- FIG. 6 is a flow diagram of an example process for delivering service level agreement targets through resource allocation in a container system.
- FIG. 7 is a flow diagram of an example process for economic-based I/O scheduling in a container system.
- FIG. 8A is an example purchase order data structure for use in purchasing services from a provider element manager in a container system.
- FIG. 8B is an example service confirmation data structure for use in confirming or rejecting the purchase of services from a provider element manager in a container system.
- FIG. 9 is an example process for managing the states of system elements in a container system.
- FIG. 10 is a block diagram of an example multi-domain software system environment for managing virtualized resources.
- FIG. 11 is a block diagram of an example virtualization environment which illustrates supply chain relationships between service entities and resources.
- FIG. 12 is a block diagram of another example virtualization environment which illustrates supply chain relationships between service entities and resources in a container system.
- FIG. 13 is a flow chart illustrating a process for resource scaling in the virtualization environment of FIG. 11 .
- FIG. 14 is a flow chart illustrating a process for service entity scaling in the virtualization environment of FIG. 11 .
- FIG. 15 is a block diagram of yet another example virtualization environment which illustrates the supply chain relationships between service entities and resources in a virtualization environment and can cooperate with the processes described in FIGS. 13-14 .
- FIG. 16 is a block diagram of an example virtualization environment in which resources are managed by an action manager.
- FIG. 17 is a block diagram illustrating the data flow for managing resources in the virtualization environment of FIG. 16 .
- FIG. 18 illustrates an exemplary block diagram of a virtualization environment in which a virtual machine is determining whether to take an action.
- FIG. 1 is a block diagram of an example container system or environment 100 in which resources are managed.
- the example system 100 includes two servers 102 and 104 that run respective container systems 110 and 112 .
- the container system 110 at the server 102 , allocates computer resources (or compute resources) of the server 102 —e.g., CPUs, memories, storage volume, storage, and/or network I/O pathways and bandwidth—to two containers 120 and 122 .
- the container system 112 at the server 104 allocates resources of the server 104 to containers 124 and 126 .
- the containers 120 , 122 , 124 and 126 execute respective containerized applications 130 , 132 , 134 , and 136 .
- the servers 102 and 104 may be physical machines with physical computer resources (or compute resources).
- the server 102 may be a virtual machine with virtualized resources while the server 104 is a physical server.
- the containers 120 , 122 , 124 and 126 may be distinct containers, or replicated copies of a single container.
- a group of containers may be clustered into a container-Point-of-Delivery (cPOD) system, to run related applications.
- cPOD container-Point-of-Delivery
- a multi-tier Web service may include a containerized Web server (shown as the application 130 ), a containerized application server (shown as the application 134 ), and a containerized database server (shown as the application 136 ).
- the Web server provided by the application 130 can maintain significant level of communications with the application server provided by the application 134 .
- the I/O pathway between the applications 130 , 134 traverses the application 130 , the container 120 , the container system 110 , an operating system 106 , a network interface card (NIC) 140 , a data network 160 , a NIC 142 , an operating system 108 , the container system 112 , the container 124 , and the application 134 .
- NIC network interface card
- a container manager 236 can migrate (ship) the container 120 , with the application 130 , from the server 102 to the server 104 .
- This migration replaces the I/O pathway from the application 130 to the application 134 with a pathway that includes the application 130 , the container 120 , the container system 112 , the operating system 108 , the container system 112 , the container 124 , and the application 134 .
- this modified I/O pathway entirely can be handled by the server 104 through memory transfers.
- This in-memory I/O pathway can support very high memory transfers bandwidth and very low latency, thus, improving the cPOD performance.
- FIG. 1 Although a specific environment 100 including the two servers 102 and 104 is shown in FIG. 1 and described above, it will be understood that the environment 100 is illustrative only.
- the environment 100 may include more than two servers, and each of the servers 102 and 104 may be associated with any number of containers as desired.
- FIG. 2 is a block diagram of an example software system 200 for managing resources in container systems, such as the container system 100 .
- the software system 200 may be used to allocate server and I/O resources (such as CPU, memory, flash storage, hard drive storage and I/O bandwidth) to containers.
- the software system 200 also may be used, for example, to monitor, detect, and handle congestion conditions at a resource (e.g., I/O pathway, memory, and so on) and to move containers among available servers to optimize or improve application performance and resource utilization.
- a resource e.g., I/O pathway, memory, and so on
- the software system 200 monitors, controls, and otherwise interacts with various managed container system elements (also referred to herein as service elements or computer elements) through respective instrumentation.
- instrumentation refers generally to any software and/or hardware that provides an ability to monitor, control, or otherwise interact with a computer element, such as to detect operations events, reconfigure parameters, diagnose errors, and write trace information.
- a computer application may be managed using a management tool.
- FIG. 2 Several example container system elements are shown in FIG. 2 as part of an Information Technology (IT) Container Stack (ITCS) 202 , including applications components 210 , container systems 212 , servers 214 , storage systems 216 , networks 218 , and operating resources 220 (such as power supplies, cooling systems, and rack space).
- ITCS Information Technology
- the ITCS 202 may include, for example, a proper subset or a proper superset of these container system elements 210 , 212 , 214 , 216 , 218 , and 220 .
- the software system 200 includes a platform layer 230 , which provides an infrastructure to manage, for example, the I/O flows in a container system (such as the example container system environment 100 shown in FIG. 1 ).
- the platform layer 230 includes element managers 234 , 236 , 238 , 240 , 242 , and 244 . More particularly, the platform layer 230 includes an application manager 234 , a container system manager 236 , a server manager 238 , a storage manager 240 , a network manager 242 , and an operations manager 244 .
- These element managers 234 , 236 , 238 , 240 , 242 , and 244 use management instrumentation of respective elements to monitor and control the respective elements of the ITCS 202 .
- the server manager 238 may use built-in management instrumentation, such as Management Information Bases (MIBs) of the server it is managing, to monitor the server's CPU, memory, and I/O interfaces (such as a Host Bus Adapter (HBA) and NICs) and to control their operational parameters.
- the server manager 238 may access such management instrumentation using standardized protocols (such as Simple Network Management Protocol (SNMP)) or specialized mechanisms.
- SNMP Simple Network Management Protocol
- a proper superset or only a proper subset of these element managers 234 , 236 , 238 , 240 , 242 , and 244 may be desired or needed in certain environments. For example, when the containers do not access storage, the use of a storage manager 240 may not be needed.
- an operating system element manager (not shown) may be included as part of platform layer 230 .
- the platform layer 230 also includes one or more types of modeling databases 245 .
- the databases 245 may include supply chain modeling (SCM) databases 246 and operations databases 248 .
- the platform layer 230 also includes a platform manager 250 , which, as explained in greater detail below, can be responsible for general provisioning, initializing, and management tasks.
- the software system 200 shown in FIG. 2 also includes a functional management layer 252 , which includes user interface (UI) software 260 for use by administrators or other users to monitor and control a container system (such as the example container system environment 100 shown in FIG. 1 ).
- UI user interface
- an administrator may use the UI software 260 to set proactive automation policies to optimize or improve performance and resource utilization, detect and resolve operational problems and performance bottlenecks, allocate priorities and usage charges to different applications, and plan capacity expansions.
- the functional management layer 252 also includes a collection of functional managers 272 , 274 , 276 , and 278 , which are used to enable users to monitor, control, and automate the underlying automated management mechanisms of container systems according to the principles described herein.
- the software system 200 may alternatively include, for example, a proper subset or a proper superset of these functional managers.
- the functional management layer 252 includes an application manager 272 , which, for example, enables users to select or configure respective parameters of a computer agent or process to partition application components among different containers, allocates virtual budgets to applications based on the business value of their services, as described in greater detail below, and specifies the resources required by the applications.
- the application manager 272 uses the parameters to create respective records in the operations databases 248 .
- the platform manager 250 uses the operations records to initialize respective application managers 234 , which use the operations records to deploy the applications 210 , according to the principles described below. Additional functions of monitoring and controlling applications may be incorporated into the application manager 272 .
- the functional management layer 252 also includes a performance manager 274 , which allows users to monitor and control the delivery of Service Level Agreements (SLAs) to applications.
- SLAs Service Level Agreements
- a user of the software system 200 can specify target SLA parameters—such as latency or transaction rate—of one or more particular applications.
- the SLA parameters are used by the software system 200 to adjust the performance of the applications using the principles described below.
- a user can also monitor the SLA parameters value, as well as the respective virtual payments made by an application, thereby correlating the application's budget with its SLA performance. Additional functions of monitoring and controlling the performance of applications, as well as the other elements of the ITCS 202 , may be incorporated into the performance manager 274 .
- a capacity manager 276 monitors relationships between the supply and demand of resources in the ITCS 202 .
- the capacity manager 276 may monitor the relationships over a predetermined time period, which can range from short term (such as a few minutes or one hour) to long term (such as one day, week, month or year).
- the capacity manager 276 maintains full accounting of revenues and costs and provides monitoring of these accounts and notifications upon certain accounting events.
- the capacity manager 276 by itself or with the assistance of an incorporated or separate Return-on-Investment (ROI) manager (not shown), enables a user to monitor the ROI of the elements in the ITCS 202 .
- the ROI is defined as revenue divided by cost, where revenue is the income from virtual payment collected by a selected element and cost is the virtual payments by the element for the resources that the element uses.
- a large ROI may indicate to the capacity manager 276 that there is excess demand over supply of the element capacity, and a sustained high ROI may thus indicate insufficient capacity.
- the capacity manager 276 compares a monitored ROI with specific and potentially predetermined ROI targets, which may be configured by an administrator or other user, to recommend capacity increases of particular elements to meet demand According to the supply chain economic principles described below, the ROI of an element in the ITCS 202 may be considered as a central metric of economic value.
- the ROI may be calculated at any appropriate time and for any appropriate duration over which revenue and cost are considered.
- the principles described herein provide an accounting framework to quantify and measure the value generated by components of the ITCS 202 .
- the ITCS 202 there are raw resources that generate real (non-virtual) costs, such as monetary costs that are paid to an electric company.
- real (non-virtual) costs such as monetary costs that are paid to an electric company.
- applications that play roles in generating real (non-virtual) revenues, such as monetary sales revenue received from customers.
- a back-charging manager 278 monitors and accounts for the virtual cash flows between elements in the ITCS 202 and enables users to flexibly compute financial metrics of interest. For example, users can monitor metrics describing the allocation of application budgets to acquire supply chain resources, the allocation of a resource among the workloads of different applications, the ROI efficiency of different resources, and application budgets required to deliver particular SLAs. These metrics and other parameters may be used to support policies on budgeting applications, adjusting the budgets to represent changing prices, capacity, and demand of resources along the supply chain, and converting virtual currency used within the software system 200 to real currency (such as United States dollars, or euros) that is generated by the business units who own the applications and that may be used to pay for IT resources.
- real currency such as United States dollars, or euros
- the platform manager 250 can manage a container system using any suitable means described herein, including using a process 300 as shown in FIG. 3 .
- FIG. 3 a flow diagram of the example process 300 for using the platform manager 250 in a container system (such as the container system 100 ) is shown.
- the platform manager 250 initializes, or launches, the functional managers 272 , 274 , 276 and 278 of the functional management layer 252 for a specific container environment (step 302 ).
- the platform manager 250 discovers the managed container system elements of the ITCS 202 in the container environment (step 304 ). This discovery is handled, for example, through standard processes to get configuration data from the container system, OS, server, network, and storage systems.
- the platform manager 250 also initializes, or launches, an element manager (such as one or more of element managers 234 , 236 , 238 , 240 , 242 , and 244 , described above) for each group of respective elements of a given class of elements that have been discovered (step 306 ).
- an element manager such as one or more of element managers 234 , 236 , 238 , 240 , 242 , and 244 , described above
- the platform manager 250 may detect a Dell® server and a SUN® server, and the corresponding groups of respective elements may both be assigned respective element managers.
- the platform manager 250 configures the element managers to monitor and control the respective elements via respective management instrumentation.
- the platform manager 250 populates and initializes the platform modeling databases 245 —for example, the supply chain modeling databases 246 and the operational databases 248 (step 308 )—and starts monitoring certain potential changes of the managed environment (step 310 ). For example, the container system 100 may be monitored to determine if there have been any container changes, such as any added, deleted, or migrated container (decision block 312 ). If a container change has been detected, the platform manager 250 again initializes the element managers as described above.
- the presence of container systems is evaluated to determine if there have been any container system changes, such as any added or deleted container system (decision block 314 ). If a container system change has been detected, the platform manager 250 again discovers the managed container system elements of the ITCS 202 in the container environment as described above. Otherwise, the platform manager 250 evaluates whether there have been any major network changes (decision block 316 ), in which case the platform manager 250 similarly re-discovers the managed container system elements of the ITCS 202 in the container environment as described above. For example, the platform manager 250 may discover loss or gain of network I/O pathways, congestion or under-utilization of an I/O pathway, low or excessive latency of an I/O pathway, or packet losses along an I/O pathway.
- the platform manager 250 evaluates whether there have been any major storage changes (decision block 318 ). For example, the platform manager 250 may discover storage I/O congestion, or alternate I/O pathways that would provide better (i.e., lower) access latency. If major storage changes have been detected, the platform manager 250 again discovers the managed container system elements of the ITCS 202 in the container environment as described above.
- the platform manager 250 determines whether to continue monitoring of the same (decision block 320 ). If the platform manager 250 decides to continue monitoring, the platform manager 250 again starts the monitoring of potential changes of the managed environment. Otherwise, the process 300 ends (end block 322 ).
- the platform manager 250 may evaluate whether there has been any major storage changes (decision block 318 ) before determining whether there has been any major network changes (decision block 316 ).
- additional steps may be included, for example, to protect the software system 200 against its own failures.
- Such additional steps may include, for example, inserting between steps 308 and 310 described above the steps (not shown) of creating a mirror and backup copies of the platform image (including the databases 246 and 248 ), running a second instance of the software system 200 in standby mode and monitoring the primary instance of the software system 200 , and switching to the standby instance of the software system 200 upon detecting the failure of the first instance of the software system 200 .
- the software system 200 described above can operate using a supply chain software model of the ITCS 202 that it manages.
- each container system element 210 , 212 , 214 , 216 , 218 , and 220 of the ITCS 202 is modeled as a provider and a consumer of services.
- FIG. 4 is an example model 400 for service provision and consumption in a supply chain container environment. According to various embodiments as shown in FIG. 4 , which includes references to the container system elements of the ITCS 202 shown in FIG.
- the server 214 may consume services of the operating resources 220 , including, for example, power 402 , cooling 404 , physical space 406 , a share of capital expenditure (CAPEX) costs 408 , and a share of operating expenditure (OPEX) costs 410 .
- the server 214 further may consume the resources of the networks 218 , including, for example, a local area network (LAN) 420 and a storage area network (SAN) 422 .
- LAN local area network
- SAN storage area network
- the server 214 may provide the container systems 212 with various physical resource services, including, for example, CPU bandwidth 430 , memory 432 , network I/O bandwidth 434 , and storage I/O bandwidth 436 .
- the container systems 212 may also consume storage resources 438 from the storage element 216 , and, in turn, may offer services (such as services 440 and 442 ) to the application 210 .
- the application 210 may offer services to respective business activities of one or more business units 450 .
- the allocation of resources and the processing of workloads through the supply chain may be performed through the use of virtual currency.
- supply chain elements use virtual currency to pay for the services they consume and to price the services they offer.
- a selected application 210 may receive a budget from its business users reflecting the business value of the services that it offers.
- the application 210 may shop for a container system 212 that offers the lowest priced processing services that the application 210 requires, and may use its virtual budget to pay for these services.
- the container system 212 may use its income of virtual currency to pay for the services offered by the server 214 , the network 218 , and the storage system 216 .
- Each of the container systems elements 210 , 212 , 214 , 216 , 218 , and 220 of the ITCS 202 may price their services in virtual currency to reflect their costs, and additionally, or alternatively, to balance supply and demand.
- resource pricing may also be based one or both of capacity or performance characteristics.
- the server 214 may offer multiple types of processors or CPUs, each with respective clock rates and other characteristics, at different prices.
- storage I/O resources in the storage system 216 and network I/O resources in the network 218 may be priced according to their bandwidth and latency characteristics. This manner of pricing can take into account that, as noted above, I/O pathways internal to a server (i.e., interconnections of containers co-located with a single server, e.g., the containers 120 and 122 as shown in FIG.
- components and resources associated with internal I/O pathways may be priced lower than components and resources (alone or in the aggregate) for pathways traversing switches and/or involving multiple servers.
- components and resources associated with such internal I/O pathways may be priced higher to account for an expected increase in performance and thus value to the acquiring entity.
- the supply chain model of the ITCS 202 is primarily maintained by the supply chain model databases 246 shown in FIG. 2 .
- the supply chain model databases 246 may include one or more financial databases to debit and credit the respective accounts of customers and providers to reflect the transfer of virtual payments, as discussed in greater detail below. It will be understood, however, that non-monetary transactions may be entered into between a consumer and a provider.
- the supply chain model databases 246 may be object-relationship databases, such that elements of the supply chain are modeled as objects corresponding to services to be offered.
- objects refers to data structures including data fields and methods. Examples of service objects include simple and composite service objects.
- simple service objects or objects relating to the provision of a single type of service—may include the following types of attributes:
- the “service-identifier” attribute may itself include the following types of attributes as descriptors of the service that may be used for a particular class of services: ⁇ name, type, description, element manager>.
- a CPU service provided by a Dell® server with an Intel iQ9550® processor managed by an element manager ServerEM015 may be assigned the following identifier: ⁇ Dell4, CPU, iQ9550, ServerEM015>.
- the “units” attribute may measure the quantity of service, such as 5 Mhz (CPU), 2 GB (memory) or 10 Mbps (net I/O).
- the “used” attribute may refer to the amount of the service or resource capacity that is already committed.
- the “available” attribute may refer to the amount that remains to meet new demands.
- the “duration” attribute may indicated the period of time over which service is to be rendered.
- the “price(demand)” attribute may refer to a method whose input is the demand by a service consumer, for a number of service units it requires, which computes the price in virtual currency units, as set by the service provider.
- the pricing functions used by simple service objects can be flexibly adapted by element managers to reflect different pricing goals and mechanisms.
- a server may be shared by 10-100 containers, which preferably utilize no more than 50% of its capacity to avoid congestion.
- the percentage of average demand to capacity of a given server resource preferably falls between 0.5%-5%.
- a commodity service may be priced at fixed cost-based price. For example, suppose the percentage of average demand to capacity for CPU usage by a container is 0.2%. In such a scenario, the shifting of a container among servers would have negligible impact on the quality of CPU services seen by the containers. CPUs can therefore be priced at a fixed level to merely reflect the costs of providing CPUs. In general, a commodity service may be priced at a fixed level, independently of demand. However, when the ratio of average demand to capacity is sufficiently large, arriving demands may easily deplete the supply absent pricing control, thus requiring higher prices to balance the supply and demand.
- Such a pricing function is proportional to costs, penalizing high utilization.
- CPU is priced at a relatively small multiplier of the cost base of CPU
- storage I/O is priced at a relatively large multiplier of the cost base of I/O.
- Composite service objects which are objects that include more than one service object and which relate to the provision of multiple types of services, may take the following form according to various embodiments:
- service-k is either a simple or composite service object and is referred to as a component of the composite service.
- “duration” attributes of all components of a composite service are identical, and their common value is called the duration of the composite service.
- a hardware server may be described by the following composite service object:
- Memory-2, NIC-3, NIC-4 and HBA-2 indicate respective simple service objects associated with respective memory services, LAN-interface services provided by two NICs, and SAN I/O services provided by HBA-2.
- the HBA-2 may itself be described by a simple service object as follows:
- This service object indicates that the duration of the composite service is one hour, as the durations of all components of a composite service are identical.
- the price of a composite service is defined as the sum of the prices of all its components.
- the price of a server object is the sum of the prices of the units of CPU, memory, network I/O and storage I/O required by a consumer.
- the supply chain model databases 246 are maintained by element managers (such as element managers 234 , 236 , 238 , 240 , 242 , and 244 shown in FIG. 2 ), which handle the service objects corresponding to the respective elements that they manage.
- element managers such as element managers 234 , 236 , 238 , 240 , 242 , and 244 shown in FIG. 2
- an element manager is initialized by the platform manager 250 , and subsequently the element manager proceeds to populate the supply chain model databases 246 with respective service objects it is responsible for.
- the element manager continues to update the dynamic attributes of its respective service objects (such as the “used” and “available” attributes).
- a server manager 238 that is responsible for managing HBA resources will initialize the supply chain model databases 246 with corresponding simple service objects relating to the HBA. The server manager 238 will then monitor and update the “used” and “available” attributes of this simple service object by periodically accessing the HBA instrumentation.
- a consumer element manager (such as one of element managers 234 , 236 , 238 , 240 , 242 , and 244 shown in FIG. 2 ), desiring services from a provider element manager, queries the supply chain model databases 246 in search of the best priced provider or providers of the desired services.
- the query specifies requirements and the service or services the element manager is requesting.
- a query may take the following form:
- Such a query may retrieve records of composite service objects of the servers 214 offering the respective CPU, memory, storage I/O and network I/O capacity at the lowest price.
- the consumer element manager acquires these records of lowest-priced service objects, it can proceed to extract the identities of the element managers posting these service offerings.
- the consumer element manager may then pursue direct interactions and contract with one or more respective provider element managers to acquire and pay for the desired services.
- multiple consumers may query the supply chain model databases 246 simultaneously for similar services, and thus potentially interfere with each other's shopping processes. Such interference may be avoided, for example, by providing standard locking mechanisms to maintain atomicity of the query and purchase transactions.
- various embodiments may use an auction, or bidding model, rather than a shopping model, to match demand and supply.
- consumer element managers may post respective bids for services in a bidding database, which a provider element manager may then query for the highest bid price offered for its services and contract to serve it.
- the shopping model is generally preferred to bidding in situations where consumers' demands arrive asynchronously and unpredictably. In such cases, an arriving consumer can find the low-cost provider by searching the supply chain model databases 246 .
- a bidding process requires providers to poll, whether constantly or at intervals, the bidding database to detect arrivals of new bids, while bidding consumers may be required to wait until enough providers have polled the bidding database and accepted the bids, and thus contract with providers based at least in part on chance.
- bidding may offer benefits over shopping, and those situations may be handled using the principles described herein.
- FIG. 5 is a flow diagram of an example process 500 for deploying a new consumer element (such as a container) so with a provider element (such as a server) in a container system that is used according to various embodiments for balancing the demand and supply of services.
- a new consumer element such as a container
- a provider element such as a server
- the dynamic load balancing approach illustrated by example process 500 provides an effective solution to several of the resource management problems described above.
- process 500 may be used to improve the balancing of demands by containers and the supply of server resources; it may also be used to balance the resource bundle allocated to a container, e.g., to match the amount of CPU, memory and storage I/O bandwidth allocated to the container, in order to improve the use of its virtual budget to best service its resource demands.
- a consumer element manager shops for lowest cost provider for a bundle of services by querying the supply chain model databases 246 as described above (step 502 ), and contacts the provider element manager to buy services (step 504 ).
- the bundle of services to be purchased may include CPU, memory, and storage I/O.
- the provider element manager determines whether the consumer budget is sufficient to pay the price for the requested provider services (decision block 506 ). If it is determined that there is sufficient budget, the provider element manager deploys the consumer at the provider, which proceeds to process its workload (step 508 ). For example, CPU and memory resources that have been purchased may be allocated to a container by the underlying scheduler of the container system, which may include the use of a traditional operating systems scheduling algorithm. The server element manager configures the scheduler parameters to accomplish fairly accurate allocation of the CPU and memory. Memory may be allocated by specifying an amount of memory to be provided. The container system can allocate physical memory, based on these specifications, or support virtual memory mechanisms that permit over 100% utilization of physical memory. Additionally, the CPU may be allocated by configuring reservations and shares parameters of the scheduler.
- reservations may be used to allocate a reserved CPU slice, using a time-shared round-robin scheduler, while shares allocate the remaining CPU bandwidth through a Weighted Fair Queuing scheduler.
- CPU reservations and shares may be viewed as separate services, and may be individually priced according to supply and demand. For example, a low-priority application may be unable to buy reservations, and may thus need to settle for shares, which may be priced lower.
- a high-priority, mission-critical application may have sufficient budget to afford sufficient reservations to support its needs.
- the consumer element manager initiates a credit check process to decide whether the consumer can increase its budget or sufficiently lower its service demands, and thus continue to run (decision block 510 ). For example, suppose the consumer is a container whose budget is short of paying the cost of a provider server. In that case, the container may use credit it has accumulated to pay for the service, obtain additional budget from the applications it serves, or reduce its demand for services and the corresponding price to the point where it can afford to pay. If one or more of these scenarios is possible, the consumer uses credit, increases its budget and/or lowers its service demands (step 512 ), and the provider element manager thus deploys the consumer at the provider as described above. Otherwise, if none of these options is available, the consumer is suspended and then will either terminate or re-launch when adequate budget becomes available to it (step 514 ), as described in greater detail below.
- the provider element manager or the consumer element manager monitors consumer resource usage and adjusts allocation of resources to optimize or improve the use of the consumer's budget (step 516 ). For example, the provider element manager may find that the consumer is using only 20% of one service it bought, while using 90% of another service it bought. In that case, the provider element manager may reduce the allocation of the first service and use the corresponding released budget to increase the allocation of the second resource.
- the provider element manager Upon completion or termination of the consumer service period, the provider element manager notifies the consumer element manager (step 518 ), which may proceed to shop for a new provider offering lowest cost services to meet the consumer's needs (step 520 ).
- the consumer element manager determines whether the price of the new provider found is lower than the price of the old provider (where the consumer resides at the time), or according to some embodiments, whether it is lower by a threshold amount (decision block 522 ). Assuming it is, the consumer element manager moves the consumer to the new provider, in which case it may also adjust the budget to reflect the price of moving, if any (step 524 ).
- a price of moving may be factored into the decision making process for whether the consumer should be moved to the new provider, and such price may be subtracted or deducted from the available budget. Otherwise, if the consumer element manager decides to keep the consumer with the old provider, it does not adjust the budget to reflect the price of moving. In either case, the provider element manager (of the new or old provider) checks to see if the consumer budget is sufficient to pay for the provider as described above.
- the process of shopping for a new provider 520 may depend on specific characteristics of the consumer, the resource, and/or the provider.
- the containers 120 and 124 shown in FIG. 1 may need to exchange high-bandwidth latency-sensitive communications through a congested switch in the network 160 .
- internal I/O pathways (including at either the server 102 or the server 104 ) may offer higher bandwidth and lower latency, and thus result in improved performance. Therefore, according to various embodiments, such internal I/O pathways may be priced lower than I/O pathways involving, for example, multiple servers 102 and 104 and network 160 .
- the consumer element manager may determine that it would be more economical or efficient to move a consumer element from the server 102 to the server 104 based on reduced I/O pathway pricing. For example, the consumer element manager may discover that the container 120 should be moved to the server 104 to obtain one or more resources and communicate with one or more other elements located at the server 104 . This can be the case where, for example, it is determined at the step 522 that the overall price of providing container 120 with necessary resources is reduced at least in part because of a lower price of the I/O pathway should container 120 be moved to server 104 . In that case, at step 524 , the container 120 may be moved to server 104 so that the I/O pathway becomes more (or entirely) local to server 104 , thus benefiting from higher expected bandwidth capacity and lower latency.
- the budget of the consumer element may also be adjusted (e.g., increased or decreased) based at least in part in such change in pricing.
- the pricing of resources e.g., associated with the I/O pathway
- the process of shopping for a new provider 520 may depend on functional characteristics of the consumer or provider.
- the server 102 may be used to support development of containerized applications.
- the server 104 the provider, for example—may be used for testing the containerized application 130 —the consumer, in this example.
- the process 500 may be used to select a new provider (the server 104 ), from among a group of servers providing rests of containerized applications, to run tests (consumer) of the containerized application 130 .
- the server 104 may be a production system running containerized applications and the process 500 may be used to dispatch the containerized application 130 , and its container 120 , from the development server 102 to the production server 104 .
- a consumer may need to contract with two or more providers to be deployed, as in the case of a container that needs to acquire a bundle of resources offered by a server as well as SAN switch bandwidth and storage space at a storage array.
- deployment of the consumer can be supported by extending step 502 to shop for multiple providers and then repeating the remaining steps for each of these providers.
- the example process 500 shown in FIG. 5 may be modified or extended to enable the adjustment of resource allocations to obtain desired service level agreements (SLAs).
- SLAs service level agreements
- the above-described supply chain economic principles may also be used to manage software licenses, such as temporary (time-limited) software licenses.
- software licenses such as temporary (time-limited) software licenses.
- type such as authorizations of software use per user, per CPU, per server, or per container
- licenses may be modeled as resources to be purchased by an application manager 234 , much like other resources that it may purchase from the container 212 .
- License element managers (while not shown, may be included as part of the platform layer 230 ) may be used to set the prices of the licenses based on costs and demands In this manner, license management may be greatly simplified and unified with the allocation of other types of resources.
- an application that is unable to acquire a needed license may suspend its operations and release its resources, as explained below, thus increasing the overall efficiency of the system.
- licenses may be more efficiently used, since in situations where the licenses are highly utilized, they will be allocated to high priority tasks, while lower priority tasks may be suspended until they can afford the licenses. As soon as a license is no longer needed, it may be released and available for other tasks.
- an administrator may consider the ROI of licenses, as with other resources, to plan the expansion, or contraction, of licenses capacity. For example, if a license's ROI is above a certain threshold, it may be desirable to acquire more licenses to increase the supply to meet demand.
- FIG. 6 is a flow diagram of an example process 600 for delivering service level agreement targets through resource allocation in a container system, which includes many of the steps of process 500 shown in FIG. 5 and discussed above. Although not required, for the purpose of simplifying the following description, it is assumed that the target service level agreement relates to an application running on a container. However, the service level of other types of computer elements may be controlled in the following manner according to various embodiments.
- the container element manager monitors and obtains the value of the SLA parameter of interest, such as the average transaction rate of an application, the average transaction delay of an application, the average communications latency of the application, or the number of transactions performed within a predetermined prior time period by an application (step 604 ).
- the container element manager may monitor the value of the SLA parameter, through respective instrumentation, and inform the container element manager of the SLA parameter.
- the application may define its SLA goal as 100 transactions per second, in which case the SLA parameter of interest is transaction-rate.
- SLA parameters can be assumed to increase monotonically with the amount of resources allocated to an application, the management of SLAs may be accomplished as described herein by finding a budget and a respective resource allocation that will accomplish the target SLA value.
- the container element manager determines whether the SLA parameter of interest is below a desired target (decision block 606 ), in which case, for example, the application's payments to the container (e.g., of virtual currency units) are increased such that the container's budget is increased, and it is able to purchase more resources to increase the SLA parameter of the application (step 608 ). After such an increase, the container's budget use is again monitored and optimized or improved as described above.
- the application's payments to the container e.g., of virtual currency units
- the container manager determines that the SLA parameter is at or above the desired target, it is determined whether the SLA parameter exceeds the desired target by more than an acceptable threshold (decision block 610 ), in which case the payments are reduced, thus reducing the container's budget and the resources it buys, saving on applications costs, and keeping the SLA performance within a desired tolerance range (step 612 ). After such a reduction, the container's budget use is again monitored and optimized or improved as described above. If the SLA parameter is within the acceptable range, however, a reduction is not applied, and the process is repeated until it is determined that the consumer service period has been completed or terminated.
- the process 600 for delivering service level agreement targets through resource allocation in a container system may be modified, adapted, and/or simplified for certain resources and SLA metrics.
- the SLA parameter may be selected as the latency-hop-count, e.g., the number of physical switches traversed by an I/O pathway.
- I/O pathways between elements located, or resident, at the same server e.g., the containers 120 and 122 in FIG. 1
- Such I/O pathways may also be referred to as having Class-0 Latency SLA.
- I/O pathways between elements located or resident at different servers may be described as having a latency-hop-count of 1, and may be referred to as having Class-1 Latency SLA.
- an I/O pathway may involve two or more physical switches, and may be described as having a latency-hop-count of 2 (or more) and referred to, for example, as having Class-2 Latency SLA.
- the latency-hop-count associated SLA value may be described with respect to the ordinal preference ⁇ Class-0, Class-1, Class-2, . . . Class-n ⁇ , where Class-0 is preferred to Class-1, Class-1 is preferred to Class-2, and so on to the extent additional Classes are defined.
- the process 600 described with respect to FIG. 6 can be modified in a manner consistent with the above description so as to simplify the monitoring and control of SLA values to classification of the I/O pathway into latency class.
- the SLA-delivery process 600 described above may be flexibly adapted to achieve various goals, such as improving its handling of stochastic fluctuations of an SLA parameter.
- the steps of increasing (step 608 ) and decreasing (step 612 ) payments by the application to the container may use standard mechanisms of Stochastic Approximation theory, including the Robbins-Monro or Kiefer-Wolfowitz algorithms, to regulate the changes in payments to assure convergence.
- Such a design may be implemented, for example, to achieve more desirable results in connection with non-monotonic SLA parameters.
- an embodiment using a Robbins-Monro procedure may replace steps 606 - 612 with the following iteration: R(n+1) ⁇ R(n)+a(n)[SLATarget ⁇ SLAParameter(R(n))]
- n is a counter of the iterations
- R(n) is a vector describing the resource bundle allocated after n iterations
- SLATarget is the desired value of the SLAParameter
- SLAParameter(R(n)) is the observed value of the SLAParameter after n iterations.
- the SLA-delivery process 600 described above uses an economic model and virtual currency units to control SLA levels
- other manners of controlling SLA levels may be used according to various embodiments.
- the allocation of resources to a container, or to an application may be independent of any economic budget or transfer of virtual currency units, and may instead be based on other measures of an application's or container's importance.
- the process 500 described above may also be modified or extended according to various other embodiments.
- the process 500 described above may be modified or extended as shown in FIG. 7 to facilitate the handling of storage I/O.
- FIG. 7 is a flow diagram of an example process 700 for economic-based I/O scheduling in a container system, which includes many of the steps of the process 500 shown in FIG. 5 and discussed above.
- the consumer is a container
- the provider is a server
- the resource is storage I/O.
- the resource being managed may be other types of I/O, such as network I/O.
- the server element manager monitors storage or network I/O usage by one or more containers, such as by collecting data from one or more of the container system, the HBAs (step 702 ), or the NIC.
- the server element manager may be configured to prevent congestion along storage I/O pathways, as might occur in cases of usage levels approaching the capacity limits.
- the server element manager may prevent congestion by using pricing functions as described below that increase prices dramatically when utilization approaches 50% of the capacity.
- the server element manager optimizes or improves the resources allocated to containers, as described above (step 516 ), such that containers acquire a share of the storage I/O resources that is commensurate with and optimally reflects their budget.
- the server element manager then periodically estimates both the average storage I/O capacity used and the average available I/O capacity, and updates the respective attributes of the storage I/O objects in the above-described supply chain model databases 246 with this usage data (step 704 ). It is noted that the usage data reported to the supply chain model databases 246 will impact price computations, with excessive utilization of storage I/O capacity resulting in respective price increases, and higher prices in turn deflecting demand by new or existing containers to servers with lower utilization (and prices) of storage I/O.
- price competition over using storage I/O resources may result in migration of low budget containers from overloaded servers to other servers where storage I/O resources are more highly available, and are thus priced lower.
- Higher priority containers may use their higher budgets or credit to obtain a preferential share of storage I/O resources.
- the server element manager also computes the actual (versus projected) costs expended by each container, and applies these prices to handle its current commitments to containers (step 706 ). For example, higher usage of storage I/O results in higher prices and immediate costs assigned to containers, such that containers of lower priority and high storage use requirements may quickly exhaust their budget or credit and be suspended or terminated, as described below. In this manner, the low priority containers relinquish storage I/O capacity to containers having a higher priority and, thus, a higher budget.
- the server element manager evaluates whether the container's budget is sufficient to pay the cost (decision block 708 ). If it is, the service period of the container continues until it ends, and the server element manager notifies the container element manager of the completion of the service period (step 518 ).
- the server element manager evaluates whether the container's credit (costs minus budget) exceeds an acceptable credit threshold (decision block 710 ).
- the container's credit may have higher budgets and credits and can thus afford to overpay the server element manager to guarantee that they do not run out of storage I/O resources. If it is determined that the container's credit exceeds the threshold, the container element manager initiates a credit check process to decide whether the container can increase its budget or sufficiently lower its service demands, and thus continue to run (decision block 712 ).
- the container makes any necessary adjustments (such as a budget increase in the case of high priority containers, or reduced service demands) and continues to run (step 714 ), until the service period has ended and the server element manager has notified the container manager of the termination of the service period as described above. Otherwise, the server element manager suspends or terminates the container execution and notifies the container element manager, which becomes responsible for addressing the suspension or termination (step 716 ).
- the server element manager Upon termination of the service period and notification to the container element manager, the server element manager reports usage data to the container element manager and settles any credit, overpayments or underpayments with the container element manager (step 718 ). The container element manager may then proceed to shop for a new server offering lowest cost services to meet the container's needs (step 520 ), as explained above.
- the economic-based scheduling process 700 described above may be used effectively to de-correlate peaks of competing, bursty I/O flows. For example, consider the scenario of four containers sharing a common server and a 4 Mbps Fiber Channel HBA, where the containers generate average storage I/O flows of 250 Mbps, 250 Mbps, 200 Mbps and 300 Mbps, respectively. The aggregate demand average of 1 Gbps consumes only 25% of the HBA capacity. A resource scheduler may limit its consideration to only the average demand which, in this case, would be manageable by the HBA and SAN. However, consider an alternate scenario where the I/O traffic streams are bursty, with a peak/average ratio of five for each container.
- the four I/O streams associated with the containers are uncorrelated, their peaks will be likely dispersed and the peak of the aggregate stream will generally be less than 2 Gbps, which can be handled by the HBA and SAN with negligible or relatively few queuing delays.
- their peaks may be compounded to generate, for example, up to 5 Gbps peaks, utilizing 125% of the capacity and generating sustainable congestion, delays, and losses.
- the scheduling process 700 described above reduces the likelihood of compounded peaks, since they result in peak prices and a corresponding depletion of budgets and credits of low budget containers, leading to suspension, termination, or migration of such containers to servers with lower storage I/O prices until they find servers where their peaks are sufficiently de-correlated from other containers.
- the economic-based scheduling process 700 described above substantially reduces interference not only between traffic averages, but it also reduces the interference between correlated traffic peaks. This results in smoother, less bursty, aggregate workloads, which may permit more efficient processing.
- the process 700 described above to manage storage I/O flows may applied to other forms of I/O, such as network I/O.
- network I/O typically utilize network-I/O flows, such as Ethernet (e.g., Fibre Channel over Ethernet (FCoE)), Transmission Control Protocol/Internet Protocol (TCP/IP) (e.g., Network File System (NFS)), and SAN (e.g., Fibre Channel (FC), Internet Small Computer System Interface (iSCSI)), to transfer information such as storage access commands.
- Ethernet e.g., Fibre Channel over Ethernet (FCoE)
- TCP/IP Transmission Control Protocol/Internet Protocol
- NFS Network File System
- SAN e.g., Fibre Channel (FC), Internet Small Computer System Interface (iSCSI)
- the scheduling process 700 is therefore independent of the specific underlying network, and of the specific access commands carried by the described flows. Accordingly, the process 700 may be applied to schedule network I/O flows and thereby provide similar or identical benefits to those associated with storage I/O flows, such as smoothing the peaks of bursty traffic and/or supporting priority services.
- scheduling process 700 The order of steps described above with respect to scheduling process 700 is illustrative only, and can be done in different orders. Moreover, the aforementioned beneficial effects are true not only for I/O streams, but for workloads sharing other resources as well.
- the contracting of services between a consumer and a provider may include the use of a standard request-response protocol (such as SOAP) to submit a purchase order to the provider and transfer a respective payment.
- a standard request-response protocol such as SOAP
- the provider may deploy the service requested by the consumer and respond with a service confirmation.
- FIG. 8A is an example purchase order data structure 800 issued by a consumer element manager for use in purchasing services from a provider element manager.
- the first two fields of the data structure 800 source-ID field 802 and provider-ID field 804 , respectively identify the source consumer and destination provider.
- the third field, transaction-ID field 806 identifies the particular purchase order.
- the fourth field of the data structure 800 service field 808 , identifies the service and provides parameters to quantify the purchase.
- the fifth field of the data structure 800 , payment field 810 provides payment data including payment amount and authentication data to establish the validity of the payment.
- the sixth field of the data structure 800 authentication field 812 , provides data to authenticate the validity of the purchase order transaction.
- FIG. 8B is an example service confirmation data structure 850 issued by the provider element manager for use in confirming or rejecting the purchase of services by the consumer element manager.
- the first three fields of the data structure 850 , source-ID field 852 , provider-ID field 854 and transaction-ID field 856 correspond to the first three fields of the data structure 800 described above.
- the fourth field of the data structure 850 , service confirmation field 858 includes data to confirm the service and enable the source to access it. Alternatively, assuming the provider has rejected the transaction, the service confirmation field 858 would include data with the reason or reasons for rejection, such as insufficient resources or a price change.
- the fifth field of the data structure 850 , authentication field 860 provides data to authenticate the validity of the service confirmation.
- various embodiments may also be used to address the problems of container sprawling and energy consumption in container systems using supply chain economics.
- these embodiments may be used to suspend or terminate containers that are no longer needed or productive.
- These embodiments may also be used to terminate containers, or to disallow their re-activation if in a standby state, that are determined to be inconsistent with the current versions of their container system and applications.
- energy consumption these embodiments may be used to consolidate and shift containers into fewer servers, for example, while still providing desired SLA performance, and switching other unused or non-productive servers OFF or into standby mode to reduce energy use.
- the supply chain software model and processes described above provide mechanisms and metrics to quantify how productive or non-productive a service element is.
- the following description details an example process 900 , shown in FIG. 9 , for managing the states of container system elements, which as explained further below, may be used to address sprawling and energy consumption issues.
- the system element is a container, although the general principles that follow may be readily adapted for any type of system element.
- a container is first initialized, for example, through the use of an initialize signal generated by a management station (step 902 ) or an automated action of a container manager.
- an application element may interpret events generated by a launch as an initialize signal.
- the container After being initialized, the container attempts to obtain an initial budget to acquire resources for its operations (step 904 ). It is next determined whether the container was successful in obtaining an initial budget (decision block 906 ), in which case the container tries to acquire the resources needed to launch a respective service component (step 908 ). Otherwise, it begins the termination procedure by releasing any resources allocated to it (step 910 ).
- the container If the container is successful at acquiring resources (decision block 912 ), it is provisioned, deployed, and remains in an active state (step 914 ) until it receives a signal to switch the service element OFF to an idle or standby state (step 916 ). After the terminate signal has been received, the container begins the termination procedure by releasing resources allocated to it, as described above.
- the container will wait an amount of time for sufficient resources to become available before attempting to acquire resources again (step 918 ). For example, during this waiting period, the container may use an exponential “backoff” mechanism, whereby the container repeats its attempts to acquire resources, but doubles the waiting period between repetitions with every failure. If it is determined that the container should continue to try to acquire resources (decision block 920 ), it will do so as described above in step 908 . Otherwise, for example, if failures persist beyond some timeout period, the container abandons attempts to launch and begins to terminate.
- the container determines whether the container should remain in a standby state (decision block 922 ), in which case the execution of the container stops, but it remains in a suspended or standby state and retains sufficient state data, for example, by using storage services to retain state data in image form, and for which the container may be required to pay (step 924 ). Otherwise, the container terminates execution and may be deleted (step 926 ).
- the applications being executed by the container are first terminated, and then the container is terminated.
- a graceful termination may be pursued through a recursive termination of the supply chain elements supported by the container.
- a container element manager may issue a terminate signal to a corresponding operating system manager, which propagates the signal to an application manager, which in turn signals termination to is application.
- the application may then begin the termination steps as described above with respect to the process 900 , after which a termination complete signal to the application manager, and is forwarded to the operating system manager, which in turn sends a terminate signal and receives a termination complete signal back from the operating system.
- the operating system's termination complete signal may be forwarded to the container manage, which can signal the container to terminate.
- a notification procedure may be invoked to notify administrators of pending terminations or suspensions, such that termination or suspension may only be completed once administrator permission has been received.
- a container in standby state it is determined whether termination should follow (such as by receipt of a terminate signal) (decision block 928 ), in which case the container terminates execution as described above. Otherwise, for example, if it is determined that the container should re-activate, the container seeks to obtain a budget to acquire resources for its operations as described above, for example, upon receiving an initialize signal. It will be understood that the specific actions described above in connection with process 900 may be modified for non-container system elements, and that the order of steps in process 900 are also illustrative only.
- a process such as process 900 described above may be used to control container sprawling by suspending or terminating non-productive system elements, such as containers.
- ROI the relationship between the payments it collects from applications and the prices it pays for underlying server and I/O resources. If the container's ROI is greater than one, the container is earning more than it expends, and the container may be classified as being productive in creating applications value that exceeds the costs of the infrastructures it uses. However, if the container's ROI is less than one, this means that the container produces less value than the cost of resources it consumes, and the container may thus be classified as non-productive.
- ROI is one example of a metric of productivity that may be used in determining whether a system element should be suspended or terminated, or whether it should remain active.
- a process such as process 900 described above may be used to assure, for example, that applications' budgets are sufficient to keep one or more containers' ROI greater than one, and to notify applications' administrators (element managers) as needed when budgets are low. It the ROI of one or more containers remains less than one for more than a threshold period, for example, it may indicate that an application's budget is too low to sustain productive operation, and that the corresponding, non-productive container should be suspended or terminated.
- a container may receive a terminate signal to switch it OFF to an idle or standby state (per step 916 of process 900 described above) as soon as the container's productivity level or score (for example, measured by its ROI) has been determined to be less than one for a predetermined time period. Additionally, for example, the length of time that the container's ROI has been less than one may be a factor in deciding whether the container should be terminated, or only suspended for the time being.
- the process 900 described above and similar processes may also be used for energy management.
- such processes may be used to suspend or terminate (switch OFF) servers that are classified as being non-productive, as in the case where a server's ROI is less than one for a sufficiently long period of time.
- the server element manager much like the case of the container manager described above, can monitor the ROI and detect termination or suspension conditions.
- the server manager may then pursue a termination process, similar to the recursive termination process described above, where all containers on the server are first terminated, or moved to another server, before the server manager suspends the server into Standby state (so as to consume less energy and cooling resources, for example) or switches the server OFF.
- process 900 and similar processes may also be used to assure consistency of a suspended container with changes in applications.
- the container manager may prevent such inconsistencies by sending a terminate signal, as described above, to all containers whenever their respective operating system or applications software has changed, thus causing the applicable containers to transition from standby to terminate state, at which point it may be deleted.
- FIG. 10 is a block diagram of an example multi-domain software system environment 1000 for managing virtualized resources in “multi-cloud” systems.
- container environment 1000 includes two example software systems 1002 and 1004 , each of which is similar to the more detailed example software system 200 shown in FIG. 2 , and which operate in a first and second domain, respectively.
- the software system 1002 operating in the first domain includes a user interface subsystem 1006 and one or more functional managers 1008 and 1010 . Together, these elements make up a functional management layer 1012 of software system 1002 , and provide specific management applications as described above in connection with FIG. 2 .
- Software system 1002 also includes one or more element managers 1014 and 1016 , which monitor and control one or more respective container stack elements 1018 and 1020 .
- the software system 1002 also includes one or more databases 1022 (such as the supply chain databases 246 and operations databases 248 described with reference to FIG. 2 ), as well as a platform manager 1024 . These elements are included in a platform layer 1026 of the software system 1002 to provide the infrastructures for monitoring the container stack elements 1018 and 1020 , modeling these container stack elements as part of a supply chain economy, and controlling the operations of the container stack elements, as described above.
- the software system 1004 operates in the second domain, includes similar elements as the software system 1002 , and also includes a proxy manager 1030 .
- the domain software system 1004 exports one or more resources or services to the domain software system 1002 by using the proxy manager 1030 .
- the proxy manager 1030 exports instrumentation to monitor and control these provided resources to one or more of the element managers 1014 and 1016 , such as container element managers, of the first domain software system 1002 .
- the first domain software system 1002 may view the second domain software system 1004 as a service element integral with its supply chain model.
- the second domain software system 1004 is in complete control of the resources (or services) and capabilities exported to the first domain software system 1002 .
- the software system 1004 may be an external cloud provider exporting raw server services to the software system 1002 .
- the software system 1002 can access these services, using its local element managers 1014 and 1016 , to allocate, for example, CPU, memory, and storage resources at the second domain software system 1004 and then monitor and control their use and operations.
- software systems 1002 and 1004 are separately owned and/or managed.
- software system 1002 may be owned and operated by a small business that experiences steady computing needs except for two hours in each day, during which time its computing needs are consistently elevated.
- software system 1002 may lease or purchase additional computing resources from software system 1004 (e.g., owned by Amazon.com, Inc.) on an as-needed basis and transfer excess workloads to software system 1004 (“bursting”).
- computing resources from software system 1004 may be leased or purchased to facilitate the execution of a multi-tier web service by a cluster of containers (or applications).
- the software system 1002 may lease or sell resources from software system 1004 to execute this cluster of containers (or applications) and then migrate the container cluster (or application cluster).
- the migration may take place from a private cloud of a small business to the public cloud of another business (e.g., of Amazon, Inc.). It is noted that, according to various embodiments, even if needed computing resources are available from within software system 1002 , such resources may be purchased from software system 1004 based on relative price offerings.
- the asymmetric relationship between software systems 1002 and 1004 shown in FIG. 10 and described above may be extended to provide full symmetry.
- the first domain software system 1002 would incorporate its own proxy manager (not shown) to export services to the second domain software system 1004 , which would integrate it within its supply chain through one or more of its respective element managers.
- the supply-chain principles discussed herein may be used to scale containers up/down, by adding or removing resources to existing container components, or to scale containers out, by adding more container components or suspending containers.
- the decisions to scale up/down and out are based on a supply chain 1100 outlined in FIG. 11 , and the revenues and expenses of each of the entities involved in that Figure.
- the supply chain 1100 may also be used to determine the sizing and placement of containers, when a selected container is deployed, and to determine future sizing and placement requirements based on anticipated changes in container load.
- the supply chain 1100 may include two types of entities, namely, Service Entities (SEs), such as a Virtual Machine 1110 or a Container 1120 , and Resources, such as CPU Allocation (CPUAllocation) 1102 and Memory Allocation (MemAllocation) 1101 .
- SEs Service Entities
- Resources such as CPU Allocation (CPUAllocation) 1102 and Memory Allocation (MemAllocation) 1101 .
- CPUAllocation CPU Allocation
- MemAllocation Memory Allocation
- the market may suggest an increase of (scaling up) the Memory Allocation 1101 of the Container 1120 , or it may suggest the creation of another instance (scaling out) of the Container 1120 .
- decisions to scale up/down will apply to Resources only, while decisions to scale out will apply to SEs only.
- the MemAllocation 1101 of the Container 1120 may reduce as a result of congestion for resources at the Virtual Machine level. Increased utilization of MemAllocation 1101 of the Virtual Machine 1110 will lead to increased MemAllocation price. In turn, the increased MemAllocation price increases expenses of MemAllocation for the Container 1120 , leading to a decision to reduce the size of MemAllocation of the Container 1120 .
- the Container 1120 consumes directly from a Physical Machine 1210 .
- the MemAllocation size may also reduce as a result of congestion for resources at the Physical Machine level. Increased utilization of Physical Machine MemAllocation will lead to increased MemAllocation price, which in turn increases expenses for MemAllocation on the Container 1120 , leading to a decision to reduce the size of MemAllocation of the Container 1120 .
- Container MemAllocation size may increase as a result of over provisioned resources at the Virtual Machine level. Decreased utilization of Virtual Machine CPUAllocation due to a high capacity will lead to decreased CPUAllocation price, which in turn decreases expenses for CPUAllocation on the Container 1120 . If the Container 1120 has high revenues for CPUAllocation this would lead to a decision to increase the capacity of CPUAllocation on the Container 1120 .
- the price of MemAllocation can be set to a minimum value to force higher expenses if attempting to maintain the size of the MemAllocation of the Container at or below some value. This advantageously avoids unnecessary resizing only for the purpose of having additional MemAllocation. Accordingly to other embodiments, the price of MemAllocation can be set to a maximum value.
- FIG. 13 shows an example process 1300 , which illustrates how a decision is made to scale a resource allocation up or down.
- the process 1300 first determines if the revenue/expense of a commodity is greater than a predetermined value X (decision block 1301 ). If so, then the capacity of the resource is scaled up until the revenues are equal to the expenses (step 1302 ). If the revenue/expense of the resource is less than a predetermined value Y (decision block 1303 ), then the resource allocation is scaled down until the revenues are equal to the expenses (step 1305 ). Otherwise, if the revenues/expense of the resource is within the range defined by the values X and Y (decision blocks 1301 and 1303 ), then the resource allocation is not scaled (step 1304 ).
- the values of X and Y provide a mechanism to tune the responsiveness of the system to increases or decreases in demand.
- the value of revenues/expenses captures the profitability of the resource allocation (or the SE). If the ratio is >1, the resource is profitable. If it is ⁇ 1, it is losing money.
- revenues and expenses can refer to the accumulated revenues and expenses over a period of time. Different periods of time can be used to adjust the decision-making behavior (e.g., aggressive versus conservative behavior). Short time frames lead to aggressive decisions, where the system responds very quickly to changes in the supply and demand anywhere along the supply chain. This can be used, for example, to respond quickly to congestion for resources and guarantee the quality of service offered to the entities in the system. Long time frames dampen the effects of short-term changes in the supply and demand, and reflect accurately the longer-term trends of the demand and supply.
- FIG. 14 illustrates an exemplary process 1400 for scaling SEs.
- the process 1400 concerns creating a new instance of a SE, or suspending the operation of an existing SE, depending on the expenses and revenues of the SE.
- the process 1400 first determines whether the revenue/expense of a SE is greater than a predetermined value X (decision block 1401 ). If so, then a new instance of the SE is created (step 1402 ). If the revenue/expense of the SE is less than a predetermined value Y (decision block 1403 ), then the operation of the SE is suspended (step 1405 ). Otherwise, if the revenues/expense of the SE is within the range defined by the values X and Y (decision blocks 1401 and 1403 ), then the SE is unchanged (step 1404 ).
- the supply-chain principles discussed herein also may be used to manage application performance in other virtualization systems.
- an application server requires a certain amount of memory and CPU resources.
- a database will also require a certain amount of storage.
- the application In order for the application to perform adequately, the application must be allocated a sufficient amount of resource. In order for the infrastructure to be utilized efficiently, the application should only consume what it requires at any given point in time.
- the supply-chain principles discussed in FIGS. 13 and 14 can be used to scale up/down, by adding or removing resources allocated to the application, or to scale out, by adding more application components, or suspend application components.
- Some examples of application resources include, without limitation, java heap, thread pools, and connection pools in an application server or data space and log space in a relational database.
- the supply chain 1500 includes the two types of entities discussed with reference to FIG. 11 .
- the supply chain 1500 illustrates the SEs, such as the Physical Machine 1210 , the Virtual Machine 1120 , or an Application Server 1530 , and the Resources, such as Memory (Mem) 1501 , Virtual Memory (VMem) 1502 , and Heap 1503 .
- the SEs such as the Physical Machine 1210 , the Virtual Machine 1120 , or an Application Server 1530
- the Resources such as Memory (Mem) 1501 , Virtual Memory (VMem) 1502 , and Heap 1503 .
- the resources and SEs have expenses and revenues.
- the revenues of a virtual central processing unit (VCPU) 1504 sold by the Virtual Machine 1120 are generated from the Application Server 1530 buying this resource.
- Expenses of the VCPU 1504 come from paying to acquire a necessary resource, such as CPU 1505 , from the underlying Physical Machine 1210 hosting the Virtual Machine 1120 .
- a SE has revenues which can be the sum of the revenues of the resources it sells, while its expenses can be the sum of the expenses of the resources it buys.
- the revenues of the Virtual Machine 1120 can be the sum of the revenues of the VCPU 1504 and the VMem 1502 that it sells to the Application Server 1530 in FIG. 15 , while its expenses are the sum of the expenses to acquire the CPU 1505 and Mem 1501 from the Physical Machine 1210 .
- Revenues and expenses can depend on the prices of resources, which in turn can be a function of supply, e.g., attributes of the resource such as its capacity, as well as the demand—how much of the capacity is currently utilized by resources or SEs consuming this resource.
- price is a function of the utilization (U) of the resource, and depends on it through the formula:
- an application server requires java heap in order to process transactions.
- This java heap is allocated from the underlying virtual machine's virtual memory allocation.
- the demand for java heap is very high (e.g., generating revenue for the application server), and the price of virtual memory from the virtual server (e.g., determined by the combination of supply and demand) is sufficiently low, then the application server will be able to buy more virtual memory from the virtual server and allocate additional java heap.
- the application server will decrease its allocation of java heap and return virtual memory to the virtual machine to be used by other applications.
- the buyer can be assigned a budget for purchasing the resources.
- VMem can be set to a minimum value to force higher expenses if attempting to maintain the size of the Heap at or below some value. This advantageously avoids unnecessary resizing only for the purpose of having additional VMem. Accordingly to other embodiments, the price of VMem can be set to a maximum value.
- the market may suggest to increase (scale up) the Heap size of an Application Server, or it may suggest to create another instance (scale out) of the Application Server. These decisions can be based on the process 1300 for resizing resources and process 1400 for scaling SEs as discussed above.
- revenues and expenses can refer to the accumulated revenues and expenses over a period of time and different periods of time can be used to adjust the decision-making behavior (e.g., aggressive versus conservative behavior). For example, longer periods of time can be used to anticipate future needs for extra application servers based on steadily increasing revenues that reflect an increase in demand. Conversely, a longer term decrease in revenues indicates that the steady state operation of a system may not require a particular SE.
- supply chain economic principles and other principles explained above serve several purposes and provide several potential benefits, both expressly numerated and otherwise.
- these principles can be used to provide a common software framework and abstractions to unify and automate the management of container systems. More specifically, they can be used to optimize or improve the allocation of IT resources (such as I/O resources or software licenses) to best process applications workloads according to their business value.
- IT resources such as I/O resources or software licenses
- the principles of supply chain economics can also be used to balance workloads to minimize disruptive operating conditions, such as I/O congestion, and to reduce resource waste by terminating or switching-off underutilized resources.
- These principles can also be used to empower business units to monitor and control the delivery of SLAs to their applications, as well as the ROI of individual elements and the overall container system.
- these principles can be used to handle the management of virtual resources in a multi-cloud (or multi-domain) system.
- FIG. 16 illustrates an exemplary system 1600 for regulating access of consumers 1610 (e.g., electronic applications) to resources and services (e.g., storage).
- this regulation occurs through the use of access permits (not shown) that the consumer 1610 acquires from an intermediate entity—an Action Manager (AM) 1620 —prior to accessing the resource or service.
- the AM 1620 regulates access to a provider 1630 of the resource or service.
- regulating access includes controlling the number of concurrent accesses, and/or the rate at which consumers 1610 access the resource, as desired.
- the AM 1620 can sell multiple types of action permits, regulating access to a number of resources. Each permit can be associated with a predetermined price. Additionally and alternatively, this price can be dynamically adjusted taking into consideration the availability of permits possessed by the AM 1620 .
- Permits sold by the AM 1620 can create both revenues and expenses for the AM 1620 .
- the revenues come from the price the consumer 1610 has to pay to the AM 1620 to buy the permit.
- the expenses come from the price the AM 1620 has to pay to the resource provider 1630 for the right to sell these permits.
- the AM 1620 may need to pay for Input/output Operations Per Second (IOPS) offered by a storage controller in order to allow access to the consumer 1610 .
- IOPS Input/output Operations Per Second
- the price that the AM 1620 pays for the right to sell these permits is determined by the provider 1630 based on one or more of the following parameters: the capacity and the percentage the provider 1630 wishes to make available to the consumers 1610 ; the current load of the provider 1630 ; and the rate at which the provider 1630 wishes its resources to be accessed.
- the AM 1620 dynamically can adjust the number of permits it possesses at any time, depending on its revenues and its expenses. For example, if the AM 1620 is profitable (e.g., the charges based on price it is selling the permits to the consumer 1610 is higher than the charges based on price it pays to the provider 1630 for the right to sell these permits), the AM 1620 can consider increasing the number of permits it sells. Alternatively, if the AM 1620 is losing money, the AM 1620 can consider decreasing the number of permits it is selling.
- the AM 1620 can be used to avoid I/O congestion in storage controllers when several VMs request to execute heavy-storage applications (e.g., VM Reboots, Antivirus database updates, OS Updates, and so on) at the same time.
- the AM 1620 limits the number of concurrent consumers that can access the provider 1630 . It may limit access across types of applications or within each type of application. For example, permits can be priced and provided for all anti-virus, OS updates, etc. separately, or all of them may be constrained by the same permits.
- the provider 1630 is the storage controller, while the consumer 1610 is the application performing the heavy-storage task. For instance, the application can be performing an anti-virus update on the virtual machine.
- the consumer 1610 sends the AM 1620 a request 1601 to acquire the appropriate number of permits (e.g., 5) for the provider 1630 (e.g., a storage controller) of the storage associated with the VM.
- the provider 1630 e.g., a storage controller
- the AM 1620 subsequently determines 1602 if the request includes a sufficient budget, and if the AM 1620 has enough permits to satisfy the request 1601 . If so, the AM 1620 replies to the consumer 1610 with the appropriate permits and charges.
- the consumer 1610 accesses 1602 the storage through the provider 1630 and performs the update. After completing the update, the consumer 1610 releases 1604 the permits such that the AM 1620 can re-sell them.
- the AM pays 1605 the provider 1630 for the use of the permits it is selling. According to various embodiments, payment for the use of permits can occur before, after, or simultaneously with storage access.
- the number of concurrent accesses to a resource may vary.
- the AM 1620 adjusts the number of permits it is selling, to reflect the ability of the provider 1630 to satisfy concurrent requests by consumers 1610 .
- the AM 1620 pays the provider 1630 for the use of the permit
- the AM 1620 adjusts the number of the permits it sells based on how profitable it is. If demand for permits for a specific provider 1630 is high, the AM 1620 raises the prices for this permit, advantageously increasing revenues.
- the AM 1620 can request the right to sell more permits from the provider 1630 . If the provider 1630 agrees, the provider 1630 raises the price the AM 1620 has to pay for these rights. As the demand increases, the provider 1630 continues to increase the price it charges the AM 1620 . At a threshold price, the AM 1620 can no longer make a profit, and the AM 1620 does not request any further increase in the number of rights it can sell. Similarly, the number of permits sold by the AM 1620 can decrease as a result of reduced demand by consumers 1610 , or increased prices by the provider 1630 .
- the AM 1620 controls rate of concurrent accesses to a particular resource.
- the AM 1620 limits the rate at which the applications are accessing the storage controller to perform the heavy-storage tasks. In this case, once the application releases the permit, and until the predetermined period of time has elapsed, the AM 1620 cannot resell this permit.
- the storage controller can charge the AM 1620 a very small amount for the right to sell a first predetermined number of permits within a period of time, and then increase the price to infinity for permits beyond the first predetermined number in this period.
- the consumer request to access one or more permits is made directly to the resource or service provider.
- the AM 1620 controls the total number and/or the rate at which a group of consumers accesses a group of resources.
- “Formulating” includes the attributes taken into account when considering the option to move to the new provider.
- the cost of moving can be part of the comparison between two different alternatives (e.g., keeping a VM in an existing infrastructure or moving the VM to an external cloud provider). Cost can be expressed in actual currency or any unit suitable for the comparison. For example, moving time can be expressed in a real value that quantifies the cost of the VM downtime. In contrast, if there is a strict limit on acceptable downtime, the cost of moving the VM can be expressed in terms of time.
- “Evaluating” includes making the decision (e.g., initiating an action based on the decision) and determining the right time to take the action. Compared to other economics-based decision-making systems, one embodiment described herein postpones the decision for the future, advantageously waiting for a sufficient amount of time until the decision-maker is convinced that the decision is the right one.
- a virtualization system 1800 controls moves of VMs 1810 between different storage (or resource) providers 1820 to avoid frequent moves of VMs 1810 between different storage providers 1820 in a datacenter (DC) 1830 or across different datacenters.
- DC datacenter
- the VM 1810 is evaluating a move to one or more service providers 1820 , such as storage providers SP 1 , SP 2 , . . . SP N .
- service providers 1820 are used herein as an example, it will be understood that the concepts disclosed herein can be applied to other types of service or resource providers.
- the cost C(Ai) of moving to provider i is set to a value that is proportional to the size of the data to be moved from the current SP to SP i , multiplied by a factor P i that captures the ‘proximity’ of the current SP to SP i .
- P i could be set to 1
- the consumer periodically checks the prices at the current and each provider i, calculates the saving for this period and adds them to the savings from the previous periods.
- the price of the new provider for the current period may be higher than that of the current provider, and as a result the savings for this period will be negative and will decrease the total savings from previous rounds.
- the current provider gives the consumer some credit (e.g., equal to C(A)) to convince the consumer to stay.
- the consumer accepts the credit, and periodically checks the price of the new provider. If the price is cheaper, the consumer can use this credit to subsidize any loss of not having moved there. If it is more expensive, the consumer adds her gain to the credit. If the consumer runs out of credit, then the consumer can decide to move.
- the system accounts for the fact that a decision that looks good now may not be good in the future. For example, a consumer that buys bandwidth from a network provider may see a cheaper price offered right now by a new provider. However, the new provider may change the price an hour later, and this new price may be higher than the price of the current provider an hour later.
- the system accounts for the actual behavior of other users. Assume a VM is interested in the latency of accessing data stored on a disk, and a decision is made to move its data from the current to a new disk that currently has lower latency. For large amounts of data, the move could take hours to complete. While the move takes place, other consumers who also see a slightly reduced latency move to the same new provider—effectively increasing the latency for everyone, and making it a bad decision.
- the amount of time it takes to determine that the decision may be good is related to the cost of performing the action. Therefore, expensive decisions are carefully validated over longer periods than cheaper decisions, ensuring that undertaking the cost of the action will pay off in the future.
- the systems and methods above minimize bad decisions and decisions that would frequently alternate between the current and the new provider.
- Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.
- Embodiments of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible program carrier for execution by, or to control the operation of, data processing apparatus.
- the tangible program carrier can be computer readable medium, such as a machine-readable storage device, a machine-readable storage substrate, a memory device, or a combination of one or more of them.
- data processing apparatus data processor
- processing device encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers.
- the apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
- a computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
- a computer program does not necessarily correspond to a file in a file system.
- a program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code).
- a computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
- the processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output.
- the processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
- processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer.
- a processor will receive instructions and data from a read only memory or a random access memory or both.
- the essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data.
- a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
- mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
- a computer need not have such devices.
- a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, to name just a few.
- PDA personal digital assistant
- GPS Global Positioning System
- Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.
- semiconductor memory devices e.g., EPROM, EEPROM, and flash memory devices
- magnetic disks e.g., internal hard disks or removable disks
- magneto optical disks e.g., CD ROM and DVD-ROM disks.
- the processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
- embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.
- a display device e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor
- keyboard and a pointing device e.g., a mouse or a trackball
- Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
- Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described is this specification, or any combination of one or more such back end, middleware, or front end components.
- the components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a LAN and a wide area network (“WAN”), e.g., the Internet.
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Finance (AREA)
- Accounting & Taxation (AREA)
- Economics (AREA)
- Strategic Management (AREA)
- Human Resources & Organizations (AREA)
- General Engineering & Computer Science (AREA)
- General Business, Economics & Management (AREA)
- Marketing (AREA)
- Development Economics (AREA)
- Quality & Reliability (AREA)
- Entrepreneurship & Innovation (AREA)
- Game Theory and Decision Science (AREA)
- Operations Research (AREA)
- Tourism & Hospitality (AREA)
- Educational Administration (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
Description
price[x]=cost/(1−(U+x)/C)4
priceCPU[0.02C]=costCPU/(1−0.08C/C)4=costCPU/0.924=1.4*costCPU
priceI/O[0.2C]=costI/O/(1−0.8C/C)4=costI/O/0.24=625*costI/O.
R(n+1)←R(n)+a(n)[SLATarget−SLAParameter(R(n))]
S(tA)>=C(A)
Claims (23)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/940,583 USRE48714E1 (en) | 2009-06-26 | 2020-07-28 | Managing application performance in virtualization systems |
Applications Claiming Priority (8)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/492,890 US8914511B1 (en) | 2009-06-26 | 2009-06-26 | Managing resources in virtualization systems |
US201462077860P | 2014-11-10 | 2014-11-10 | |
US201462077898P | 2014-11-10 | 2014-11-10 | |
US201462077852P | 2014-11-10 | 2014-11-10 | |
US14/539,973 US9852011B1 (en) | 2009-06-26 | 2014-11-12 | Managing resources in virtualization systems |
US201562159883P | 2015-05-11 | 2015-05-11 | |
US14/810,327 US9830192B1 (en) | 2014-11-10 | 2015-07-27 | Managing application performance in virtualization systems |
US16/940,583 USRE48714E1 (en) | 2009-06-26 | 2020-07-28 | Managing application performance in virtualization systems |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/810,327 Reissue US9830192B1 (en) | 2009-06-26 | 2015-07-27 | Managing application performance in virtualization systems |
Publications (1)
Publication Number | Publication Date |
---|---|
USRE48714E1 true USRE48714E1 (en) | 2021-08-31 |
Family
ID=77417422
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/940,583 Active USRE48714E1 (en) | 2009-06-26 | 2020-07-28 | Managing application performance in virtualization systems |
Country Status (1)
Country | Link |
---|---|
US (1) | USRE48714E1 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220021601A1 (en) * | 2019-01-31 | 2022-01-20 | Huawei Technologies Co., Ltd. | Route computation method and related device |
US20220164208A1 (en) * | 2020-11-23 | 2022-05-26 | Google Llc | Coordinated container scheduling for improved resource allocation in virtual computing environment |
US11403131B2 (en) * | 2020-02-05 | 2022-08-02 | International Business Machines Corporation | Data analysis for predictive scaling of container(s) based on prior user transaction(s) |
US11411825B2 (en) * | 2015-08-25 | 2022-08-09 | Vmware, Inc. | In intelligent autoscale of services |
Citations (90)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6055571A (en) | 1997-11-20 | 2000-04-25 | Nec Usa, Inc. | Computer network with microeconomic flow control |
US20020147611A1 (en) | 2000-05-22 | 2002-10-10 | Greene William S. | Method and system for realizing a rendezvous service in a management operations center implemented in a global ecosystem of interrelated services |
US6480861B1 (en) | 1999-02-26 | 2002-11-12 | Merrill Lynch, Co., Inc | Distributed adaptive computing |
US20030154123A1 (en) * | 2002-02-13 | 2003-08-14 | Al Subbloie | System for managing equipment, services and service provider agreements |
US6735553B1 (en) | 2000-07-13 | 2004-05-11 | Netpredict, Inc. | Use of model calibration to achieve high accuracy in analysis of computer networks |
US20050005271A1 (en) | 2003-07-02 | 2005-01-06 | Clymer Shawn Allen | Methods, systems and computer program products for early warning of potential service level agreement violations |
US20050132363A1 (en) | 2003-12-16 | 2005-06-16 | Vijay Tewari | Method, apparatus and system for optimizing context switching between virtual machines |
US20050256683A1 (en) | 2002-08-14 | 2005-11-17 | Claus Hillermeier | Method and arrangement for designing a technical system |
US20060020939A1 (en) | 2004-06-10 | 2006-01-26 | International Business Machines Corporation | Exception handling in the processing of proposal requests in a grid computing environment |
US20060045100A1 (en) | 2004-09-02 | 2006-03-02 | Wolfgang Klausberger | Method for improving quality-of-service management in networks |
US7013296B1 (en) * | 1999-06-08 | 2006-03-14 | The Trustees Of Columbia University In The City Of New York | Using electronic security value units to control access to a resource |
US20060090163A1 (en) | 2004-10-21 | 2006-04-27 | Magnus Karisson | Method of controlling access to computing resource within shared computing environment |
US20060143617A1 (en) * | 2004-12-29 | 2006-06-29 | Knauerhase Robert C | Method, apparatus and system for dynamic allocation of virtual platform resources |
US20060167984A1 (en) | 2005-01-12 | 2006-07-27 | International Business Machines Corporation | Estimating future grid job costs by classifying grid jobs and storing results of processing grid job microcosms |
US20060168584A1 (en) * | 2004-12-16 | 2006-07-27 | International Business Machines Corporation | Client controlled monitoring of a current status of a grid job passed to an external grid environment |
US20060184937A1 (en) * | 2005-02-11 | 2006-08-17 | Timothy Abels | System and method for centralized software management in virtual machines |
US20060190482A1 (en) | 2005-02-22 | 2006-08-24 | Microsoft Corporation | Method and system for resource management |
US20060188011A1 (en) | 2004-11-12 | 2006-08-24 | Hewlett-Packard Development Company, L.P. | Automated diagnosis and forecasting of service level objective states |
US20060212332A1 (en) | 2005-03-16 | 2006-09-21 | Cluster Resources, Inc. | Simple integration of on-demand compute environment |
US20060244607A1 (en) | 2005-03-29 | 2006-11-02 | Zhong-Min Liu | RFID conveyor system |
US7140039B1 (en) | 1999-06-08 | 2006-11-21 | The Trustees Of Columbia University In The City Of New York | Identification of an attacker in an electronic system |
US20060265470A1 (en) * | 2005-05-19 | 2006-11-23 | Jerome Rolia | System and method for determining a partition of a consumer's resource access demands between a plurality of different classes of service |
US20070105630A1 (en) * | 2005-10-14 | 2007-05-10 | Leviathan Entertainment, Llc | Satisfaction of Financial Obligations in a Virtual Environment Via Virtual and Real World Currency |
US7272855B1 (en) | 1999-06-08 | 2007-09-18 | The Trustees Of Columbia University In The City Of New York | Unified monitoring and detection of intrusion attacks in an electronic system |
US20080052387A1 (en) * | 2006-08-22 | 2008-02-28 | Heinz John M | System and method for tracking application resource usage |
US20080080552A1 (en) | 2006-09-28 | 2008-04-03 | Microsoft Corporation | Hardware architecture for cloud services |
US20080083031A1 (en) | 2006-12-20 | 2008-04-03 | Microsoft Corporation | Secure service computation |
US20080104608A1 (en) | 2006-10-27 | 2008-05-01 | Hyser Chris D | Starting up at least one virtual machine in a physical machine by a load balancer |
US20080109241A1 (en) | 2006-11-02 | 2008-05-08 | D Alo Salvatore | Method, System and Computer Program For Metering Usage of Software Products With a Dynamically Optimised License Use |
US20080127348A1 (en) | 2006-08-31 | 2008-05-29 | Kenneth Largman | Network computer system and method using thin user client and virtual machine to provide immunity to hacking, viruses and spy ware |
US20080154837A1 (en) | 2006-12-21 | 2008-06-26 | Tomohiro Morimura | Performance evaluating apparatus, performance evaluating method, and program |
US20080155169A1 (en) | 2006-12-21 | 2008-06-26 | Hiltgen Daniel K | Implementation of Virtual Machine Operations Using Storage System Functionality |
US20080163194A1 (en) | 2007-01-02 | 2008-07-03 | Daniel Manuel Dias | Method and apparatus for deploying a set of virtual software resource templates to a set of nodes |
US20080201409A1 (en) * | 2007-02-20 | 2008-08-21 | Sun Microsystems, Inc | Method and system for managing computing resources using an electronic broker agent |
US20080244607A1 (en) * | 2007-03-27 | 2008-10-02 | Vladislav Rysin | Economic allocation and management of resources via a virtual resource market |
US20080250143A1 (en) * | 2007-04-06 | 2008-10-09 | France Telecom | Grid accounting method and system |
US7441023B2 (en) | 2003-09-25 | 2008-10-21 | Emc Corporation | Method and apparatus for modeling and analyzing MPLS and virtual private networks |
US20080301027A1 (en) | 2007-05-31 | 2008-12-04 | Boss Gregory J | Method, system, and program product for selecting a brokering method for obtaining desired service level characteristics |
US7464132B1 (en) | 2002-06-12 | 2008-12-09 | Emc Corporation | Method and apparatus for reference model change generation in managed systems |
US20080320474A1 (en) | 2007-06-19 | 2008-12-25 | Sun Microsystems, Inc. | Dynamic, temporary resource pools for use in virtualization |
US20080319926A1 (en) * | 2007-06-25 | 2008-12-25 | Akm Kamrul Alam | Method for dynamic usage billing |
US20090007126A1 (en) | 2007-06-29 | 2009-01-01 | Sun Microsystems, Inc. | Swap cap resource control for use in virtualization |
US7496655B2 (en) | 2002-05-01 | 2009-02-24 | Satyam Computer Services Limited Of Mayfair Centre | System and method for static and dynamic load analyses of communication network |
US20090052333A1 (en) | 2007-08-24 | 2009-02-26 | Vankov Vanko | Traffic independent survivability analysis |
US20090089406A1 (en) | 2007-09-30 | 2009-04-02 | Sun Microsystems, Inc. | Virtual cluster based upon operating system virtualization |
US20090164356A1 (en) | 2007-10-12 | 2009-06-25 | Alexander Bakman | Method, system and apparatus for calculating chargeback for virtualized computing resources |
US20090172666A1 (en) * | 2007-12-31 | 2009-07-02 | Netapp, Inc. | System and method for automatic storage load balancing in virtual server environments |
US20090182565A1 (en) * | 2008-01-10 | 2009-07-16 | At&T Services, Inc. | Aiding Creation of Service Offers Associated with a Service Delivery Framework |
WO2009091580A1 (en) | 2008-01-17 | 2009-07-23 | Aspera, Inc. | A practical model for high speed file delivery services supporting guaranteed delivery times and differentiated service levels |
US7590601B2 (en) | 2006-03-17 | 2009-09-15 | Wildtangent, Inc | Licensing media consumption using digital currency |
US20090240628A1 (en) | 2008-03-20 | 2009-09-24 | Co-Exprise, Inc. | Method and System for Facilitating a Negotiation |
US20090249488A1 (en) | 2008-03-28 | 2009-10-01 | Sap Ag | Flexible appliance hosting with coordination of deployment, licensing and configuration |
US20090276771A1 (en) | 2005-09-15 | 2009-11-05 | 3Tera, Inc. | Globally Distributed Utility Computing Cloud |
US20100005173A1 (en) * | 2008-07-03 | 2010-01-07 | International Business Machines Corporation | Method, system and computer program product for server selection, application placement and consolidation |
US20100057625A1 (en) | 2008-09-03 | 2010-03-04 | International Business Machines Corporation | Negotiation of power rates based on dynamic workload distribution |
US7685270B1 (en) * | 2005-03-31 | 2010-03-23 | Amazon Technologies, Inc. | Method and apparatus for measuring latency in web services |
US20100077449A1 (en) | 2008-09-22 | 2010-03-25 | International Business Machines | Calculating multi-tenancy resource requirements and automated tenant dynamic placement in a multi-tenant shared environment |
WO2010036731A2 (en) | 2008-09-24 | 2010-04-01 | Netapp, Inc. | Adaptive scheduling of storage operations based on utilization of multiple client and server resources in a distributed network storage system |
US20100088205A1 (en) * | 2008-10-02 | 2010-04-08 | Verizon Business Network Services Inc. | Methods, Systems and Computer Program Products for a Cloud Computing Spot Market Platform |
US20100106820A1 (en) | 2008-10-28 | 2010-04-29 | Vmware, Inc. | Quality of service management |
US7711789B1 (en) | 2007-12-07 | 2010-05-04 | 3 Leaf Systems, Inc. | Quality of service in virtual computing environments |
US20100142522A1 (en) | 2008-12-04 | 2010-06-10 | James Gardner | Methods and apparatus for adaptive error correction in networks |
US20100153945A1 (en) * | 2008-12-11 | 2010-06-17 | International Business Machines Corporation | Shared resource service provisioning using a virtual machine manager |
US20100180275A1 (en) * | 2009-01-15 | 2010-07-15 | International Business Machines Corporation | Techniques for placing applications in heterogeneous virtualized systems while minimizing power and migration cost |
US7774457B1 (en) * | 2005-03-25 | 2010-08-10 | Hewlett-Packard Development Company, L.P. | Resource evaluation for a batch job and an interactive session concurrently executed in a grid computing environment |
US20100318454A1 (en) * | 2009-06-16 | 2010-12-16 | Microsoft Corporation | Function and Constraint Based Service Agreements |
US20100319004A1 (en) * | 2009-06-16 | 2010-12-16 | Microsoft Corporation | Policy Management for the Cloud |
US7877754B2 (en) * | 2003-08-21 | 2011-01-25 | International Business Machines Corporation | Methods, systems, and media to expand resources available to a logical partition |
US20110131335A1 (en) | 2009-05-08 | 2011-06-02 | Cloudkick, Inc. | Methods and systems for cloud computing management |
US20110225277A1 (en) | 2010-03-11 | 2011-09-15 | International Business Machines Corporation | Placement of virtual machines based on server cost and network cost |
US20110276951A1 (en) | 2010-05-05 | 2011-11-10 | Microsoft Corporation | Managing runtime execution of applications on cloud computing systems |
US8060875B1 (en) * | 2006-05-26 | 2011-11-15 | Vmware, Inc. | System and method for multiple virtual teams |
US20120072762A1 (en) * | 2010-09-21 | 2012-03-22 | Atchison Lee A | Methods and systems for dynamically managing requests for computing capacity |
US8181175B1 (en) * | 2008-01-28 | 2012-05-15 | Hewlett-Packard Development Company, L.P. | Accounting for resource usage time by a virtual machine |
US20120131591A1 (en) | 2010-08-24 | 2012-05-24 | Jay Moorthi | Method and apparatus for clearing cloud compute demand |
US20120226796A1 (en) * | 2011-03-01 | 2012-09-06 | Morgan Christopher Edwin | Systems and methods for generating optimized resource consumption periods for multiple users on combined basis |
US8271345B1 (en) | 2008-12-22 | 2012-09-18 | Auctionomics Inc. | Systems and method for incorporating bidder budgets in multi-item auctions |
US8286174B1 (en) | 2006-04-17 | 2012-10-09 | Vmware, Inc. | Executing a multicomponent software application on a virtualized computer platform |
US20120317578A1 (en) * | 2011-06-09 | 2012-12-13 | Microsoft Corporation | Scheduling Execution of Complementary Jobs Based on Resource Usage |
US8347302B1 (en) | 2008-10-09 | 2013-01-01 | Amazon Technologies, Inc. | System-aware resource scheduling |
US8370898B1 (en) * | 2004-06-18 | 2013-02-05 | Adaptive Computing Enterprises, Inc. | System and method for providing threshold-based access to compute resources |
US8396807B1 (en) | 2009-06-26 | 2013-03-12 | VMTurbo, Inc. | Managing resources in virtualization systems |
US8429097B1 (en) * | 2009-08-12 | 2013-04-23 | Amazon Technologies, Inc. | Resource isolation using reinforcement learning and domain-specific constraints |
US8650296B1 (en) * | 2006-10-31 | 2014-02-11 | Hewlett-Packard Development Company, L.P. | Workload reallocation involving inter-server transfer of software license rights and intra-server transfer of hardware resources |
US20140081766A1 (en) | 2005-05-16 | 2014-03-20 | Jorge Maass | Transaction Arbiter System and Method |
US20140376368A1 (en) * | 2013-06-25 | 2014-12-25 | Amazon Technologies, Inc. | Compound token buckets for burst-mode admission control |
US20140380324A1 (en) * | 2013-06-25 | 2014-12-25 | Amazon Technologies, Inc. | Burst-mode admission control using token buckets |
US8959249B1 (en) * | 2010-09-15 | 2015-02-17 | Emc Corporation | Cooperative cloud I/O scheduler |
US9589495B2 (en) | 2013-05-28 | 2017-03-07 | Innolux Corporation | Liquid crystal display and display method thereof |
US9589501B2 (en) | 2006-12-15 | 2017-03-07 | Lapis Semiconductor Co., Ltd. | Display drive circuit including an output terminal |
-
2020
- 2020-07-28 US US16/940,583 patent/USRE48714E1/en active Active
Patent Citations (98)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6055571A (en) | 1997-11-20 | 2000-04-25 | Nec Usa, Inc. | Computer network with microeconomic flow control |
US6480861B1 (en) | 1999-02-26 | 2002-11-12 | Merrill Lynch, Co., Inc | Distributed adaptive computing |
US7140039B1 (en) | 1999-06-08 | 2006-11-21 | The Trustees Of Columbia University In The City Of New York | Identification of an attacker in an electronic system |
US7272855B1 (en) | 1999-06-08 | 2007-09-18 | The Trustees Of Columbia University In The City Of New York | Unified monitoring and detection of intrusion attacks in an electronic system |
US7013296B1 (en) * | 1999-06-08 | 2006-03-14 | The Trustees Of Columbia University In The City Of New York | Using electronic security value units to control access to a resource |
US20020147611A1 (en) | 2000-05-22 | 2002-10-10 | Greene William S. | Method and system for realizing a rendezvous service in a management operations center implemented in a global ecosystem of interrelated services |
US6735553B1 (en) | 2000-07-13 | 2004-05-11 | Netpredict, Inc. | Use of model calibration to achieve high accuracy in analysis of computer networks |
US20030154123A1 (en) * | 2002-02-13 | 2003-08-14 | Al Subbloie | System for managing equipment, services and service provider agreements |
US7496655B2 (en) | 2002-05-01 | 2009-02-24 | Satyam Computer Services Limited Of Mayfair Centre | System and method for static and dynamic load analyses of communication network |
US7464132B1 (en) | 2002-06-12 | 2008-12-09 | Emc Corporation | Method and apparatus for reference model change generation in managed systems |
US20050256683A1 (en) | 2002-08-14 | 2005-11-17 | Claus Hillermeier | Method and arrangement for designing a technical system |
US20050005271A1 (en) | 2003-07-02 | 2005-01-06 | Clymer Shawn Allen | Methods, systems and computer program products for early warning of potential service level agreement violations |
US7877754B2 (en) * | 2003-08-21 | 2011-01-25 | International Business Machines Corporation | Methods, systems, and media to expand resources available to a logical partition |
US7441023B2 (en) | 2003-09-25 | 2008-10-21 | Emc Corporation | Method and apparatus for modeling and analyzing MPLS and virtual private networks |
US20050132363A1 (en) | 2003-12-16 | 2005-06-16 | Vijay Tewari | Method, apparatus and system for optimizing context switching between virtual machines |
US20060020939A1 (en) | 2004-06-10 | 2006-01-26 | International Business Machines Corporation | Exception handling in the processing of proposal requests in a grid computing environment |
US8370898B1 (en) * | 2004-06-18 | 2013-02-05 | Adaptive Computing Enterprises, Inc. | System and method for providing threshold-based access to compute resources |
US20060045100A1 (en) | 2004-09-02 | 2006-03-02 | Wolfgang Klausberger | Method for improving quality-of-service management in networks |
US20060090163A1 (en) | 2004-10-21 | 2006-04-27 | Magnus Karisson | Method of controlling access to computing resource within shared computing environment |
US20060188011A1 (en) | 2004-11-12 | 2006-08-24 | Hewlett-Packard Development Company, L.P. | Automated diagnosis and forecasting of service level objective states |
US20060168584A1 (en) * | 2004-12-16 | 2006-07-27 | International Business Machines Corporation | Client controlled monitoring of a current status of a grid job passed to an external grid environment |
US20060143617A1 (en) * | 2004-12-29 | 2006-06-29 | Knauerhase Robert C | Method, apparatus and system for dynamic allocation of virtual platform resources |
US20060167984A1 (en) | 2005-01-12 | 2006-07-27 | International Business Machines Corporation | Estimating future grid job costs by classifying grid jobs and storing results of processing grid job microcosms |
US20060184937A1 (en) * | 2005-02-11 | 2006-08-17 | Timothy Abels | System and method for centralized software management in virtual machines |
US20060190482A1 (en) | 2005-02-22 | 2006-08-24 | Microsoft Corporation | Method and system for resource management |
US20060212332A1 (en) | 2005-03-16 | 2006-09-21 | Cluster Resources, Inc. | Simple integration of on-demand compute environment |
US7774457B1 (en) * | 2005-03-25 | 2010-08-10 | Hewlett-Packard Development Company, L.P. | Resource evaluation for a batch job and an interactive session concurrently executed in a grid computing environment |
US20060244607A1 (en) | 2005-03-29 | 2006-11-02 | Zhong-Min Liu | RFID conveyor system |
US7685270B1 (en) * | 2005-03-31 | 2010-03-23 | Amazon Technologies, Inc. | Method and apparatus for measuring latency in web services |
US20140081766A1 (en) | 2005-05-16 | 2014-03-20 | Jorge Maass | Transaction Arbiter System and Method |
US20060265470A1 (en) * | 2005-05-19 | 2006-11-23 | Jerome Rolia | System and method for determining a partition of a consumer's resource access demands between a plurality of different classes of service |
US20090276771A1 (en) | 2005-09-15 | 2009-11-05 | 3Tera, Inc. | Globally Distributed Utility Computing Cloud |
US20070105630A1 (en) * | 2005-10-14 | 2007-05-10 | Leviathan Entertainment, Llc | Satisfaction of Financial Obligations in a Virtual Environment Via Virtual and Real World Currency |
US7590601B2 (en) | 2006-03-17 | 2009-09-15 | Wildtangent, Inc | Licensing media consumption using digital currency |
US8286174B1 (en) | 2006-04-17 | 2012-10-09 | Vmware, Inc. | Executing a multicomponent software application on a virtualized computer platform |
US8060875B1 (en) * | 2006-05-26 | 2011-11-15 | Vmware, Inc. | System and method for multiple virtual teams |
US20080052387A1 (en) * | 2006-08-22 | 2008-02-28 | Heinz John M | System and method for tracking application resource usage |
US20080127348A1 (en) | 2006-08-31 | 2008-05-29 | Kenneth Largman | Network computer system and method using thin user client and virtual machine to provide immunity to hacking, viruses and spy ware |
US20080080552A1 (en) | 2006-09-28 | 2008-04-03 | Microsoft Corporation | Hardware architecture for cloud services |
US20080104608A1 (en) | 2006-10-27 | 2008-05-01 | Hyser Chris D | Starting up at least one virtual machine in a physical machine by a load balancer |
US8650296B1 (en) * | 2006-10-31 | 2014-02-11 | Hewlett-Packard Development Company, L.P. | Workload reallocation involving inter-server transfer of software license rights and intra-server transfer of hardware resources |
US20080109241A1 (en) | 2006-11-02 | 2008-05-08 | D Alo Salvatore | Method, System and Computer Program For Metering Usage of Software Products With a Dynamically Optimised License Use |
US9589501B2 (en) | 2006-12-15 | 2017-03-07 | Lapis Semiconductor Co., Ltd. | Display drive circuit including an output terminal |
US20080083031A1 (en) | 2006-12-20 | 2008-04-03 | Microsoft Corporation | Secure service computation |
US20080154837A1 (en) | 2006-12-21 | 2008-06-26 | Tomohiro Morimura | Performance evaluating apparatus, performance evaluating method, and program |
US7840517B2 (en) | 2006-12-21 | 2010-11-23 | Hitachi, Ltd. | Performance evaluating apparatus, method, and computer-readable medium |
US20080155169A1 (en) | 2006-12-21 | 2008-06-26 | Hiltgen Daniel K | Implementation of Virtual Machine Operations Using Storage System Functionality |
US20080163194A1 (en) | 2007-01-02 | 2008-07-03 | Daniel Manuel Dias | Method and apparatus for deploying a set of virtual software resource templates to a set of nodes |
US20080201409A1 (en) * | 2007-02-20 | 2008-08-21 | Sun Microsystems, Inc | Method and system for managing computing resources using an electronic broker agent |
US20080244607A1 (en) * | 2007-03-27 | 2008-10-02 | Vladislav Rysin | Economic allocation and management of resources via a virtual resource market |
US20080250143A1 (en) * | 2007-04-06 | 2008-10-09 | France Telecom | Grid accounting method and system |
US20080301027A1 (en) | 2007-05-31 | 2008-12-04 | Boss Gregory J | Method, system, and program product for selecting a brokering method for obtaining desired service level characteristics |
US20080320474A1 (en) | 2007-06-19 | 2008-12-25 | Sun Microsystems, Inc. | Dynamic, temporary resource pools for use in virtualization |
US20080319926A1 (en) * | 2007-06-25 | 2008-12-25 | Akm Kamrul Alam | Method for dynamic usage billing |
US20090007126A1 (en) | 2007-06-29 | 2009-01-01 | Sun Microsystems, Inc. | Swap cap resource control for use in virtualization |
US20090052333A1 (en) | 2007-08-24 | 2009-02-26 | Vankov Vanko | Traffic independent survivability analysis |
US20090089406A1 (en) | 2007-09-30 | 2009-04-02 | Sun Microsystems, Inc. | Virtual cluster based upon operating system virtualization |
US20090164356A1 (en) | 2007-10-12 | 2009-06-25 | Alexander Bakman | Method, system and apparatus for calculating chargeback for virtualized computing resources |
US7711789B1 (en) | 2007-12-07 | 2010-05-04 | 3 Leaf Systems, Inc. | Quality of service in virtual computing environments |
US20090172666A1 (en) * | 2007-12-31 | 2009-07-02 | Netapp, Inc. | System and method for automatic storage load balancing in virtual server environments |
US20090182565A1 (en) * | 2008-01-10 | 2009-07-16 | At&T Services, Inc. | Aiding Creation of Service Offers Associated with a Service Delivery Framework |
US20090276271A1 (en) | 2008-01-17 | 2009-11-05 | Aspera, Inc. | Practical model for high speed file delivery services supporting guaranteed delivery times and differentiated service levels |
US8051017B2 (en) | 2008-01-17 | 2011-11-01 | Aspera, Inc. | Practical model for high speed file delivery services supporting guaranteed delivery times and differentiated service levels |
WO2009091580A1 (en) | 2008-01-17 | 2009-07-23 | Aspera, Inc. | A practical model for high speed file delivery services supporting guaranteed delivery times and differentiated service levels |
US8181175B1 (en) * | 2008-01-28 | 2012-05-15 | Hewlett-Packard Development Company, L.P. | Accounting for resource usage time by a virtual machine |
US20090240628A1 (en) | 2008-03-20 | 2009-09-24 | Co-Exprise, Inc. | Method and System for Facilitating a Negotiation |
US20090249488A1 (en) | 2008-03-28 | 2009-10-01 | Sap Ag | Flexible appliance hosting with coordination of deployment, licensing and configuration |
US20100005173A1 (en) * | 2008-07-03 | 2010-01-07 | International Business Machines Corporation | Method, system and computer program product for server selection, application placement and consolidation |
US20100057625A1 (en) | 2008-09-03 | 2010-03-04 | International Business Machines Corporation | Negotiation of power rates based on dynamic workload distribution |
US20100077449A1 (en) | 2008-09-22 | 2010-03-25 | International Business Machines | Calculating multi-tenancy resource requirements and automated tenant dynamic placement in a multi-tenant shared environment |
WO2010036731A2 (en) | 2008-09-24 | 2010-04-01 | Netapp, Inc. | Adaptive scheduling of storage operations based on utilization of multiple client and server resources in a distributed network storage system |
US20100088205A1 (en) * | 2008-10-02 | 2010-04-08 | Verizon Business Network Services Inc. | Methods, Systems and Computer Program Products for a Cloud Computing Spot Market Platform |
US8347302B1 (en) | 2008-10-09 | 2013-01-01 | Amazon Technologies, Inc. | System-aware resource scheduling |
US20100106820A1 (en) | 2008-10-28 | 2010-04-29 | Vmware, Inc. | Quality of service management |
US20100142522A1 (en) | 2008-12-04 | 2010-06-10 | James Gardner | Methods and apparatus for adaptive error correction in networks |
US20100153945A1 (en) * | 2008-12-11 | 2010-06-17 | International Business Machines Corporation | Shared resource service provisioning using a virtual machine manager |
US8271345B1 (en) | 2008-12-22 | 2012-09-18 | Auctionomics Inc. | Systems and method for incorporating bidder budgets in multi-item auctions |
US20100180275A1 (en) * | 2009-01-15 | 2010-07-15 | International Business Machines Corporation | Techniques for placing applications in heterogeneous virtualized systems while minimizing power and migration cost |
US20110131335A1 (en) | 2009-05-08 | 2011-06-02 | Cloudkick, Inc. | Methods and systems for cloud computing management |
US20100319004A1 (en) * | 2009-06-16 | 2010-12-16 | Microsoft Corporation | Policy Management for the Cloud |
US20100318454A1 (en) * | 2009-06-16 | 2010-12-16 | Microsoft Corporation | Function and Constraint Based Service Agreements |
US8914511B1 (en) | 2009-06-26 | 2014-12-16 | VMTurbo, Inc. | Managing resources in virtualization systems |
US8762531B1 (en) | 2009-06-26 | 2014-06-24 | VMTurbo, Inc. | Managing resources in virtualization systems |
US8661131B1 (en) | 2009-06-26 | 2014-02-25 | VMTurbo, Inc. | Managing resources in virtualization systems |
US8396807B1 (en) | 2009-06-26 | 2013-03-12 | VMTurbo, Inc. | Managing resources in virtualization systems |
US8433801B1 (en) | 2009-06-26 | 2013-04-30 | VMTurbo, Inc. | Managing resources in virtualization systems |
US8429097B1 (en) * | 2009-08-12 | 2013-04-23 | Amazon Technologies, Inc. | Resource isolation using reinforcement learning and domain-specific constraints |
US20110225277A1 (en) | 2010-03-11 | 2011-09-15 | International Business Machines Corporation | Placement of virtual machines based on server cost and network cost |
US8478878B2 (en) | 2010-03-11 | 2013-07-02 | International Business Machines Corporation | Placement of virtual machines based on server cost and network cost |
US20110276951A1 (en) | 2010-05-05 | 2011-11-10 | Microsoft Corporation | Managing runtime execution of applications on cloud computing systems |
US20120131591A1 (en) | 2010-08-24 | 2012-05-24 | Jay Moorthi | Method and apparatus for clearing cloud compute demand |
US8959249B1 (en) * | 2010-09-15 | 2015-02-17 | Emc Corporation | Cooperative cloud I/O scheduler |
US20120072762A1 (en) * | 2010-09-21 | 2012-03-22 | Atchison Lee A | Methods and systems for dynamically managing requests for computing capacity |
US20120226796A1 (en) * | 2011-03-01 | 2012-09-06 | Morgan Christopher Edwin | Systems and methods for generating optimized resource consumption periods for multiple users on combined basis |
US20120317578A1 (en) * | 2011-06-09 | 2012-12-13 | Microsoft Corporation | Scheduling Execution of Complementary Jobs Based on Resource Usage |
US9589495B2 (en) | 2013-05-28 | 2017-03-07 | Innolux Corporation | Liquid crystal display and display method thereof |
US20140376368A1 (en) * | 2013-06-25 | 2014-12-25 | Amazon Technologies, Inc. | Compound token buckets for burst-mode admission control |
US20140380324A1 (en) * | 2013-06-25 | 2014-12-25 | Amazon Technologies, Inc. | Burst-mode admission control using token buckets |
Non-Patent Citations (30)
Title |
---|
"Apache Hadoop NextGen MapReduce (YARN)", The Apache Software Foundation, Nov. 2014, 2 pages. http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html. |
"Big Ideas: Demystifying Hadoop", EMC, YouTube Video, Published Jul. 2012. https://www.youtube.com/watch?v=xJHv5t8jcM8. |
"Kubernetes Design Overview", GitHub, GoogleCloudPlatform, Oct. 2014, 4 pages. https://github.com/GoogleCloudPlatform/kubernetes/blob/master/DESIGN.md. |
"MapReduce Tutorial", The Apache Software Foundation, Nov. 2014, 24 pages. http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html. |
Byde et al., "Market-Based Resource Allocation for Utility Data Centers", HP Laboratories Bristol, Sep. 2003, 16 pages. |
Dailianas et al., "MarketNet: Market-based Protection of Network Systems and Services—an Application to SNMP Protection", INFOCOM 2000, Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies, IEEE, vol. 3, Mar. 2000, 10 pages. |
Dailianas et al., "MarketNet: Market-based Protection of Network Systems and Services—an Application to SNMP Protection", INFOCOM 2000, Nineteenth Annual Joint Congerence of the IEEE Computer and Communications Societies, IEEE, vol. 3, Mar. 2000, 10 pages. |
Dailianas et al., "Use of Currency for Access Control in Large-Scale Information Systems", Ph.D. Thesis Proposal, Columbia University, Department of Computer Science, Sep. 1998, 33 pages. |
Dias et al., "Online Traffic-aware Virtual Machine Placement in Data Center Networks", Global Information Infrastructure and Networking Symposium (GHS), 2012, 8 pages. |
Erickson er al., "Using Network Knowledge to Improve Performance in Virtualized Data Centers", PhD thesis, Stanford University, May 2013, 10 pages. |
Ferguson er al., "Economic Models for Allocating Resources in Computer Systems", Market Based Control of Distributed Systems. World Scientific, 2006, 26 pages. |
Ferguson et al., "Microeconomic Algorithms for Load Balancing in Distributed Computer Systems", IBM T.J. Watson Research Center, Nov. 2008, 9 pages. |
Gao et al., Economics-Inspired Decentralized Control Approach for Adaptive Grid Services and Applications, International Journal of Intelligent Systems, vol. 21, 2006, 20 pages. |
Huberman et al., "Distributed Computation as an Economic System", Journal of Economic Perspectives, vol. 9, No. 1, 1995, 12 pages. |
Meng et al., "Improving the Scalability of Data Center Networks with Traffic-aware Virtual Machine Placement", INFOCOM, 2010 Proceedings IEEE, Mar. 2010, 9 pages. |
Sandholm, "Managing Service Levels in Grid Computing Systems: Quota Policy and Computation; Computational Market Approaches", KTH Computer Science and Communication, Licentiate Thesis, Stockholm, Sweden, 2007, 105 pages. |
Sarangan et al., "Comparative Study of Protocols for Dynamic Service Negotiation in the Next-Generation Internet", IEEE Communications Magazine, vol. 44, No. 3, Mar. 2006, 6 pages. |
Stonebraker et al., "An Economic Paradigm for Query Processing and Data Migration in Mariposa", IEEE Xplore, 1994, 10 pages. |
Sutherland et al., "A Futures Market in Computer Time", Aiken Computation Lab, vol. 11, No. 6, Jun. 1968, 3 pages. |
Turner et al., "A Lightweight Current Paradigm for the P2P Resource Market", Electronic Commerce Research—ECR, 2003, 15 pages. |
U.S. Appl. No. 09/589,495, filed Jun. 7, 2000, Yemini et al. |
U.S. Appl. No. 09/589,501, filed Jun. 7, 2000, Yemini et al. |
Waldspurger et al., "Spawn: A Distributed Computational Economy", IEEE Transactions on Software Engineering, vol. 18, No. 2, Feb. 1992, 15 pages. |
Wang et al., "Pricing Network Resources for Adaptive Applications", IEEE/ACM Transactions on Networking, vol. 14, No. 3, Jun. 2006, 14 pages. |
Yemini et al., "MarketNet: A Market-Based Architecture for Survivable Large-Scale Information Systems", Columbia University, Department of Computer Science, Aug. 2002, 6 pages. |
Yemini et al., "MarketNet: A Survivable, Market-Based Architecture for Large-Scale Information Systems", Final Technical Report, Air Force Research Laboratory, Aug. 2002, 27 pages. |
Yemini et al., "MarketNet: Market-Based Protection of Information Systems", ICE '98 Proceedings of the first international conference on Information and computation economics, 1998, 12 pages. |
Yemini et al., "MarketNet: Protecting Access to Information Systems Through Financial Market Controls", Decision Support Systems—Special Issue on Information and Computational Economics, vol. 28, No. 1-2, Mar. 2000, 20 pages. |
Yemini et al., "MarketNet: Using Virtual Currency to Protect Information Systems", Research and Advanced Technology for Digital Libraries, Second European Conference, ECDL '98, 1998, 12 pages. |
Yuksel et al., "Effect of Pricing Intervals on Congestion-Sensitivity of Network Prices", Telecommunication Systems, vol. 28, No. 1, 2005, 21 pages. |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11411825B2 (en) * | 2015-08-25 | 2022-08-09 | Vmware, Inc. | In intelligent autoscale of services |
US20220021601A1 (en) * | 2019-01-31 | 2022-01-20 | Huawei Technologies Co., Ltd. | Route computation method and related device |
US11403131B2 (en) * | 2020-02-05 | 2022-08-02 | International Business Machines Corporation | Data analysis for predictive scaling of container(s) based on prior user transaction(s) |
US20220164208A1 (en) * | 2020-11-23 | 2022-05-26 | Google Llc | Coordinated container scheduling for improved resource allocation in virtual computing environment |
US11740921B2 (en) * | 2020-11-23 | 2023-08-29 | Google Llc | Coordinated container scheduling for improved resource allocation in virtual computing environment |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10671953B1 (en) | Systems, apparatus and methods for cost and performance-based movement of applications and workloads in a multiple-provider system | |
US10346216B1 (en) | Systems, apparatus and methods for management of software containers | |
US9888067B1 (en) | Managing resources in container systems | |
US9830192B1 (en) | Managing application performance in virtualization systems | |
US10924537B2 (en) | Systems, apparatus and methods for cost and performance-based management of resources in a cloud environment | |
US9858123B1 (en) | Moving resource consumers in computer systems | |
US10552221B2 (en) | Systems, apparatus and methods for managing resources in computer systems | |
US10673952B1 (en) | Systems, apparatus, and methods for managing computer workload availability and performance | |
US11080084B1 (en) | Managing resources in virtualization systems | |
US9805345B1 (en) | Systems, apparatus, and methods for managing quality of service agreements | |
US11689613B2 (en) | Systems, apparatus and methods for cost and performance-based management of resources in a cloud environment | |
US8433801B1 (en) | Managing resources in virtualization systems | |
US11386371B2 (en) | Systems, apparatus and methods for cost and performance-based movement of applications and workloads in a multiple-provider system | |
USRE48680E1 (en) | Managing resources in container systems | |
USRE48714E1 (en) | Managing application performance in virtualization systems | |
US9830566B1 (en) | Managing resources in computer systems using action permits | |
US10819776B2 (en) | Automated resource-price calibration and recalibration by an automated resource-exchange system | |
US20180060894A1 (en) | Methods and systems that generated resource-provision bids in an automated resource-exchange system | |
US10552586B1 (en) | Systems, apparatus and methods for management of computer-based software licenses | |
US11502972B2 (en) | Capacity optimization in an automated resource-exchange system | |
Youn et al. | Cloud broker and cloudlet for workflow scheduling | |
USRE48663E1 (en) | Moving resource consumers in computer systems | |
US11272013B1 (en) | Systems, apparatus, and methods for managing computer workload availability and performance | |
Costache et al. | Market-based autonomous resource and application management in private clouds | |
Assudani et al. | Power efficient and workload aware scheduling in cloud |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
FEPP | Fee payment procedure |
Free format text: PETITION RELATED TO MAINTENANCE FEES GRANTED (ORIGINAL EVENT CODE: PTGR); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: TURBONOMIC, INC., MASSACHUSETTS Free format text: CHANGE OF NAME;ASSIGNOR:VMTURBO, INC.;REEL/FRAME:056408/0116 Effective date: 20160815 Owner name: VMTURBO, INC., MASSACHUSETTS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CROUCHMAN, CHARLES;BORODIANSKY, MICHAEL;DAILIANAS, APOSTOLOS;AND OTHERS;SIGNING DATES FROM 20151111 TO 20151112;REEL/FRAME:056368/0653 |
|
AS | Assignment |
Owner name: TURBONOMIC, INC., MASSACHUSETTS Free format text: CHANGE OF NAME;ASSIGNOR:VMTURBO, INC.;REEL/FRAME:056819/0173 Effective date: 20160815 |
|
AS | Assignment |
Owner name: TURBONOMIC, INC., MASSACHUSETTS Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE APPLICATION NUMBER PREVIOUSLY RECORDED AT REEL: 056408 FRAME: 0116. ASSIGNOR(S) HEREBY CONFIRMS THE CHANGE OF NAME;ASSIGNOR:VMTURBO, INC.;REEL/FRAME:057603/0146 Effective date: 20160815 |
|
AS | Assignment |
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW YORK Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:TURBONOMIC, INC.;REEL/FRAME:062202/0030 Effective date: 20221221 |